An LLM judging violent repricings
When a market moves 20¢ in an hour, can an LLM tell real news from panic before the book settles? Early on it looked like it could. Past its 30-trade gate, the high-confidence calls lost money.
The hypothesisThe general LLM-forecaster failed — the market beats the model on ambient news. But a dislocation is different: a 20¢-in-an-hour reprice is either real information or panic, the wires lag the move by minutes, and a model reading fresh headlines against resolution rules can classify which is which while the book is still displaced. Judge the move, don't predict the world.
The test
An event-triggered daemon scans top markets for violent hourly repricings on real volume (sports excluded — those moves are just the game), then has an LLM judge each one up to three times as headlines develop: given the resolution rules, dated fresh news, and the price withheld, is the new level justified? Verdicts with the model's own confidence score become would-enter signals at the live executable ask, settled at resolution. Same model family that failed the forecasting test — different question shape.
The result
At our August update the high-confidence calls stood at 11 wins and 4 losses, +73% at the ask, and we wrote that this looked like an honest confidence signal. The next 21 settlements reversed it. Past the 30-trade gate, the high-confidence cohort is 13 wins, 23 losses, −18.0% ROI at the logged ask over 36 settled calls. Counting one position per market (some dislocations were judged more than once), it is 7 wins in 21 markets, −27.5%. All confidence levels together: 30 wins in 83, −21.1%.
By month, one position per market, the high-confidence calls were positive in June (+40%, 4 markets) and July (+53%, 3), then −71% in August (5) and −60% in September (9). The early run was seven markets — the sample size at which strategies most often look like they work.
Conclusion
Negative. Once the sample grew, the confidence score did not separate good calls from bad ones: the high-confidence calls won 36% of the time, the same rate as every call together. This was the last LLM configuration in the lab still testing positive, and it joins the forecasting test in the failed column. In August this entry warned that fifteen good settlements is where most published strategies declare victory. This time the strategy was ours.