The Backtest You Should Not Trust
In this essay
Suppose an AI system reads earnings reports, assigns sentiment scores, and buys the most optimistic companies. The backtest looks excellent. The model is sophisticated, the prompt is detailed, and the equity curve rises smoothly.
Before asking which model produced the scores, ask a less glamorous question: exactly when could the strategy have known each input? An answer that sounds like “sometime that quarter” is not precise enough to support a trading simulation.
Build an information clock
A financial event has several relevant times: the period it describes, when it was first released, when a data vendor delivered it, and when the research system could actually use it. These are not the same timestamp.
A report describing a quarter ending March 31 may arrive weeks later. Joining it to prices using March 31 silently gives the strategy information from the future. Revised economic data cause a similar problem if the backtest uses today’s cleaned history instead of the vintage originally available.
For every feature, retain an available_at field in addition to the economic period. Then define the decision time and the first executable trade time. A report released after the exchange closes cannot justify a trade at that day’s already-observed closing price.
1 | report period → public release → ingestion → decision → executable order |
This is a proposed audit sequence, not a guarantee that a particular dataset captures every delay correctly.
A modern language model complicates historical research
A model evaluated on an old filing may have encountered later information about the issuer during training. Asking it to “pretend it is 2018” does not establish an information boundary. Even if retrieval supplies only dated documents, the model’s parameters can contain later associations.
That does not make language models useless for historical research. It limits the claim the experiment can support. A retrospective test may be an experiment in extracting a signal from documents, while still falling short of a valid estimate of contemporaneously achievable returns.
Possible mitigations include using a model with a documented training cutoff before the test period, comparing against simple text baselines, and running a genuinely prospective paper-trading test. Each has limitations: a stated cutoff is not a complete inventory of training examples, and paper trading may model execution imperfectly.
The test report should say which risk is unresolved rather than treating the prompt as a historical firewall.
Count the strategies you tried
Testing many candidate signals and publishing only the winner changes the meaning of the winning result. Parameter choices include prompts, model versions, rebalance rules, universes, transaction-cost assumptions, and dates. They all belong in the experiment log.
Bailey and colleagues’ paper on the probability of backtest overfitting formalizes the problem of selecting a strategy for strong in-sample performance that then disappoints out of sample. It is evidence about a research failure mode, not proof that any particular strategy is overfit.
A practical defense is to record the candidate family before testing, choose on a development window, and reserve an untouched chronological test window. If repeated inspection of that window drives new choices, it has become development data. Calling it “out of sample” does not restore its independence.
Trade something that could actually have traded
An idealized fill price can contribute more apparent performance than the signal. If the decision requires today’s closing data, using today’s exact closing price as an ordinary executable fill needs a defensible mechanism. Otherwise, the next realistically available opportunity is the appropriate starting point.
Include commissions where relevant, bid-ask spreads, plausible slippage, and turnover. Short positions add borrow availability and cost. Capacity matters when the trade size is meaningful relative to available liquidity.
Survivorship bias is another quiet source of improvement. A universe built from companies that exist today removes some historical failures. Use a point-in-time universe and retain delisted securities where the experiment requires them.
Evaluate the research process as well as the return
A useful report makes the following items available together:
| Audit question | Evidence to retain |
|---|---|
| What was knowable? | Release timestamps and data vintages |
| What was tried? | Full experiment and prompt log |
| What was executable? | Decision timing and fill rules |
| What was paid? | Cost, turnover, and borrow assumptions |
| What was held out? | Frozen chronological test protocol |
Also compare the strategy with a suitable passive benchmark and a simpler rule. Report drawdowns and exposure, rather than a single average return. Test whether a small number of observations explain most of the result.
These checks can reject weak evidence; passing them does not establish a permanent edge. Markets change, investors adapt, and a historical relationship may be accidental.
The most useful backtest is one that makes it easy to discover why the strategy might fail. A beautiful chart is a poor substitute for that kind of honesty.