A strategy that tests well and then fails in a live account is the normal outcome, not the exception. Usually one or more of the following is responsible.
1. Overfitting
The most common problem and the hardest to notice.
The moment you sweep parameters looking for the best combination, you have stopped designing a strategy and started carving a shape that fits the accidental bumps in past data. If moving a buy trigger from −1.0% to −1.1% transforms the result, that is noise, not a discovery.
Warning signs:
- Small parameter changes move results a lot
- Five or six filters stacked together
- One specific value works and its neighbours do not
A healthy strategy sits on a broad plateau. If −1.0% works, then −0.9% and −1.1% should work similarly. A single sharp peak will not repeat out of sample.
Test it by shifting every parameter by ±20%. If the result halves, the strategy is not usable.
2. Look-ahead bias
Using today's close to decide a trade you booked at today's open. It sounds obvious, but it hides well in real code.
- Including today's close in a 20-day average, then trading on that average today
- Building weekly indicators from a week that has not finished
- Using split- and dividend-adjusted prices while forgetting the adjustment was applied retroactively
- Assuming fills at prices between the high and the low
One rule covers all of it: any value used in a decision must have been final at the moment of the decision. Deciding on the previous close and filling at today's close is the safest convention.
Look-ahead bias produces implausibly good numbers. If annual returns exceed 100%, a bug is far more likely than genius.
3. Survivorship bias
Test the past using only tickers that exist today and every company that went to zero has quietly left your sample. Of course the results look good.
With leveraged ETFs the issue takes a different shape: when did the product launch?
- SOXL: listed 2010
- TQQQ: listed 2010
- FNGU: listed 2018
You cannot test these through the 2008 crisis. The data does not exist. A "20-year backtest" may in practice be a look at the post-2010 bull market only, and that period was historically unusual.
Synthesising longer histories from the underlying index is possible, but you then have to model expense ratios and daily rebalancing costs, and confidence drops accordingly.
4. Execution costs and liquidity
Backtests assume you always get the size you want at the price you want. You do not.
Slippage. You asked for the close and filled slightly worse. The more often a strategy trades, the more this accumulates. A daily strategy losing 0.1% round trip can shed tens of percent a year.
Fees and taxes. High-turnover strategies look very different after tax, and backtest figures are almost always pre-tax.
Thin books. In a crash there is no size at your price. The exact moment your strategy says buy is the moment execution is hardest.
A quick test: raise the fee from 0.1% to 0.3%. If the profit disappears, the strategy was being eaten by trading costs all along.
5. Regime dependence
The deepest problem. A strategy is built for a particular market environment, and if your test window contains only that environment, you have not validated anything.
Split-buy strategies do well in markets that fall and then recover. They buy the dip and sell the bounce. In a market that grinds down without recovering, they simply keep accumulating.
Testing only 2010–2021 hides this, because corrections in that period generally recovered quickly. A year like 2022, which declined all the way through, has to be included.
Windows worth covering:
- Q4 2018: short and sharp
- Feb–Mar 2020: crash followed by a very fast recovery
- All of 2022: a long decline with no recovery
- 2015–2016: directionless chop
Yearly breakdowns reveal a strategy's character. Good and bad years should be mixed; if one year created all the profit, look again.
Checklist before trusting a result
- [ ] Results hold when parameters shift ±20%
- [ ] Indicators are computed from prior-day values
- [ ] The instrument actually existed during the test window
- [ ] Profit survives tripling the fee
- [ ] A genuine decline is inside the window
- [ ] Returns are not concentrated in a single year
- [ ] It beats buy-and-hold on drawdown, not just return
A backtest does not tell you what a strategy will earn. It tells you how a strategy behaved in specific past conditions. That is genuinely useful, but expecting more is where the trouble starts.