What backtesting is, and whether it actually works
Backtesting is running a trading rule over historical prices to see what it would have done. The definition is the easy part. The question worth answering is the one people actually type: does it work.
The short answer
Yes, under three conditions, and the third one is almost never enforced. The rule must never see the future, the fills must be realistic, and the result must be corrected for how many variants you tried. Only 2 of the 20 platforms assessed here do the third.
So most backtests are not wrong arithmetic. They are the best of many attempts, reported as if it had been the only one.
How it works, mechanically
The engine steps through history in order. At each bar it shows the rule only what would have been visible at that moment, and when the rule fires it simulates a fill: at a price, at a time, with a cost. Then it updates the position and moves on. What comes out is a track record of something that never happened.
Two of those steps carry all the risk. What the rule was allowed to see decides whether the result is even possible, and what the fill cost decides whether it is achievable. Everything else is bookkeeping.
The three conditions
1. The rule never sees the future
Sounds obvious, leaks constantly. Using the closing price of the bar you are trading, a moving average that includes today, a dataset revised after the fact, or a stock universe that only contains companies still listed today. Each one hands the rule information it could not have had, and each one improves the result.
2. The fills are realistic
A backtest that fills at the mid, instantly, at any size, is describing a market that does not exist. Spreads widen exactly when your signal fires, size moves the price, and a stop that gaps through fills where the market opened rather than where you asked. Strategies that trade often live or die entirely on this.
3. The result is corrected for how many things you tried
This is the one that fails. Test enough rules and one clears any threshold by luck alone, so a result is only meaningful next to the number of attempts that produced it. A ten-year record with a Sharpe of 1.0 stops being statistically significant once you admit to about 31 tested variants, and a parameter sweep passes that in an afternoon.
Why a good result proves less than a bad one
A backtest is close to conclusive when it fails. If a rule loses money over twenty years of history, no amount of hope fixes that. A good result is weaker evidence, because there are many ways to produce one by accident and only one way to produce it by having an edge.
This asymmetry is the whole practical value. Use a backtest to reject, and treat what survives as a hypothesis rather than a conclusion: something that still has to clear the trial count, hold up on data you kept aside, and eventually survive real money.
How much history you actually need
More than people assume, and the arithmetic is not negotiable. A strategy with an annual Sharpe of 1.0 needs roughly 5.1 years before you can say with 95 percent confidence that it is better than nothing, assuming well-behaved returns. Real strategies are negatively skewed and fat-tailed, which roughly doubles it.
Most retail backtests run on five to ten years and report a Sharpe between 0.5 and 1.5. Set those two facts next to each other and a large share of published backtests are too short to support their own conclusion, before anyone asks how many variants were tried.
What the tools do about it
We assessed the validation machinery of 20 backtesting platforms against their own documentation. 2 correct a result for the number of variants tested. One reports a probability that the winner was selected by luck.
That is not a complaint about the tools. Running the test is your job either way, and most of it costs nothing: keep the last slice of history untouched, write down every variant you try, and compare the survivor against the holdout exactly once. Counting your attempts is the single largest improvement available to most people, and it requires no software at all.
Frequently asked questions
What is backtesting?
Running a trading rule over historical prices to see what it would have done. You define the entry, the exit and the sizing, replay the history bar by bar, and record the trades the rule would have taken. The output is a track record of something that never happened.
How does backtesting work?
The engine steps through the data in order, and at each bar it shows the rule only what would have been visible at that moment. When the rule fires, the engine simulates a fill at a price and time you have to specify, applies costs, and updates the position. The whole exercise rests on two things being honest: what the rule was allowed to see, and what the fill would really have cost.
Does backtesting actually work?
Yes, under conditions almost no tool enforces. A backtest is a valid experiment when the rule never sees the future, the fills are realistic, and the result is corrected for how many variants you tried. That last condition is the one that fails: only 2 of the 20 platforms assessed here correct for the trial count, so most backtests are reporting the best of many attempts as if it were the only one.
Is backtesting accurate?
It is exact about the past and says nothing certain about the future. The arithmetic is not where accuracy is lost. It is lost in the assumptions around it: the fills you assumed, the costs you left out, the companies missing from your data because they went bust, and the variants you tried before keeping this one.
How long does a track record have to be?
Longer than people expect. A strategy with an annual Sharpe of 1.0 needs about 5.1 years before you can say with 95 percent confidence that it is above zero, assuming well-behaved returns. Negatively skewed, fat-tailed returns, which is most real strategies, need roughly twice that.
Is backtesting worth it?
It is worth it as a filter, not as a forecast. A backtest is very good at telling you a rule does not work, and weak at telling you one does. Treat a bad result as close to conclusive and a good result as a hypothesis that still has to survive the trial count, an out-of-sample period, and eventually real money.
Where to go next
Educational content only, not investment advice.