Statistical validity of a backtest, and what it actually takes
A backtest can model every fill correctly and still report a number that is not real, because the number was found by trying many things and keeping the best one. Statistical validity is the question that catches that: not whether the simulation was accurate, but whether the result survives once the search that produced it is accounted for.
What does statistical validity mean for a backtest?
It means the reported result survives once you account for how it was found: a track record long enough to not be noise, an honest count of every variant tried, and a check that the winning configuration was not simply the luckiest one in the search. A backtest can be executed with a perfect fill model and still fail this, because execution accuracy and statistical honesty are separate problems.
It is the fourth of our four rigour axes, alongside look-ahead control, execution realism and data correctness, and it is the one that judges the verdict itself rather than the simulation that produced it.
The four things it actually requires
A genuine out-of-sample split
Data the strategy never touched during development, held apart, and only evaluated once. Financial samples overlap in time, so an ordinary train/test split still leaks unless the boundary is purged and embargoed. Seewalk-forward vs CPCV.
A counted, corrected number of trials
Test enough configurations and one clears any threshold by chance alone. The correction has to use the real count, including the attempts you would rather forget. Seehaircutting a Sharpe.
Deflation for track length and shape
A short record needs a much higher Sharpe to mean the same thing as a long one, and skew and kurtosis change the answer further. Seethe deflated Sharpe ratioand minimum track record length.
A formal check on the selection step itself
Everything above can pass and the procedure that picked the winner can still be worthless. A direct test of that procedure is the strongest defence available. Seeprobability of backtest overfitting.
How the 20 platforms score on it
The statistical-validity axis of our published rigour rubric, 0 to 10, assessed from vendor documentation and, where public, installed source. The band is our own scale for what a score actually means, derived from the same assessed number, not a separate judgement call per row.
| Platform | Statistics axis | Band |
|---|---|---|
| Minerva* | 9 / 10 | Full battery: CPCV, deflation, PBO |
| AmiBroker | 5 / 10 | Built-in walk-forward or Monte Carlo |
| QuantRocket | 5 / 10 | Built-in walk-forward or Monte Carlo |
| StrategyQuant X | 5 / 10 | Built-in walk-forward or Monte Carlo |
| TradeStation | 5 / 10 | Built-in walk-forward or Monte Carlo |
| VectorBT | 5 / 10 | Built-in walk-forward or Monte Carlo |
| Wealth-Lab | 5 / 10 | Built-in walk-forward or Monte Carlo |
| MetaTrader 5 | 4 / 10 | In/out-of-sample split, no correction |
| NinjaTrader | 4 / 10 | In/out-of-sample split, no correction |
| QuantConnect | 4 / 10 | In/out-of-sample split, no correction |
| Freqtrade | 3 / 10 | In/out-of-sample split, no correction |
| TrendSpider | 3 / 10 | In/out-of-sample split, no correction |
| Backtesting.py | 2 / 10 | Summary stats only |
| Backtrader | 2 / 10 | Summary stats only |
| Composer | 2 / 10 | Summary stats only |
| Forex Tester | 2 / 10 | Summary stats only |
| NautilusTrader | 2 / 10 | Summary stats only |
| TradingView | 2 / 10 | Summary stats only |
| Zipline Reloaded | 2 / 10 | Summary stats only |
| Trade Ideas | 1 / 10 | Summary stats only |
* Commercial relationship with this site, scored on the same published rubric as every other platform. Disclosure
The finding
The average statistics-axis score across the 20 platforms assessed is 3.6 out of 10. 1 score 7 or higher, the band that requires walk-forward plus a formal defence such as purged CV, a corrected multiple-testing bar, deflated Sharpe, or a PBO figure: Minerva. 6 sit in the middle band, shipping built-in walk-forward or Monte Carlo alone with no correction layered on top. 13 score 4 or below, leaving the whole problem to the user: Backtesting.py, Backtrader, Composer, Forex Tester, Freqtrade, MetaTrader 5, NautilusTrader, NinjaTrader, QuantConnect, Trade Ideas, TradingView, TrendSpider, Zipline Reloaded.
Assessed from vendor documentation and, where public, installed source, on ourpublished rubric.
Checking your own setup in five steps
- Write down every configuration you tried, including the ones you abandoned in the first five minutes. This number is the input every correction below needs and the one people undercount by habit.
- Confirm the out-of-sample data was never seen during tuning,not just never used in the final backtest run. A parameter chosen after eyeballing the full history has already leaked, whatever split is applied afterward.
- Apply a trial-count correction to the Sharpe, Bonferroni if you want the conservative answer with the least machinery.
- Check the record is long enough for the corrected Sharpe to mean anything,using the minimum-track-record-length calculation rather than a fixed rule of thumb.
- If you can, estimate a PBO figure on the same candidate set. It is the one check here that judges the selection procedure directly instead of by proxy.
Frequently asked questions
What does statistical validity mean for a backtest?
It means the reported result survives once you account for how it was found: a track record long enough to not be noise, an honest count of every variant tried, and a check that the winning configuration was not simply the luckiest one in the search. A backtest can be executed with a perfect fill model and still fail this, because execution accuracy and statistical honesty are separate problems.
How is this different from execution realism?
Execution realism asks whether each simulated trade filled the way a real trade would have, given slippage, gaps and liquidity. Statistical validity asks a question one level up: given the fills were realistic, is the resulting performance figure real, or is it the best of many attempts dressed up as one result? A backtest can pass one and fail the other.
What is the minimum a backtest needs to clear this bar?
Two things, honestly done: a true out-of-sample period the strategy never touched during development, and a written-down count of every configuration tried before landing on the one reported. Most of the failures on this axis trace back to one of those two being skipped, not to a missing advanced technique.
Can a backtest be statistically valid and still lose money live?
Yes. Statistical validity establishes that the number is real, not that the edge it measured will persist. A genuinely significant result can still be measuring a regime that ends. It rules out the specific failure of mistaking search luck for skill; it says nothing about regime change, which is a different risk entirely.
How many of the 20 platforms score well on this axis?
1 of the 20 platforms assessed score 7 or higher out of 10 on statistical validity: Minerva. 13 score 4 or below, meaning the platform ships summary statistics and little else, leaving every correction to the user.
Related
Educational content only, not investment advice. The scores above describe published and installed-source engine behaviour and are not a judgement on any specific strategy.