Why backtests beat live performance, and how to measure the gap before you trade
A backtest beats live performance when it is simultaneously too kind to execution (unrealistic fills, slippage, latency) and too generous about overfitting (how many parameter combinations were tried before this one was kept). Live trading forces a strategy to confront both weaknesses at once; a backtest only confronts whichever one its engine happens to model well.
Which failure mode is actually bigger, bad fills or overfitting?
Neither dominates universally. Of the 20 platforms assessed on this site, only2 score 7 or higher out of 10 on overfitting protection, and neither of those two can place a live trade at all. Among the platforms that do trade live, the strongest overfitting guard tops out at 6.5, a full tier below the two backtest-only engines.
That split matters because the two gaps compound rather than cancel. A platform that models fills honestly but lets a search loop run unchecked still inflates its headline result, through the parameter sweep rather than the fill. A platform that guards the search loop but fills every order at a clean, same-bar price inflates it the other way, throughexecutioninstead. Either one alone is enough to produce a number live trading will not repeat.
Does realistic execution modelling make up for weak overfitting protection?
No. Of the 4 platforms whose fill model scores 7 or higher for execution realism, Minerva is the only one that also scores 7 or higher on overfitting protection, and it has no live trading. The other 3, NautilusTrader, QuantConnect, QuantRocket, all trade live and all score 5.5 or below on overfitting protection. Crossing rigour.axes.execution against validation.score and specs.live_trading across all 20platforms is a table this site has not run before.
| Platform | Live trading | Execution realism | Overfitting protection | Profile |
|---|---|---|---|---|
| Minerva* | No | 8.0 | 9.5 | Realistic fills and strong overfitting protection |
| QuantConnect | Yes | 8.0 | 5.5 | Realistic fills, weak overfitting protection |
| QuantRocket | Yes | 7.0 | 5.0 | Realistic fills, weak overfitting protection |
| NautilusTrader | Yes | 8.0 | 2.0 | Realistic fills, weak overfitting protection |
| VectorBT | No | 5.0 | 8.5 | Strong overfitting protection, unrealistic fills |
| StrategyQuant X | No | 6.0 | 6.5 | Neither scores 7 or higher |
| TradeStation | Yes | 6.0 | 6.5 | Neither scores 7 or higher |
| Wealth-Lab | Yes | 6.0 | 6.5 | Neither scores 7 or higher |
| NinjaTrader | Yes | 6.0 | 6.0 | Neither scores 7 or higher |
| AmiBroker | Yes | 5.0 | 6.0 | Neither scores 7 or higher |
| MetaTrader 5 | Yes | 6.0 | 4.0 | Neither scores 7 or higher |
| Backtrader | Yes | 6.0 | 2.5 | Neither scores 7 or higher |
| Freqtrade | Yes | 4.0 | 4.5 | Neither scores 7 or higher |
| Forex Tester | No | 5.0 | 3.0 | Neither scores 7 or higher |
| Zipline Reloaded | No | 6.0 | 2.0 | Neither scores 7 or higher |
| Backtesting.py | No | 4.0 | 3.5 | Neither scores 7 or higher |
| TradingView | Yes | 5.0 | 2.0 | Neither scores 7 or higher |
| TrendSpider | Yes | 3.0 | 3.5 | Neither scores 7 or higher |
| Composer | Yes | 3.0 | 3.0 | Neither scores 7 or higher |
| Trade Ideas | Yes | 2.0 | 2.5 | Neither scores 7 or higher |
* Commercial relationship with this site, scored on the same published rubric as every other platform. Disclosure
The finding
Minerva is the only platform assessed here that scores 7 or higher on both execution realism and overfitting protection, and it has live trading. VectorBT, the only other platform to score 7 or higher on overfitting protection alone, likewise cannot place a live order. Meanwhile NautilusTrader, QuantConnect, QuantRocket model fills realistically (execution realism 7 or higher) but each score 5.5 or below on guarding their own search against overfitting, and all 3 support live trading. On this site's own measurements, the platforms doing the most to prevent a backtest from lying to itself are the ones that never have to find out live whether it was lying.
Assessed from vendor documentation and, where public, installed source, on ourpublished rubric.
How do I measure my own backtest's live-performance gap before I trade it?
Score your own setup on the two axes separately: whether your simulator models slippage, spread and gap-through fills the way a real market would, and whether you fixed the parameter count and held out data before tuning rather than after. A strategy can fail on either axis alone and still disappoint live, so check both, not just the one your platform happens to report.
- Audit the fill model first. Does it charge slippage, does a stop that gaps through fill at the open rather than the stop price, and does size ever cap the fill? See execution realism for the specific things to check.
- Count how many variants were actually tried. A grid search run in the background is still a search, even if only the best result is shown. Seeparameter optimization and overfitting.
- Confirm the out-of-sample boundary was real. Data the optimizer never touched, tested once. Seein-sample vs out-of-sample.
- Paper-trade the exact configuration before funding it. A short live-data, no-capital run is the cheapest way to see the execution gap directly, before either kind of overstatement costs real money.
Frequently asked questions
Why does a backtest usually outperform the same strategy traded live?
A backtest beats live performance when it is simultaneously too kind to execution (unrealistic fills, slippage, latency) and too generous about overfitting (how many parameter combinations were tried before this one was kept). Live trading forces a strategy to confront both weaknesses at once; a backtest only confronts whichever one its engine happens to model well.
Which failure mode is actually bigger, bad fills or overfitting?
Neither dominates universally. Of the 20 platforms assessed on this site, only 2 score 7 or higher out of 10 on overfitting protection, and neither of those two can place a live trade at all. Among the platforms that do trade live, the strongest overfitting guard tops out at 6.5, a full tier below the two backtest-only engines.
Does realistic execution modelling make up for weak overfitting protection?
No. Of the 4 platforms whose fill model scores 7 or higher for execution realism, Minerva is the only one that also scores 7 or higher on overfitting protection, and it has no live trading. The other 3, NautilusTrader, QuantConnect, QuantRocket, all trade live and all score 5.5 or below on overfitting protection.
If my platform scores well on one axis, can I skip checking the other?
No. The two measure different failure modes and neither substitutes for the other. A fill model can be scrupulously realistic about slippage and gaps while the strategy behind it was still picked by an unguarded sweep of a thousand parameter combinations, and a platform that guards hard against overfitting can still assume every order fills at the exact close with no slippage at all.
How do I measure my own backtest’s live-performance gap before I trade it?
Score your own setup on the two axes separately: whether your simulator models slippage, spread and gap-through fills the way a real market would, and whether you fixed the parameter count and held out data before tuning rather than after. A strategy can fail on either axis alone and still disappoint live, so check both, not just the one your platform happens to report.
Related
Educational content only, not investment advice. The scores above describe published and installed-source engine behaviour and are not a judgement on any specific strategy.