Execution realism: what a backtest's fill model quietly assumes away
Two backtests can run the identical strategy over the identical data and disagree on returns because one filled every order at the exact signal price and the other charged slippage, spread and a delay for it. Execution realism is that gap: whether the engine's fill model reflects what a live order actually costs, or assumes a frictionless market that only exists inside the simulation. Unlike a statistical flaw, a weak fill model does not show up as an error. It shows up as a return that looks achievable and is not.
Where the realism actually breaks down
No fill delay, no spread, no slippage at all
In our assessment, Trade Ideas' OddsMaker replays the scanner's own alert history in order, but its own help documentation states plainly that there is no spread modelled and stop or target exits are resolved by estimating a historical price rather than simulating a fill, an estimate the guide itself warns "can be very unreliable if you set your stop loss too tight." Commission is configurable and genuinely itemised, which matters for realism, but it sits alongside a fill model with no delay and no market-impact concept of any kind.
Cost accounting without a real fill model
Composer itemises regulatory fees to the published formula (SEC, FINRA TAF, FINRA CAT, all to the exact cent) and lets slippage and a spread markup be configured, which is more cost detail than most engines bother with. But the simulation itself runs on daily adjusted closes only, and a symphony's logic is evaluated and filled at that same daily close, so the fill model has nowhere for a gap, a delay or an order-size effect to live even though the cost accounting around it is precise.
Gap-aware stops and an engine-enforced delay done right
QuantConnect's LEAN explicitly refuses to fill limit, stop, stop-limit and trailing-stop orders on the bar carrying the order's own timestamp, waiting for the next bar instead, and its equity fill model resolves a gapped-through stop at the worse of the bar's open or the stop price rather than the unreachable trigger. Layered on top is an Almgren-style permanent-plus-temporary market impact model and per-brokerage fee schedules, the kind of size-aware cost that most engines, including several with careful cost accounting elsewhere, do not attempt.
How the 20 platforms score on it
Part of our published rigour rubric: how much of a fill's real-world friction, delay, spread, slippage, gap risk and market impact, each engine actually simulates by default. 0 to 10, from vendor documentation and, where available, installed source. Not a proxy for the platform's overall rigour score, which also weighs look-ahead prevention, data handling and statistics.
| Platform | Execution score | Overall rigour |
|---|---|---|
| Minerva* | 8 / 10 | 8.5 |
| NautilusTrader | 8 / 10 | 5.8 |
| QuantConnect | 8 / 10 | 6.8 |
| QuantRocket | 7 / 10 | 6.8 |
| Backtrader | 6 / 10 | 3.8 |
| MetaTrader 5 | 6 / 10 | 5.3 |
| NinjaTrader | 6 / 10 | 5.5 |
| StrategyQuant X | 6 / 10 | 5.8 |
| TradeStation | 6 / 10 | 5.5 |
| Wealth-Lab | 6 / 10 | 5.8 |
| Zipline Reloaded | 6 / 10 | 5.5 |
| AmiBroker | 5 / 10 | 4.5 |
| Forex Tester | 5 / 10 | 4.3 |
| TradingView | 5 / 10 | 4.3 |
| VectorBT | 5 / 10 | 5.3 |
| Backtesting.py | 4 / 10 | 3.5 |
| Freqtrade | 4 / 10 | 4.0 |
| Composer | 3 / 10 | 3.3 |
| TrendSpider | 3 / 10 | 3.8 |
| Trade Ideas | 2 / 10 | 2.3 |
* Commercial relationship with this site, scored on the same published rubric as every other platform. Disclosure
The finding
The average execution score across the 20 platforms assessed is 5.5 out of 10. 4 score 7 or higher: Minerva, NautilusTrader, QuantConnect, QuantRocket. 5 score 4 or below: Backtesting.py, Composer, Freqtrade, Trade Ideas, TrendSpider. Itemised fees are common; a fill model that treats gaps, size and spread as real constraints is not, and the two do not predict each other.
Assessed from vendor documentation and, where public, installed source, on ourpublished rubric.
Four ways to test your own setup for it
- Re-run with slippage and spread set to zero, then compare.If the result changes meaningfully, the friction the engine was modelling was doing real work, and you should ask exactly what model it was using before trusting the non-zero version either.
- Force a gap through a stop and check the fill price.A fill at the exact stop level regardless of how far price gapped past it is a sign the engine is not modelling gap risk at all.
- Scale your order size up by 10x and see if anything changes.If the fill price is identical at any size, there is no market-impact model, and the strategy's apparent capacity is fiction above whatever size the live market can actually absorb.
- Read the fee schedule against your actual broker's.A flat percentage commission is a rough proxy at best; regulatory pass-through fees, tiered per-share pricing and minimums all move the real number, sometimes by more than slippage does.
Frequently asked questions
What is execution realism in a backtest?
How closely a backtest simulates the actual cost and timing of getting a trade filled: whether the fill price accounts for slippage, spread and market impact, whether a gapped-through stop fills at a reachable price, and whether commissions are itemised rather than assumed away. A strategy can pass every statistical test and still be unrealistic here.
Is slippage the same thing as market impact?
No. Slippage is the gap between a quoted price and the price you actually get, usually modelled as a fixed tick or percentage penalty applied to every fill. Market impact is the cost your own order size adds to that gap: a large order moves the price against itself. Most engines model slippage; almost none model impact, because it needs a size-aware model, not a constant.
What happens when a stop order gaps through its trigger price?
A realistic engine fills the order at the next reachable price, typically the bar’s open, rather than at the stop price itself, because the stop price was never actually tradable once the market gapped past it. An engine that fills at the stop price regardless is quietly assuming liquidity that was not there.
Does a good commission model make an engine execution-realistic?
On its own, no. Itemised regulatory fees and per-share commissions are the easiest part of execution to get right because they are published schedules, not simulation. An engine can itemise fees to the cent and still fill every order at a same-bar price with no slippage, spread or impact modelling at all.
How many platforms model execution realistically by default?
Of the 20 platforms assessed on our rubric, 4 score 7 or higher out of 10 on execution and 5 score 4 or below. The gap tracks whether the fill model treats size and gaps as real constraints, not how polished the platform otherwise looks.
Related
Educational content only, not investment advice. The scores above describe published and installed-source engine behaviour and are not a judgement on any specific strategy.