In-sample vs out-of-sample data, and why a single split isn't walk-forward
Every validation test on this site rests on one boundary: data the strategy was built on, and data it was not. Get that boundary right and the tests above it mean something. Get it wrong, or draw it only once, and no amount of downstream statistics rescues the result.
What is the difference between in-sample and out-of-sample data?
In-sample data is what a strategy was built, tuned, or optimized on. Out-of-sample data is held back from that process entirely and used, once, to check the result. The split only proves anything if the boundary is real: if the "held-out" data influenced a parameter choice in any way before the final test, the out-of-sample result is not actually out-of-sample.
That last failure is the common one. It rarely looks like cheating; it looks like eyeballing a full equity chart before picking a lookback window, or re-running "just one more" variant after seeing how the holdout responded. Both quietly move the holdout into the training set.
Why one split is a weak test
A single split answers one narrow question: did this configuration, fit on this window, work on the window right after it? Shift the cut date by a month and the answer can change materially, and a single split gives you no way to see how much of the result was that one arbitrary choice rather than the strategy.
Two structures fix this, in different directions. Rolling walk-forward repeats the same split many times moving forward through history, refitting at each step, so the result is a sequence of out-of-sample periods rather than one. Combinatorial cross-validation splits the history into groups and reassembles held-out combinations into many complete paths, trading strict time order for a distribution of outcomes. Seewalk-forward vs CPCVfor what each one actually proves.
How the 20 platforms actually split it
Crossing two fields from our assessment: the spec-sheet flag for "offers walk-forward or a similar split" against the deep-audited feature that only counts a genuinely rolling, multi-window walk-forward, plus whether the platform instead runs combinatorial cross-validation. The gap between the two flags is not an error; each summary states the reason a platform lands where it does.
| Platform | Out-of-sample mechanism | Validation score |
|---|---|---|
| VectorBT | Rolling walk-forward | 8.5 |
| StrategyQuant X | Rolling walk-forward | 6.5 |
| TradeStation | Rolling walk-forward | 6.5 |
| Wealth-Lab | Rolling walk-forward | 6.5 |
| AmiBroker | Rolling walk-forward | 6.0 |
| NinjaTrader | Rolling walk-forward | 6.0 |
| QuantRocket | Rolling walk-forward | 5.0 |
| Minerva* | Combinatorial split, no sequential walk-forward | 9.5 |
| QuantConnect | Single train/test split only | 5.5 |
| MetaTrader 5 | Single train/test split only | 4.0 |
| Freqtrade | No built-in split | 4.5 |
| Backtesting.py | No built-in split | 3.5 |
| TrendSpider | No built-in split | 3.5 |
| Composer | No built-in split | 3.0 |
| Forex Tester | No built-in split | 3.0 |
| Backtrader | No built-in split | 2.5 |
| Trade Ideas | No built-in split | 2.5 |
| NautilusTrader | No built-in split | 2.0 |
| TradingView | No built-in split | 2.0 |
| Zipline Reloaded | No built-in split | 2.0 |
* Commercial relationship with this site, scored on the same published rubric as every other platform. Disclosure
The finding
Of the 20 platforms assessed, 7 run genuine rolling walk-forward: AmiBroker, NinjaTrader, QuantRocket, StrategyQuant X, TradeStation, VectorBT, Wealth-Lab. One runs combinatorial cross-validation instead of a sequential walk-forward: Minerva. 2 offer only a single, static train/test split that our rubric does not count as walk-forward: MetaTrader 5, QuantConnect. 10 ship no built-in out-of-sample mechanism at all, leaving the entire boundary to the user.
Assessed from vendor documentation and, where public, installed source, on ourpublished rubric.
Common ways a split leaks anyway
- Looking at the full chart before choosing the cut date.The boundary itself becomes a tuned parameter the moment its placement is informed by what it separates.
- Re-running after seeing the out-of-sample result. One adjustment based on the holdout's outcome, however small, folds it back into training data.
- Overlapping label horizons across the boundary. If a target is built from the next N bars, samples near the cut leak information across it unless purged. Seelook-ahead bias.
- Treating one out-of-sample pass as final. A single walk-forward window is still one path; the fix is repetition, not a bigger holdout.
Frequently asked questions
What is the difference between in-sample and out-of-sample data?
In-sample data is what a strategy was built, tuned, or optimized on. Out-of-sample data is held back from that process entirely and used, once, to check the result. The split only proves anything if the boundary is real: if the "held-out" data influenced a parameter choice in any way before the final test, the out-of-sample result is not actually out-of-sample.
Is a single train/test split the same as walk-forward?
No, and our rubric treats them as different tests. A single split gives you one path: fit once, test once. Rolling walk-forward repeats that split many times across the history, refitting at each step, which is why 7 of the 20 platforms assessed here ship it built in while 2 offer only a static, one-time split.
Why do MetaTrader 5 and QuantConnect count as "single split" rather than walk-forward?
MetaTrader 5's optimizer Forward field cuts the range once and re-runs the winning parameters on the remainder; QuantConnect enforces one reserved holdout window per organization. Both genuinely separate in-sample from out-of-sample data, and both stop there: neither rolls the split forward across multiple windows, which is what our rubric requires to count as walk-forward.
Does combinatorial cross-validation count as out-of-sample testing?
Yes, differently. Minerva runs combinatorial purged cross-validation instead of sequential walk-forward, splitting history into groups and reassembling held-out blocks into many paths. That produces genuine out-of-sample evaluation, but some training groups sit after a given test block in time, so it answers "how variable is this result" rather than walk-forward's "would this have worked run forward live."
How many platforms have no built-in split at all?
10 of the 20 assessed, leaving the entire boundary to the user: cutting a holdout, refusing to look at it, and testing the survivor once. That costs nothing but discipline, and it is the same procedure a platform with automated walk-forward runs for you.
Related
Educational content only, not investment advice. The scores above describe published and installed-source engine behaviour and are not a judgement on any specific strategy.