Walk-forward vs combinatorial purged cross-validation
Most comparisons rank these two, with CPCV winning. That framing is wrong. They answer different questions, and the one you need depends on what you are about to do with the answer.
What each one actually proves
Walk-forward
Fit on a window, test on the window immediately after it, roll forward, repeat. Every test period sits strictly after the data used to fit it, so the sequence mirrors how the strategy would have been run in real time.
It proves: the rule would have held up if you had run it forward, with the parameters you would actually have had at each point.
Its weakness: you get one path. Shift the start date by a month and the result can change materially, and you have no way to see how much of what you are looking at was that choice.
Combinatorial purged cross-validation
Split the history into groups, hold out several combinations of them, purge the training samples whose outcome windows overlap the held-out blocks, embargo a margin after, and reassemble the held-out pieces into many complete backtest paths.
It proves: how much of your result was the particular slicing of history. Instead of one Sharpe you get a distribution, and the spread is the finding.
Its weakness, rarely stated: in a combinatorial split, some training groups sit after the test block in time. The result is not a simulation of trading forward. That is a deliberate trade, made to buy many paths, but it means CPCV does not answer the question walk-forward answers.
What the platforms ship
Of 20 platforms assessed, 7 offer built-in walk-forward and 2 offer purged or combinatorial cross-validation. One ships both: VectorBT. 12 ship neither, leaving the entire question to you.
Assessed from each vendor's own documentation on the rubric published on ourmethodology page. A capability you could write yourself does not count as shipped.
Who offers which
| Platform | Walk-forward | Purged CV | Validation |
|---|---|---|---|
| Minerva* | No | Yes | 9.5 |
| VectorBT | Yes | Yes | 8.5 |
| StrategyQuant X | Yes | No | 6.5 |
| TradeStation | Yes | No | 6.5 |
| Wealth-Lab | Yes | No | 6.5 |
| AmiBroker | Yes | No | 6.0 |
| NinjaTrader | Yes | No | 6.0 |
| QuantRocket | Yes | No | 5.0 |
* Commercial relationship with this site, scored on the same published rubric as every other platform. Disclosure
How to choose, in practice
- Deciding whether to trade something: walk-forward. You want the version of the question that matches what you are about to do.
- Comparing many candidate rules: purged cross-validation, then a correction for how many you compared. The spread across paths is what separates a robust rule from a lucky slice.
- Labels with overlapping horizons: purging is not optional. If your target is built from the next N bars, an ordinary split leaks by construction, and the leak flatters you.
- Either way: neither test knows how many variants you tried. That correction is separate, and it is the onealmost nobody ships.
Doing it without a platform
Walk-forward needs no library. Pick a training window and a test window, step through the history, refit at each step using only data available at that point, and keep the test results end to end. The two mistakes that make it worthless are refitting on data that includes the test window, and tuning the window sizes until the walk-forward result improves, which turns the walk-forward itself into the thing you overfitted.
Frequently asked questions
Is CPCV strictly better than walk-forward?
No, and treating it that way is the common mistake. CPCV gives you many backtest paths instead of one, which makes the result far less dependent on where you happened to cut the data. But its combinatorial splits mean some training sets sit after the test block in time. That is fine for the question it answers, which is how variable is this result across possible histories, and it is not the question walk-forward answers, which is would this have worked if run forward in real time.
Can I just use walk-forward and skip the rest?
Walk-forward is the minimum bar, not the whole exam. It catches a rule tuned to the entire history, but it says nothing about how many rules you tried. Of the 20 platforms assessed here, 7 offer walk-forward and 2 correct for the trial count. Walk-forwarding two hundred variants and keeping the best leaves you with the same selection problem you started with.
What does purging actually remove?
Any training sample whose outcome window overlaps the test period. If a label is built from the next twenty bars, a training sample five bars before the test set already contains information from inside it. Purging drops those. An embargo drops a further margin after the test block, because serial correlation leaks in the other direction too.
How many splits does CPCV need?
It is set by how you group the data rather than chosen freely: with N groups and k held out per combination you get N choose k splits and a fixed number of reconstructed paths. More groups gives more paths and more compute. The point is not a magic number, it is that you get a distribution of outcomes rather than a single number to fall in love with.
Which should I run first?
Walk-forward, because it is simpler to get right and its failures are easier to read. If a rule cannot survive a plain rolling out-of-sample test, nothing more elaborate will rescue it. Reach for purged cross-validation once your labels have horizons that overlap, which is where an ordinary split silently leaks.
Related
Educational content only, not investment advice.