Probability of backtest overfitting (PBO)
Every other test on this site judges a strategy. PBO judges your selection procedure. It asks whether choosing the configuration that looked best in-sample is a sensible way to choose at all, and the answer is frequently no.
The idea
You have a set of candidate configurations. You pick the one with the best in-sample score. The question PBO answers is: across many different ways of cutting the history, how often does that winner turn out to be below median on the data you held out?
If the answer is rarely, your search is finding something real. If the answer is about half the time, your winner carries no information: you would have done as well picking a configuration at random. If the answer is more than half, your procedure is actively anti-predictive, which happens more often than people expect, because the most-tuned configuration is often the most fragile one.
How the estimate is built
- Cut the performance series of every candidate into an even number of equal blocks.
- Take every combination of half those blocks as the in-sample set, the rest as out-of-sample.
- In each combination, find the candidate with the best in-sample score.
- Look up where that same candidate ranks out-of-sample, as a percentile.
- PBO is the fraction of combinations where that percentile falls below the median.
Purging matters here as much as anywhere: if the blocks share information through overlapping label horizons, the out-of-sample ranking is contaminated and PBO comes back too optimistic.
Reading the number
| PBO | What it says about your search |
|---|---|
| Under 10% | The in-sample winner is usually genuinely better. The search is informative. |
| 10% to 30% | Usable, with the usual caveats. Worth checking whether the margin over the second-best candidate is meaningful. |
| 30% to 50% | Weak. Your winner is close to a coin flip against the rest of the field. |
| Above 50% | The procedure is anti-predictive. Picking the in-sample best is worse than picking at random, and no amount of further tuning fixes that. |
Bands as commonly applied following Bailey, Borwein, Lopez de Prado and Zhu. They are conventions for reading the statistic, not thresholds with a formal derivation.
Who reports it
Exactly one of the 20 platforms assessed reports a PBO figure: Minerva. It is the rarest capability on our rubric, and also the one that speaks most directly to the question every backtest raises.
Assessed from vendor documentation on ourpublished rubric. A number you could compute yourself in a notebook does not count as shipped, which is the standard applied to every platform here.
If your tool does not report one
You need the per-candidate performance series, not just the summary numbers, which most backtesters will export. Split into an even number of blocks, run the combinations, and count. With eight blocks you get seventy combinations, which is enough to see the shape.
A cheaper approximation, if that is too much: keep your candidates ranked in-sample, then check the rank correlation between in-sample and out-of-sample performance across a handful of different cuts. If that correlation hovers around zero, your ranking carries no signal and you already have your answer without computing PBO properly.
Frequently asked questions
What does PBO measure in one sentence?
The probability that the configuration you selected because it was best in-sample performs below median out-of-sample, estimated by splitting the history combinatorially and checking how often that happens.
What is a good PBO?
Low, and the scale is blunt on purpose. Under 10 percent is strong, 10 to 30 percent is a result worth a second look, and above 50 percent means your selection procedure is worse than picking at random, because the winner is more often than not a below-median configuration. Treat it as a traffic light, not a score to optimise.
How is PBO different from a deflated Sharpe?
They attack the same problem from opposite ends. The deflated Sharpe adjusts one number for the search that produced it. PBO evaluates the search procedure itself, asking whether picking the in-sample winner is a good way to pick at all. A strategy set can produce an acceptable deflated Sharpe and still have a terrible PBO, which tells you the selection step is the broken part.
Does PBO need CPCV?
In practice yes: the standard estimator, combinatorially symmetric cross-validation, is built on splitting the history into groups and reassembling held-out blocks into many paths. That is why the two features travel together. Of the 20 platforms assessed here, 2 ship purged or combinatorial cross-validation and 1 report a PBO figure from it.
Can I estimate it myself?
Yes, and it is the most instructive afternoon you will spend on validation. Split your history into an even number of equal blocks, take every combination of half of them as in-sample and the complement as out-of-sample, record which configuration wins in-sample and where that configuration ranks out-of-sample, and count the fraction of splits where it lands in the bottom half. That fraction is your PBO.
Related
Educational content only, not investment advice.