Skip to main content
Status of this page: P0 draft. Statistical formulas are stated in plain text; numerical examples were checked by calculation but depend on assumptions listed in each example.

1. Taxonomy

2. Multiple testing

Harvey, Liu and Zhu (2016) argue that, given the extent of data mining in factor research, the usual t-ratio of 2.0 is not an adequate criterion and a newly proposed factor should clear a hurdle above 3.0. The principle carries over to strategy research inside a firm: the more variants tried, the higher the bar.

3. Deflated Sharpe ratio (DSR)

The standard error of an estimated Sharpe ratio widens with skewness and fat tails and shrinks with sample length. DSR compares the observed Sharpe to the expected maximum Sharpe of N unskilled trials, then asks how likely the observed value is to exceed it given the non-normality-adjusted standard error. Illustration (verified numerically; assumptions stated).
  • Daily returns, T = 756 days, annualised Sharpe 1.5, normal returns: the standard error of the annualised Sharpe is about 0.58.
  • If the standard deviation of annualised Sharpe across trials is 0.5, the expected best Sharpe from pure noise is about 0.79 for 10 trials, 1.27 for 100 trials, and 1.63 for 1,000 trials.
So a Sharpe of 1.5 found among 1,000 tries is not distinguishable from the best of noise under these assumptions. The trial count and cross-trial dispersion are inputs the researcher must record; if they are not logged, DSR cannot be computed honestly.

4. Probability of backtest overfitting (PBO)

Using combinatorial symmetric cross-validation (CSCV), split the performance matrix (time by strategy-variant) into blocks, choose the best variant in-sample on half the blocks, and check its rank out-of-sample on the rest across all combinations. PBO is the fraction of splits where the in-sample best ranks below the out-of-sample median. A PBO well above 0 means selection is mostly choosing noise.

5. Purged and embargoed cross-validation

If labels span a window (for example, forward 5-day return), ordinary k-fold leaks. Purging removes training observations whose label window overlaps a test observation’s label window. Embargo also drops a buffer of observations after the test block to cover serial correlation beyond the label horizon. Combinatorial purged cross-validation extends this to produce many backtest paths and hence a distribution of performance instead of one number. Reference: Lopez de Prado, Advances in Financial Machine Learning (2018).

6. Research hygiene checklist

  1. Log every variant tried (count and parameters).
  2. Fix universe, data cut, and cost model before looking at results.
  3. Use point-in-time data and record the availability lag.
  4. Report DSR or an equivalent haircut, and PBO if a selection step exists.
  5. Hold out a final period that is touched once.
  6. Compare to a naive baseline under the same costs.
See also performance and attribution and event-driven vs vectorized backtests.

Points to verify

  • Exact formulas for DSR (Bailey and Lopez de Prado, 2014) and PBO (Bailey, Borwein, Lopez de Prado and Zhu, 2017) before implementing.
  • Embargo percentage conventions vary by implementation; there is no universal value.

Sources

  • Harvey, Liu and Zhu (2016), and the Cross-Section of Expected Returns, RFS 29(1). NBER
  • Lopez de Prado (2018), Advances in Financial Machine Learning, Wiley; description of purging and embargo summarised from public references such as the Purged cross-validation article. Link
  • Bailey and Lopez de Prado (2014), The Deflated Sharpe Ratio, Journal of Portfolio Management.
  • Bailey, Borwein, Lopez de Prado and Zhu (2017), The Probability of Backtest Overfitting, Journal of Computational Finance.