When is a backtest luck?
Try enough strategies and one of them will look brilliant. How good does the best of N backtests have to be before it is evidence rather than selection?
Each strategy earns Gaussian daily returns with zero mean and 1% volatility over T trading days. One strategy can be given a real edge. The winner's Sharpe ratio is compared with the best you should expect from N tries of pure noise:
Here V is the variance of the N Sharpe estimates and γ ≈ 0.577 is the Euler–Mascheroni constant (Bailey & López de Prado, 2014). The deflated Sharpe ratio is the probabilistic Sharpe ratio of the winner measured against SR₀ instead of zero, allowing for the skewness γ̂₃ and kurtosis γ̂₄ of its returns:
Helix applies this test as a gate; its combined book scored a deflated Sharpe ratio of 0.21 over four trials and failed.
At N = 1 the two tests agree. Raise N: the winner’s Sharpe ratio rises with it while its DSR does not. Then press Repeat to count how often each test would have called pure noise significant, and plant a real edge of 1.0 to see how often noise still beats it.
| Simulated | Theory or threshold | |
|---|---|---|
| Winner's Sharpe ratio, annualised | — | SR₀ = — |
| Average winner over 2,000 repeats | run Repeat | the formula above |
| PSR against zero | — | significant if > 0.95 |
| DSR against SR₀ | — | significant if > 0.95 |
| Noise called significant, PSR · DSR | run Repeat | share of 2,000 repeats |
| The real edge won the selection | no real edge | share of 2,000 repeats |
output/lab/exp01_overfitting.py33 lines · numpy, scipy
"""EXP-01 · Backtest overfitting: the best of N strategies with no edge, and its deflated Sharpe ratio."""
import numpy as np
from scipy import stats
rng = np.random.default_rng(7)
N, years, vol = 200, 3, 0.01 # strategies tried, sample length, daily volatility
T = 252 * years
R = rng.normal(0.0, vol, size=(N, T)) # daily returns; every true Sharpe ratio is zero
sr = R.mean(1) / R.std(1, ddof=1) # per-day Sharpe ratio of each strategy
best = int(sr.argmax())
# Expected maximum of N Sharpe estimates under H0 (Bailey & López de Prado, 2014)
g = np.euler_gamma
V = sr.var(ddof=1)
sr0 = np.sqrt(V) * ((1 - g) * stats.norm.ppf(1 - 1 / N) + g * stats.norm.ppf(1 - 1 / (N * np.e)))
def psr(sr_hat, sr_star, r):
"""Probabilistic Sharpe ratio: P(true SR > sr_star), allowing for skew and fat tails."""
g3, g4 = stats.skew(r), stats.kurtosis(r, fisher=False)
z = (sr_hat - sr_star) * np.sqrt(len(r) - 1) / np.sqrt(1 - g3 * sr_hat + (g4 - 1) / 4 * sr_hat**2)
return stats.norm.cdf(z)
ann = np.sqrt(252)
print(f"best annualised SR {sr[best] * ann:5.2f}")
print(f"E[max SR] under H0 {sr0 * ann:5.2f}")
print(f"PSR vs 0 (naive) {psr(sr[best], 0, R[best]):5.3f}")
print(f"DSR (deflated) {psr(sr[best], sr0, R[best]):5.3f}")
# Monte Carlo check of E[max]: under H0 each SR estimate is close to N(0, 1/T)
mx = rng.normal(0, 1 / np.sqrt(T), size=(5000, N)).max(1)
theory = (1 / np.sqrt(T)) * ((1 - g) * stats.norm.ppf(1 - 1 / N) + g * stats.norm.ppf(1 - 1 / (N * np.e)))
print(f"E[max SR]: simulated {mx.mean() * ann:.3f} vs formula {theory * ann:.3f}")
The same model as the page, with its own random draws, so the numbers agree in distribution rather than digit for digit.