Why 10,000 backtests produce a fake genius
A researcher runs ten thousand parameter combinations, keeps the one with the best curve, and presents it as "the strategy". The result is not a discovery. It is the maximum of ten thousand random draws, dressed as skill.
The arithmetic of trying again
If a strategy has no edge at all, its Sharpe ratio is still not zero in any finite sample — it is a random number with a spread. Try enough variations and one of them will look excellent by luck alone. The more you search, the better your best result looks, and the less it means.
This is why a backtest is not evidence on its own. Evidence is a result that survives contact with data the search never touched.
What actually protects you
Out-of-sample and walk-forward
Fit on one period, evaluate on the next, roll forward. Any transformation — normalisation, scaling, feature selection — must be fitted on the training window only. Fitting on the whole history and then "testing" on a slice is leakage with extra steps.
Parameter stability
Nudge every parameter by ±10%. A real edge degrades smoothly. An overfit one falls off a cliff, because the optimum was a spike created by noise.
Deflated Sharpe and PBO
The deflated Sharpe ratio adjusts for the number of trials, the length of the sample, and the skew and kurtosis of returns. The probability of backtest overfitting (PBO) estimates how often the "best" configuration underperforms the median one out of sample. Both are ways of asking: given how hard you searched, is this result surprising?
Fewer, simpler parameters
A strategy with two parameters and an economic rationale is worth more than one with eleven parameters and a story written after the fact.
What an audit can and cannot see
Adwa Auditor sees the code and the data you give it. It can prove the logic is causal, that it reproduces, and that the returns are not fabricated or copied from the dataset. It can flag implausible performance — a Sharpe above 8, a 95%+ win rate over 100+ trades.
It cannot know how many variations you tried before this one. That number is yours to disclose. An honest audit states its limits; a certificate that claims more than it can prove is worth less than no certificate at all.