How to verify a backtest before you trust it
Someone sends you an equity curve that goes up and to the right. It might be a real edge. It might be a strategy that read tomorrow's newspaper. From the chart alone the two are indistinguishable — which is why every serious allocator checks the code, not the picture.
The four questions that actually matter
1. Could the strategy have known this at the time?
Lookahead bias is any use of information that did not exist when the decision was made:
a shift(-1), an iloc[i+1], a normalisation using the mean of the whole
series, or a fundamentals file keyed to the reporting period instead of the publication date.
It is the single most common reason a beautiful backtest dies in production.
2. Was the result selected from many attempts?
If the author tried 4,000 variations and showed you the best one, the Sharpe ratio is not a measurement — it is the maximum of 4,000 draws. That is overfitting, and it has a signature: performance that collapses the moment the parameters are nudged, or the sample is split.
3. Does it survive its own assumptions?
Slippage, fees, latency, partial fills, and position sizing turn a paper edge into a real one or destroy it. A strategy with no execution model is a spreadsheet, not a system.
4. Is the result reproducible from the code?
Run the same file on the same data twice and get the same numbers. If it drifts, at least one of the two runs was lying — often because of unseeded randomness or a hidden dependency on wall-clock time.
How Adwa Auditor answers them
You upload the strategy file and the dataset. The code runs in an isolated container with no network and a read-only filesystem, and the audit reports:
- Static integrity scan — negative shifts, future indexing, whole-series statistics that leak the future into per-row decisions.
- Statistical vetoes — a Sharpe above 8, a win rate above 95% on 100+ trades, or output that is just the dataset's own returns re-labelled.
- Reproducibility — the same strategy re-run under three seeds must produce the same return series.
- Execution realism — whether orders, fills and positions actually exist in the code, or the returns came from nowhere.
You get a verdict and a run hash. Anyone can paste that hash into
/api/v1/verify/<hash> and confirm the audit was not edited afterwards.
Adwa does not tell you whether a strategy makes money. It tells you whether the number is real. Those are different questions, and only the second one can be automated honestly.
What a verdict means
- PASS — the logic is causal, reproducible, and consistent with the data.
- PASS_WITH_RISK — sound, but with documented limitations (no execution model, thin sample, weak regime coverage).
- REJECT — integrity is intact but the measured quality is poor.
- CODE_ERROR / INPUT_ERROR — we could not evaluate it, and we say so instead of guessing.