Most Backtests Are False Positives

Start with the honest question: before running any test, what fraction of the strategy ideas a researcher tries are genuinely profitable after costs? In liquid, well-arbitraged markets the base rate is low. Most plausible-sounding ideas are already priced in, already decayed, or were never real. That single fact drives everything in this course.

The arithmetic of low base rates

Suppose 1 in 50 candidate ideas is real, and the validation process has an 80% chance of confirming a real edge and a 5% chance of endorsing a false one. Test 1,000 ideas and validation approves $20 \times 0.8 = 16$ real strategies and $980 \times 0.05 = 49$ false ones.

Three out of four approved strategies are false positives, and nothing went wrong: no leakage, no bugs, no bad faith. A 5% false-approval rate feels rigorous and still loses to the base rate. This is the research version of the classic diagnostic-test problem, and interviewers use it in exactly that form.

Selection makes it worse

The arithmetic above assumes every trial is reported. Research does the opposite: the losers are quietly discarded and the single best result is polished and presented. Selecting the best of many attempts inflates the winner mechanically, an effect this course quantifies later through the deflated Sharpe ratio.

What follows for practice

Two habits fall directly out of the arithmetic. First, an economic prior is part of the evidence: an idea with a defensible reason to exist (a risk premium, a structural flow, a behavioural constraint) starts from a higher base rate than a pattern found by search. Second, the number of trials behind a result is a first-class statistic, worth recording and reporting alongside the Sharpe.

The rest of the course dismantles the failure modes one family at a time. The next lesson names the taxonomy the following sections build on.