Back to course

How Many Trials Stand Behind a Result

Every result this course has examined so far was weakened by the same unreported number: how many things were tried before this one was shown. This lesson makes that number the object of study.

The expected maximum

Let each trial's performance statistic be noise: mean zero, standard deviation σ\sigma. The expected value of the best of NN independent trials grows like

E[max⁡]≈σ2ln⁡NE[\max] \approx \sigma \sqrt{2 \ln N}

The right-hand side is an upper-bound approximation; the accurate expected best runs a little lower, near 1.5σ1.5\sigma at N=10N = 10, 2.5σ2.5\sigma at N=100N = 100, and 3.2σ3.2\sigma at N=1,000N = 1{,}000. The growth is slow but relentless, and nothing needs to be real for the best-of-search to look excellent; the search size alone manufactures it. A researcher who reports the winner without the count has removed the single number needed to interpret the result.

The rest of this lesson is for subscribers

Unlock every lesson in Research Validation and Backtesting, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextDeflated and Probabilistic Sharpe Ratios

Keep reading Research Validation and Backtesting

21 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them