Backtesting Forecasts
A backtest asks whether a model would have worked. It is indispensable and it is the single most misleading tool in quantitative finance, because it is easy to construct one that is wrong in a flattering direction.
The four lies
Lookahead bias. Using information not available at the time. Obvious versions are easy to avoid; subtle ones are not. Using a day's closing price in a decision made during that day, using restated fundamental data, or using an index constituent list as of today.
Survivorship bias. Testing on instruments that still exist excludes those that failed. Any long strategy looks better on a universe from which the bankruptcies have been removed.
Ignoring costs. Spreads, fees, market impact and borrow costs. A strategy trading frequently can be comfortably profitable gross and reliably unprofitable net, as the statistical versus practical significance lesson shows.
The rest of this lesson is for subscribers
Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.
Subscribe to continueTest your knowledge
Keep reading Advanced Topics in Probability and Statistics
35 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them