The Failure-Mode Checklist
Interviewers rarely ask "recite the failure modes of quantitative research." They hand you a result that looks wonderful and ask what you would check. The difference between a scattered answer and a strong one is a taxonomy: named families, checked in order, each with a question that exposes it. This lesson is that taxonomy, and the rest of the course treats each family in depth.
The six families
1. Data leakage. Information from the future reaches decisions made in the past. The probing questions: was every field knowable at decision time (look-ahead), and was the universe defined as of each date (survivorship)?
2. Label problems. Overlapping forward-looking labels inflate the apparent sample size and leak across train/test splits. The question: how many independent observations are there really, and did every statistic use that number?
3. Fitted preprocessing. Scaling, imputation, feature selection and hyperparameter tuning are all fits. The question: was every fitted step fitted inside training data alone, and how many configurations were tried?
4. Invalid validation. Random k-fold on time-ordered data grades models on interpolation they will never get to do live. The question: did the validation scheme respect time, at label level, with purge and embargo?
5. Multiple testing. The best of many trials is large even when every trial is worthless. The question: how many experiments stand behind this one reported number?
6. Economics blindness. A paper edge that ignores costs, impact, capacity and decay is a description of the past, not a plan. The question: what happens to the Sharpe after realistic costs at intended size?
Order matters. Leakage and label problems can manufacture a result from nothing, so they are checked first; costs and capacity only modify a result that survived the earlier checks. Working the list backwards wastes effort polishing numbers that a purge would have deleted.
Running the checklist
A colleague shows a daily equity strategy with a Sharpe of 3.1. In order: the fundamental fields turn out to be as-restated rather than as-published (family 1); the 10-day labels were sampled daily and split randomly (families 2 and 4); features were selected on the full sample (family 3); this was the 34th variant tried (family 5); and costs were assumed at one-tenth of the realistic spread for the small caps it trades (family 6). Each finding alone would be disqualifying. Finding all six in one artefact is not unusual, which is the lesson.
Where the course goes from here
Families 1 through 3 are the Leakage section. Family 4 is Validation on Time-Ordered Data. Family 5 gets its own section on multiple testing, and family 6 spans the sections on reading a backtest and taking a strategy from paper to live. The closing section turns the whole checklist inward, onto the project you will defend in an interview.
Test your knowledge
Keep reading Research Validation and Backtesting
21 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them