Feature and Hyperparameter Leakage
The first two lessons found leakage in the data and the labels. The third channel is the research process itself: every choice made while looking at the full dataset is a fit, whether or not it is called one.
Preprocessing is part of the model
Standardise features using the full sample's mean and variance, then split into train and test, and the training data has been scaled with statistics that include the test period. Each transformation is mild; the habit is not, because the same logic applies to imputation, winsorisation, PCA and every other preprocessing step. The rule that survives an interview is:
Anything fitted must be fitted on the training window only, then applied to the test window. The test set is data from the future: treat every statistic computed from it as unknowable, because live, it is.
The rest of this lesson is for subscribers
Unlock every lesson in Research Validation and Backtesting, and every other premium course.
Subscribe to continueTest your knowledge
Keep reading Research Validation and Backtesting
21 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them