The Data Budget
Market history is a non-renewable resource. Every look at a period spends some of its evidentiary value, and no amount of cleverness gets it back: once a result has influenced a decision, that period is no longer out of sample for anything downstream of the decision. Serious research therefore budgets its data the way it budgets capital, in writing, before the project starts.
Three-way separation
The standard budget splits history into three roles.
Exploration is where ideas are formed and killed cheaply: plots, prototypes, quick fits, unlimited looks. Nothing measured here counts as evidence, and everyone involved knows it.
Validation is where surviving candidates are compared under the discipline the coming sections build: purged and embargoed cross-validation, nested tuning, uniqueness-weighted statistics. Looks here are limited and counted, because each comparison is a trial.
The rest of this lesson is for subscribers
Unlock every lesson in Research Validation and Backtesting, and every other premium course.
Subscribe to continueTest your knowledge
Keep reading Research Validation and Backtesting
21 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them