Courses

Research Validation and Backtesting

AdvancedPremium9 sections · 21 lessons · 63 questionsLog in to track progress

Every systematic fund tests the same thing in its researcher interviews, and it is not whether you can fit a model. It is whether you can tell when a result is real. Backtests that look brilliant and lose money live, relationships that evaporate out of sample, edges that were never there: the failure modes have names, and interviewers expect you to know them.

This course builds that vocabulary and the discipline behind it: how leakage creeps into research through data, labels, features and tuning; why the standard machine-learning validation toolkit breaks on time-ordered data; and what replaces it.

The material here is the single most under-served topic in quant interview preparation. Generic ML courses teach k-fold cross-validation as if data had no arrow of time, and interview prep books stop at probability puzzles. Interviews at systematic funds do neither: they hand you a too-good backtest and ask what you would check, or take your own project apart question by question until they find the assumption you did not defend.

Each lesson introduces the failure mode, shows it with real numbers, and names the fix, so that by the end "purged cross-validation with an embargo" is not jargon but the obvious answer to a problem you have watched happen.

Most Backtests Are False PositivesWeak priors, many trials and selective reporting mean the default state of an exciting research result is wrong, before anyone has made a single mistake.Free previewThe Failure-Mode ChecklistSix named families of research failure, one checklist to run against any result, and the map of where this course treats each one.Free preview
Hypothesis-First ResearchState the mechanism, the prediction and the kill criteria before touching the data, so the data can only answer the question rather than choose it.The Data BudgetHistory is a non-renewable resource: split it into exploration, validation and a sealed holdout before the project starts, and spend each part once.
Look-Ahead and Survivorship BiasThe two classic ways a backtest trades on information that did not exist yet, and why both make strategies look better than they are.Label Overlap and Sample UniquenessWhy 2,500 overlapping observations can carry the information of 100 independent ones, and what that does to every test statistic you compute.Feature and Hyperparameter LeakageSelecting features, scaling data and tuning hyperparameters are all fits: do any of them on the full sample and the test set is no longer out of sample.
Why K-Fold Fails on Time-Ordered DataThe textbook validation tool assumes exchangeable observations. Financial data violates that three ways, and each one inflates the score.Walk-Forward and Purged Cross-ValidationThe two validation schemes that respect time: expanding walk-forward windows, and purging the training rows whose labels touch the test window.Embargoes and the Final HoldoutThe last two defences: an embargo gap that absorbs serial correlation the purge cannot see, and a holdout that is only ever read once.
How Many Trials Stand Behind a ResultThe reported strategy is the maximum of everything that was tried, and the expected maximum grows with the search even when every try is noise.Deflated and Probabilistic Sharpe RatiosTurning the trial count into arithmetic: the probability a Sharpe is real given its length and shape, and the hurdle the search size imposes.
Descriptors Before PerformanceUniverse, turnover, capacity, leverage, cost assumptions and time span: the context that decides what any performance number can possibly mean.Performance Statistics That MatterSharpe, drawdown, hit rate against payoff, and stability across time: the short list that carries information, and the decorations that do not.
Transaction Costs and the Paper SharpeSpread, fees, slippage and borrow, priced per unit of turnover: the arithmetic that decides which paper edges survive contact with execution.Impact, Capacity and TurnoverMarket impact grows with participation, which caps how much capital an edge can carry, and turnover decides how fast the cap is reached.Signal Decay and the Backtest-Live GapEdges erode through crowding, adaptation and regime change, and live performance reliably lands below the backtest. Plan for both before deployment.
From Forecast to PositionSizing turns predictions into PnL: risk targeting, conviction scaling, the cost of over-betting, and why correlated signals must share a budget.Risk-Adjusted EvaluationAccuracy is not PnL: evaluate the strategy the sizing built, in portfolio terms, against what a naive benchmark would have earned from the same inputs.
The Project-Defence RoundSystematic funds test validation by taking your own project apart, follow-up by follow-up. How the round runs, and what strong answers sound like.Twenty Questions for Your Own ProjectThe complete pre-interview checklist: run these against your project before an interviewer does, and prepare the honest answer to each.

Keep reading Research Validation and Backtesting

21 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them