One question, reported by a candidate at a major trading firm, captures an entire category of quant researcher interviews: you backtested a strategy, it did very well, and in live trading you keep losing money. What do you do?
There is no formula to recall and no trick to spot. The question tests whether you understand why research results go wrong, and candidates report it, in one form or another, at several major systematic funds. It is also the least prepared topic in quant interviews: LeetCode does not cover it, the classic brainteaser books stop at probability, and generic machine learning courses teach cross-validation as if data had no arrow of time. This guide covers the question territory candidates report, through the strongest reported phrasings and our composites of the rest, and the shape of the answers interviewers reward.
Why firms test this
A researcher who ships an overfit signal costs the firm real money twice: the losses, and the weeks of discovering that the edge was never there. Firms therefore filter for the specific scepticism that catches false results before deployment. The interview formats follow from that goal: conceptual questions like the one above, methodology follow-ups on your own projects, and take-home exercises, in at least one reported case graded on a hidden test set. What firms do not do, according to the evidence across 27 firms, is hand you someone else's flawed backtest to critique: the material they interrogate is yours.
The questions, and what a strong answer sounds like
“Your backtest looks great. Why don't you trust it?”
What it testswhether you have a systematic way to doubt a result, or just a vague sense that overfitting exists.
A composite of the methodology probes candidates report. The strong answer is a checklist run in order: was every field knowable at decision time (look-ahead bias), was the universe defined as of each date (survivorship bias), do the labels overlap and did the statistics use the honest sample size, was every fitted step fitted inside training data only, how many variants were tried before this one, and what do costs at realistic size do to the edge. Naming the families and the order, data first, economics last, is what separates a researcher from a reciter.
“Why can't you shuffle time-series data for k-fold cross-validation?”
What it testswhether your validation knowledge comes from ML courses or from working with market data.
A recurring reported theme at large systematic funds. Three reasons: serial dependence puts near-copies of test days into training; overlapping forward-looking labels share price paths across the split, so the model is graded on moves it trained on; and random folds mix every regime into both sides, testing interpolation when live trading is extrapolation. The follow-up is "so what do you do instead," and the expected vocabulary is walk-forward evaluation and purged, embargoed cross-validation. Our course derives all three mechanisms, and you can implement the splitter yourself as a coding problem.
“What is survivorship bias, and where does it enter a backtest?”
What it testswhether you reach for the mechanical fix or a vague be-careful.
Through the universe: backtesting today's index constituents removes every firm that failed along the way, and long-only strategies gain phantom return on the order of percent per year. The fix is a point-in-time universe. Look-ahead bias is the same family through timestamps: earnings carrying the quarter they describe rather than the day they were announced. The words interviewers listen for are point-in-time data and publication lags, because those are the fixes a production pipeline actually implements.
“What is overfitting? Give an example from your own work.”
What it testsfirst-hand experience. The definition is a warm-up; the example is the question.
The definition is one sentence: fitting the noise in a sample rather than the process that generated it, so performance evaporates out of sample. What distinguishes candidates is the example. A concrete confession (tuning a lookback against the same validation set until it "worked", selecting features on the full sample) with the fix you now apply (nested validation, a sealed holdout, a trial log) demonstrates the scar tissue the interviewer is actually screening for. A candidate who has never caught themselves overfitting has not done much research.
“How many things did you try?”
What it testswhether you count your own experiments, which almost nobody does.
The multiple-testing question, usually aimed at your own project. The best of ( N ) worthless strategy variants is expected to look like a two-sigma discovery at ( N = 40 ), so a reported Sharpe without a trial count is an unfinished sentence. Strong candidates volunteer the count, mention that selection inflates the winner, and ideally know the deflated Sharpe ratio exists for exactly this arithmetic.
“Gross Sharpe 2 at 60x annual turnover. Is it tradable?”
What it testscost-per-turnover instinct, the fastest sanity check in the business.
The arithmetic is one multiplication: 60 round trips a year at even 10 basis points each is 12% of the book in costs before market impact, which consumes most paper edges whole. The strong answer prices the cost stack per round trip, multiplies by turnover, and then makes the deeper point: for high-turnover strategies the cost model is not an adjustment to the result, it usually is the result, and capacity falls as turnover rises. The course lesson carries the worked version.
“Walk me through your project. What would kill it?”
What it testswhether you can see your own work the way a sceptical reviewer would.
The project defence, weighted heavily for PhD candidates and reported across several major funds. The counterintuitive rule: volunteering your project's flaws with fixes scores better than defending everything, because the round exists to discover whether you know where your own result is weakest. The project-defence lesson walks the escalation with strong and weak answers side by side.
Every question above is a variation on one theme: given how this result was produced, how much should anyone believe it? Prepare the theme, not the phrasings.
Rapid-fire vocabulary checks
Shorter versions of the same territory turn up as one-line probes, often dropped into an unrelated discussion to see whether the vocabulary is native or rehearsed. One confident sentence each is enough; each term links to the course lesson that builds it properly.
| Term | The one-sentence answer |
|---|---|
| Purged and embargoed CV | Contiguous time blocks, drop training samples whose label windows overlap the test block, and an extra buffer after it for serial correlation the labels cannot see. |
| Walk-forward vs cross-validation | Walk-forward only ever trains on the past, so it is the honest track-record simulation; purged CV is the efficient model-comparison instrument. |
| Effective sample size | Overlapping 20-day labels sampled daily carry about one twentieth the independent information of their row count, and every t-statistic should use the smaller number. |
| Deflated Sharpe ratio | The probability your Sharpe beats the best that pure luck would produce, given how many trials stood behind it. |
| The sealed holdout | A recent period no experiment may touch, opened once on the finished strategy; a holdout consulted twice is a validation set. |
| Alpha decay | Edges erode through crowding, adaptation and regime change, so live performance is planned below the backtest, not at it. |
| Regime change vs broken model | An edge can be real and still stop working when the environment that funded it moves; pre-committed review triggers beat post-hoc debates. |
| Capacity | Market impact grows with the square root of participation, so every strategy has a size at which impact consumes the edge, and it shrinks as turnover rises. |
The backtest-versus-live question, answered properly
Return to the opening question. A strong answer works through, in roughly this order: check for leakage first, because look-ahead or survivorship bias can manufacture a backtest from nothing; then the honest sample size, because overlapping labels overstate confidence; then selection, because the deployed strategy won a comparison and regression to the mean is built into deployment; then costs and impact at the size actually traded; and only then market explanations like crowding and regime change. Ending with what you would do next, cut size, pre-committed review triggers, and a decision about whether the mechanism still exists, turns a diagnostic answer into an operator's answer.
Preparing this topic
The material is genuinely learnable in a couple of weeks because it is one connected argument rather than a list of facts. The Research Validation and Backtesting course builds it in order, from the base rates of false research results through leakage, time-series cross-validation, multiple testing and costs, to a twenty-question checklist for defending your own project. The coding problems turn the concepts into implementations interviewers reportedly ask for:
And the researcher interview process guide shows where in each firm's process this topic appears, so you know whether it arrives as a phone-screen concept question or a six-hour take-home.