Overfitting and the Train-Test Gap

A typical interview question is: "What is overfitting, and how would you know if your model is doing it?" A full answer has four parts. It gives the definition and the diagnosis. It explains why financial data makes overfitting so easy. And it gives an example from your own work.

The definition

A model overfits when it fits the noise in its training sample as well as the signal. It then performs worse on new data than its fit on the training data suggests. A more useful working version: a model overfits when a simpler model would do better out of sample.

Every sample contains both signal, which repeats in new data, and noise, which does not. A flexible enough model can fit both. The part of the fit that came from noise is lost the moment the model meets new data.

Diagnose it from two numbers

Compare the error on the data the model was trained on with the error on data it has not seen.

On returns, the same comparison appears as an in-sample Sharpe ratio of 3 that falls to 0.4 out of sample, or a training $R^2$ of 0.40 against 0.02 on validation. The gap between the two numbers is the size of the overfit.

A learning curve adds a second view: plot both errors as the training set grows. If the gap narrows as data is added, the model has high variance and more data will help. If both errors level off high and close together, the model is too simple and more data will not help.

Bias, variance and noise

For squared error, the expected error of a model's prediction on new data splits into three parts:

Bias is how far the model's average prediction is from the truth. Variance is how much the prediction changes from one training sample to another. $\sigma^2$ is the noise that no model can remove. A more flexible model lowers bias and raises variance. The next lesson, the bias-variance decomposition, derives this split and works it through by hand.

In most prediction problems outside finance, the noise term is small and flexibility pays. When predicting returns, the noise term dominates: a daily return is mostly noise, and a model with a small out-of-sample $R^2$ can be valuable. With so little signal, the variance that flexibility adds usually costs more than the bias it removes. That is the main reason simple models are hard to beat on financial data.

Many features, few observations

Overfitting is guaranteed when features are plentiful relative to observations. Fit least squares with $p$ features of pure noise, plus an intercept, to $n$ observations of a target that is also noise. The expected in-sample $R^2$ is

The noise benchmark

The most convincing test of a whole research pipeline is to run it on data that contains no signal. Shuffle the target, or replace it with random numbers, and run the same feature selection, tuning and validation. If the pipeline still reports good out-of-sample results, something in it is leaking information or selecting on noise. The noise benchmark catches problems that no single statistic does, because it tests the process rather than one model.

What reduces overfitting

• Fewer and better features, chosen before looking at the results.
• Regularisation, which shrinks coefficients towards zero; see ridge, lasso and elastic net and shrinkage estimators.
• More data, when the learning curve shows high variance.
• Validation that respects time, such as walk-forward and purged cross-validation, so that the validation error is honest.
• Early stopping for models fitted by iteration; see gradient descent and its pathologies.