The Bias-Variance Decomposition

"Derive the bias-variance decomposition." Writing down the result is the easy part. A complete answer derives it, says where each assumption is used, and applies it to a real estimate. This lesson does all three.

The setup

The target is a fixed function plus noise, $y = f(x) + \varepsilon$, with $E[\varepsilon] = 0$ and $\text{Var}(\varepsilon) = \sigma^2$. A model $\hat{f}$ is fitted on a random training set $D$. Fix a point $x$ and draw a new $y$ there. The noise in that new $y$ is independent of $D$.

There are two sources of randomness: the training set, which makes $\hat{f}(x)$ random, and the noise in the new $y$. The expected squared error averages over both.

The derivation

Write $\bar{f}(x) = E_D[\hat{f}(x)]$ for the average prediction over training sets, and drop the $x$ to keep the lines short. Add and subtract $f$ and $\bar{f}$:

Square and take expectations. The three squared terms give the result. Each cross term is zero, for a stated reason:

• $E[\varepsilon \,(f - \bar{f})] = 0$, because $E[\varepsilon] = 0$ and $f - \bar{f}$ is a constant.
• $E[\varepsilon \,(\bar{f} - \hat{f})] = 0$, because $\varepsilon$ is independent of $D$, so the expectation factors into $E[\varepsilon] \, E[\bar{f} - \hat{f}] = 0$.
• $E[(f - \bar{f})(\bar{f} - \hat{f})] = 0$, because $f - \bar{f}$ is a constant and $E_D[\hat{f}] = \bar{f}$ by definition.

What remains is

The three terms are the irreducible noise, the squared bias and the variance. The decomposition holds at each point $x$. The overall error averages it over the distribution of $x$.

The result has two limits. The clean additive split is a property of squared error: for 0-1 loss there is no decomposition of this form. And the derivation assumes that new data comes from the same distribution as the training data. A drifting market adds error that none of the three terms describes.

A worked case: shrinking a mean

The same algebra applies to any estimator, and the smallest example is the most useful one in finance.

The optimal factor $c^*$ depends on $\mu$, which is the unknown. In practice the amount of shrinkage comes from a prior or from cross-validation, which is the subject of shrinkage estimators and empirical Bayes and of ridge, lasso and elastic net. The lesson of the example stands: when the noise is large next to the signal, accepting bias to cut variance lowers the error.

What capacity does to each term

As a model family grows, training error falls steadily. Bias falls too, because the family can get closer to $f$. Variance rises, because the fit follows the particular noise in each training set. The test error is their sum plus $\sigma^2$, so it typically falls and then rises: the familiar U-shaped curve.

Two common knobs show this directly:

• Nearest neighbours. Averaging the targets of the $m$ nearest neighbours gives a variance of $\sigma^2 / m$. A larger $m$ lowers variance and raises bias, because the neighbours are further from $x$.
• Averaging models. Bagging averages models fitted on resampled data. It leaves bias nearly unchanged and lowers variance, which is why it helps unstable models such as deep trees. See decision trees and bagging.

Very large models can show a second descent: test error can fall again past the point where the model fits the training data exactly. This "double descent" is documented for large networks and for minimum-norm ("ridgeless") linear regression. Whether it helps on a few thousand noisy daily observations is contested. Kelly, Malamud and Zhou, in "The Virtue of Complexity in Return Prediction", argue that it does for return prediction. Treat it as a claim to test on your own data, not as a default.