What a Model Learns

"How would you build a model to predict this?" A full answer fixes three things before it names an algorithm: what you predict, how a mistake is scored, and which set of functions you search. The algorithm comes last, because it only searches the set you chose.

The setup

Supervised learning starts from pairs $(x, y)$: features $x$ and a target $y$, drawn from some distribution. A candidate model is a function $g$ chosen from a family $\mathcal{F}$, such as all linear functions or all trees of depth three. A loss $L(y, \hat{y})$ is the cost of predicting $\hat{y}$ when the truth is $y$.

The quantity you care about is the expected risk, the average loss on new data from the same distribution:

You cannot compute it, because the distribution is unknown. You can compute the empirical risk, the average loss on the $n$ training pairs:

Training picks the $\hat{f}$ in $\mathcal{F}$ with the lowest empirical risk, possibly plus a penalty. The difference $R(\hat{f}) - \hat{R}(\hat{f})$ is the generalisation gap. It is positive on average, because $\hat{f}$ was chosen for having a low loss on exactly these $n$ points. Its training loss is an optimistic estimate of its loss on new data. That is why every model needs data it was not fitted on, and the next lesson is about the size of that gap.

The loss decides what you predict

Each loss has a best possible prediction, and it is a different summary of the distribution of $y$ given $x$:

The choice is therefore not a technical detail. It decides which quantity the model estimates.

A loss also has a statistical reading. Suppose the noise around a candidate $g$ is Gaussian with a fixed variance $\sigma^2$. The negative log-likelihood of one observation is then $(y - g(x))^2 / (2\sigma^2)$ plus a constant. Minimising squared error and maximising the likelihood give the same $\hat{f}$. In the same way, absolute error is the negative log-likelihood of Laplace noise. This is why least squares and maximum likelihood agree when the noise is Gaussian.

Signal and noise

Write the target as a signal plus noise, $y = f(x) + \varepsilon$, where $f$ is the true function and $\varepsilon$ has mean zero and variance $\sigma^2$. Under squared error, even the true $f$ has an expected loss of $\sigma^2$. The best possible out-of-sample $R^2$ is the share of the variance of $y$ that the signal explains.

For returns, that share is very small. A daily cross-sectional equity signal with an information coefficient of 0.05 is often described as good. The information coefficient (IC) is the correlation between forecast and realised return. Its $R^2$ is $0.05^2 = 0.0025$. On daily returns, a model that explains even 1% of the variance out of sample can be valuable. Three things follow.

• Training loss is mostly noise. Almost all of the variance a flexible model removes from the training loss is noise, which will not repeat.
• Evaluation needs many observations. The standard error of a sample correlation near zero is about $1 / \sqrt{n}$.
• Defaults from other fields fail. Large networks and wide hyperparameter searches work when $\sigma^2$ is small relative to the variance of the signal. Here they mostly fit noise.

Four choices before an algorithm

1. The target. A forward return over a stated horizon, its sign, its rank in the cross-section, or its volatility. Each is a different problem.
2. The loss. Choose the loss whose best prediction is the quantity you will trade on.
3. The family. The set of functions the search may return. Its size sets the trade-off between bias and variance, covered in the bias-variance decomposition.
4. The data. Which observations, how many of them are independent, and which are held back for validation.