Back to course

Ordinary Least Squares

OLS picks the line minimising the sum of squared residuals:

The least-squares criterion
min⁡β0,β1∑i(yi−β0−β1xi)2\min_{\beta_0, \beta_1} \sum_i \left(y_i - \beta_0 - \beta_1 x_i\right)^2

Minimise the squared vertical distances, which is what makes the fitted line unique and closed-form.

Setting the derivatives to zero gives a closed form:

β^1=Cov(X,Y)Var(X)β^0=Yˉ−β^1Xˉ\hat{\beta}_1 = \frac{\text{Cov}(X,Y)}{\text{Var}(X)} \qquad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\bar{X}

The slope formula deserves a moment. It is covariance normalised by the predictor's variance, which is why it is closely related to correlation:

β^1=ρXYσYσX\hat{\beta}_1 = \rho_{XY}\frac{\sigma_Y}{\sigma_X}

Correlation is the scale-free version; the regression slope is the same relationship expressed in the units you need for a hedge. The intercept formula also guarantees the fitted line passes through (Xˉ,Yˉ)(\bar{X}, \bar{Y}).

Why squared errors

Squaring is a choice, and it has consequences worth knowing.

It gives a differentiable objective with a closed-form solution, which is the practical reason. It also makes OLS the maximum likelihood estimator when errors are normal, connecting it to the wider inference framework.

The rest of this lesson is for subscribers

Unlock every lesson in Fundamentals of Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextMultiple Linear Regression

Keep reading Fundamentals of Probability and Statistics

41 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them