Back to course

Advanced Regression Techniques

OLS is unbiased and, under Gauss-Markov, the minimum-variance linear unbiased estimator. The techniques here mostly abandon unbiasedness on purpose, because total error is what matters and it decomposes as

The bias-variance decomposition
Error=Bias2+Variance+Irreducible noise\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible noise}

Prediction error splits three ways, and only two of them are yours to trade against each other.

Accepting a little bias to remove a lot of variance is often a large net win, and with correlated predictors it usually is.

Ridge: shrink everything

Loss=∑i(yi−y^i)2+λ∑jβj2\text{Loss} = \sum_i (y_i - \hat{y}_i)^2 + \lambda\sum_j \beta_j^2

The L2L_2 penalty pulls coefficients toward zero without reaching it. λ\lambda controls the strength: zero recovers OLS, large λ\lambda shrinks everything toward nothing.

Ridge is the standard answer to multicollinearity. When two factors are nearly collinear, OLS produces wild offsetting coefficients; ridge splits the effect between them and stabilises both.

The rest of this lesson is for subscribers

Unlock every lesson in Fundamentals of Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Finish course

Keep reading Fundamentals of Probability and Statistics

41 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in all seven courses, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them