Back to trackQuant Researcher
Next
Track progress145 items

Gradient Descent and Its Pathologies

"Describe gradient descent, derive the update for linear regression, and tell me what goes wrong" is a standard question at funds that use machine learning. Everyone knows least squares has a closed form, which is exactly why it is asked: the interviewer wants to see that you understand the method that takes over when the closed form is too expensive, or does not exist at all, as for logistic regression and every neural network.

The update

To minimise a loss L(β)L(\beta), start from a guess and repeatedly step against the gradient:

Gradient descent
β←β−η ∇L(β)\beta \leftarrow \beta - \eta \, \nabla L(\beta)

Step downhill by an amount proportional to the slope. The learning rate eta decides whether that step converges, crawls or diverges.

For least squares with nn observations, the loss and its gradient are

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue

Keep reading Advanced Topics in Probability and Statistics

47 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them