Back to course

Logistic Regression and Cross-Entropy

This lesson answers two questions about logistic regression: "derive the gradient of the loss" and "why not use squared error for classification?" Both have short, exact answers. The lesson then covers what goes wrong when the model is used on returns.

The model

Logistic regression models the probability that a binary label is 1. This lesson writes the weights as ww and the intercept as bb, where the regression lessons write β\beta:

p(x)=σ(w⊤x+b)σ(z)=11+e−z\begin{aligned} p(x) &= \sigma(w^\top x + b) \\ \sigma(z) &= \frac{1}{1 + e^{-z}} \end{aligned}

Taking the inverse of σ\sigma shows what is linear: the log-odds,

log⁡p1−p=w⊤x+b\log \frac{p}{1 - p} = w^\top x + b

A one-unit increase in xjx_j adds wjw_j to the log-odds, so it multiplies the odds by ewje^{w_j}. It is not a fixed change in probability, which depends on where pp starts.

The rest of this lesson is for subscribers

Unlock every lesson in Machine Learning for Quantitative Research, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue

Keep reading Machine Learning for Quantitative Research

27 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them