Back to course

Maximum Likelihood Estimation: Deep Dive

Choose the parameters that make the observed data most probable:

The log-likelihood
ℓ(θ)=∑i=1nlog⁡f(xi∣θ),θ^=arg⁡max⁡θℓ(θ)\ell(\theta) = \sum_{i=1}^n \log f(x_i \mid \theta), \qquad \hat{\theta} = \arg\max_\theta \ell(\theta)

Sum the logs and maximise. Taking logs turns a product into a sum and makes the arithmetic tractable.

The log turns a product into a sum, which is numerically stable and differentiable term by term. Maximising the log is equivalent since log is monotonic.

Why it is the default

Consistency. θ^→θ0\hat{\theta} \to \theta_0 as nn grows.

Asymptotic normality.

n(θ^−θ0)→dN(0,I(θ0)−1)\sqrt{n}(\hat{\theta} - \theta_0) \xrightarrow{d} \mathcal{N}\left(0, \mathcal{I}(\theta_0)^{-1}\right)

which hands you standard errors and confidence intervals from the curvature of the log-likelihood, at no extra work.

Asymptotic efficiency. No consistent estimator has lower asymptotic variance. MLE achieves the Cramér-Rao bound.

Invariance. If θ^\hat{\theta} is the MLE of θ\theta, then g(θ^)g(\hat{\theta}) is the MLE of g(θ)g(\theta). Estimate a variance and the MLE of the standard deviation is its square root, with no separate derivation.

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextFisher Information and the Cramer-Rao Bound

Keep reading Advanced Topics in Probability and Statistics

35 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them