Back to course

Fisher Information and the Cramer-Rao Bound

Fisher information measures how much a dataset can tell you about a parameter:

Fisher information
I(θ)=E[(∂∂θlog⁡f(X∣θ))2]=−E[∂2∂θ2log⁡f(X∣θ)]\mathcal{I}(\theta) = E\left[\left(\frac{\partial}{\partial\theta}\log f(X\mid\theta)\right)^2\right] = -E\left[\frac{\partial^2}{\partial\theta^2}\log f(X\mid\theta)\right]

The curvature of the log-likelihood, and therefore how sharply the data pins the parameter down.

The second form is the more intuitive: information is the expected curvature of the log-likelihood.

A sharply peaked log-likelihood means the data strongly distinguishes nearby parameter values, so information is high. A flat one means many values explain the data almost equally well, so information is low and the parameter is poorly determined.

The Cramér-Rao bound

For any unbiased estimator:

Var(θ^)≥1nI(θ)\text{Var}(\hat{\theta}) \geq \frac{1}{n\mathcal{I}(\theta)}

A hard floor. No unbiased estimator, however ingenious, does better. An estimator achieving it is called efficient, and MLE achieves it asymptotically.

Key takeaway

Fisher information is the curvature of the log-likelihood, and its reciprocal is the variance floor. Precision is limited by the data, not by cleverness in constructing an estimator.

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextLikelihood Ratio Tests

Keep reading Advanced Topics in Probability and Statistics

35 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them