Back to course

Shannon Entropy

Shannon entropy
H(X)=−∑ipilog⁡2piH(X) = -\sum_i p_i \log_2 p_i

Average surprise, in bits. Largest when everything is equally likely and zero when the outcome is certain.

Entropy measures uncertainty, in bits: the average number of yes/no questions needed to determine the outcome.

Entropy is a statement about the shape above. Push p to either extreme and the mass concentrates, so there is little left to be surprised by; hold it near a half and the distribution is as spread as it can be, which is exactly where entropy peaks.

The extremes

A fair coin has H=1H = 1 bit. Maximum uncertainty for two outcomes, and exactly one question needed.

A coin landing heads with probability 0.9 has H≈0.47H \approx 0.47 bits. Less uncertain, so less information conveyed by learning the result.

A certain outcome has H=0H = 0. Nothing to learn.

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextKullback-Leibler Divergence

Keep reading Advanced Topics in Probability and Statistics

35 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them