Back to course

Applications of Entropy in Inference

The maximum entropy principle

You know some things about a distribution and not everything. Which distribution should you assume?

Maximum entropy
max⁡PH(P)subject to your known constraints\max_P H(P) \quad \text{subject to your known constraints}

A principle for choosing a distribution when the data underdetermines it, rather than a result about one.

Choose the one with maximum entropy among those consistent with what you know. Any lower-entropy distribution encodes information you do not actually have.

This is a principle of intellectual honesty made operational: assume the least while respecting the constraints.

What it produces

The results are striking, because familiar distributions turn out to be the maximum entropy answer under simple constraints.

Constraints Maximum entropy distribution
Bounded range only Uniform
Known mean, positive support Exponential
Known mean and variance Normal
Known mean, on integers Geometric

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextValue-at-Risk and Expected Shortfall

Keep reading Advanced Topics in Probability and Statistics

35 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them