Back to course

Kullback-Leibler Divergence

KL divergence
DKL(P∥Q)=∑xP(x)log⁡P(x)Q(x)D_{\text{KL}}(P \parallel Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)}

The cost of using Q when the truth is P. Not symmetric, which is why it is a divergence and not a distance.

The information lost when QQ is used to approximate PP. Equivalently, the extra bits needed to encode data from PP using a code optimised for QQ.

Always non-negative, and zero only when the distributions agree.

Not a distance

DKL(P∥Q)≠DKL(Q∥P)D_{\text{KL}}(P\parallel Q) \neq D_{\text{KL}}(Q\parallel P)

The asymmetry is not a defect to be worked around. It encodes a genuine difference in what is being penalised, and choosing the direction is a modelling decision.

DKL(P∥Q)D_{\text{KL}}(P\parallel Q), forward, "mean-seeking". The expectation is over PP, so wherever PP has mass and QQ does not, the penalty is severe. This forces QQ to cover all of PP, spreading out to avoid assigning near-zero probability anywhere PP lives.

The rest of this lesson is for subscribers

Unlock every lesson in Advanced Topics in Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextMutual Information

Keep reading Advanced Topics in Probability and Statistics

35 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them