Kullback Leibler Divergence

Worked Example: Computing KL Divergence

Suppose we have two unnormalized integer weight vectors over 3 categories:
P weights = [2, 1, 1]
Q weights = [1, 2, 1]

First, we normalize them to get valid probability distributions:

1. Sum of P weights = 2 + 1 + 1 = 4. Normalized P = [0.5, 0.25, 0.25]
2. Sum of Q weights = 1 + 2 + 1 = 4. Normalized Q = [0.25, 0.5, 0.25]

The relative entropy of P with respect to Q, denoted D(P||Q), is calculated as:
D(P||Q) = Σ p_i · log₂(p_i / q_i)

One category term is negative here, but the total divergence is nonnegative. The reverse divergence D(Q||P) is also 0.25 bits in this specific symmetric case, but in general D(P||Q) ≠ D(Q||P), showing that KL Divergence is asymmetric and not a true metric.

Interactive Experiment

Explore the interactive KL Divergence module here: KL Divergence Visualizer.

In this module, you can modify the unnormalized weights of two 3-category distributions P and Q.

1. Observe Asymmetry: Set P weight 1 to 2, P weight 2 to 2, P weight 3 to 0. Then set Q weight 1 to 1, Q weight 2 to 3, Q weight 3 to 0. You will see that D(P||Q) is ~0.2075 bits, while D(Q||P) is ~0.1887 bits. This clearly demonstrates the asymmetry of KL Divergence.
2. Zero Probabilities: Set P weight 1 to 1, P weight 2 to 0, P weight 3 to 3. Then set Q weight 1 to 1, Q weight 2 to 1, Q weight 3 to 1. D(P||Q) is finite. However, D(Q||P) becomes Positive Infinity because Q has non-zero probability for category 2, while P has zero probability there. A missing reference support creates infinity.
3. Identical Distributions: Set P and Q to have the same weights. Notice that both divergences drop to 0 bits.

Limitations

The interactive visualization makes several simplifying assumptions:

• Small State Space: It limits distributions to just 3 categories. In real ML scenarios, state spaces are often continuous or extremely large discrete sets, requiring advanced integration or summation techniques.
• Unnormalized Integers: Inputs are restricted to integer weights [0, 4] for simplicity, whereas real-world probabilities are arbitrary real numbers in [0, 1].
• Lack of Smoothing: A zero numerator probability contributes 0, including when both probabilities are zero. A positive numerator probability with a zero reference probability contributes positive infinity. The module preserves these exact support conventions without adding an arbitrary epsilon.

References

For a deeper mathematical understanding of KL divergence, entropy, and related information-theoretic concepts, consult the primary academic sources listed below. They provide rigorous proofs and definitions.

• Information Theory: From Coding to Learning (Polyanskiy & Wu) - Definition 2.1, zero/support conventions, Definition 3.1.