Jensen–Shannon Divergence

Worked Example: Computing JS Divergence

Consider a setup where distribution P is Point A (100% mass on category 1) and Q is Point C (100% mass on category 3). We want to compute the JS divergence for a mixture weight w = 0.5.

P = [1.0, 0.0, 0.0, 0.0]
Q = [0.0, 0.0, 1.0, 0.0]

1. First, compute the mixture distribution M = 0.5 * P + 0.5 * Q:
   M = [0.5, 0.0, 0.5, 0.0]
2. Next, compute the KL divergence from P to M, skipping zero probabilities:
   KL(P || M) = 1.0 * ln(1.0 / 0.5) = ln(2) ≈ 0.693 nats
3. Compute the KL divergence from Q to M:
   KL(Q || M) = 1.0 * ln(1.0 / 0.5) = ln(2) ≈ 0.693 nats
4. Compute the JS divergence:
   JS = 0.5 * KL(P || M) + 0.5 * KL(Q || M) = 0.5 * ln(2) + 0.5 * ln(2) = ln(2) ≈ 0.693 nats

The divergence in bits is JS / ln(2) = 1.0 bit. The binary entropy upper bound H(0.5) is also ln(2) nats. Since the distributions are disjoint, they attain this upper bound perfectly.

Interactive Experiment

Explore the interactive Jensen-Shannon Divergence module here: JS Divergence Visualizer.

In this module, you can select specific distribution shapes and change the mixture weight.

1. Uniform vs Point: Set P to "Point A", Q to "Uniform", and Mixture weight to "0.5". The divergence decreases to ~0.318 nats, showing that overlapping distributions have a divergence strictly less than the maximum possible entropy bound (0.693 nats).
2. Asymmetric Weighting: Keep the same distributions (Point A and Point C) but change the Mixture weight to "0.25". Observe that the divergence drops to ~0.562 nats, aligning with the new lower entropy bound H(0.25). Note that at unequal weights, JS divergence is neither symmetric nor a metric.
3. Zero Weight Edges: Change the Mixture weight to "0". The JS divergence becomes exactly 0, even though the unused KL term (from P to M, since M = Q) would theoretically be infinite due to the missing support in Q. The algorithm explicitly handles this by skipping zero-weight terms.

Limitations

The interactive visualization makes several simplifying assumptions:

• Restricted State Space: The visualization models exactly four categorical bins. In practice, distributions can span continuous spaces, requiring integration rather than summation.
• Finite Selections: Only a few specific distribution shapes are provided to highlight bounds and limits cleanly. It does not contain an arbitrary parameterizer for every possible categorical distribution.
• Fixed Controls: Controls only allow jumping to predefined mixture weights instead of a continuous slider, avoiding noisy float edge cases and ensuring exact evaluations at boundaries like 0 and 1.

References

For further mathematical background on JS divergence and its use in information theory and machine learning:

• Jianhua Lin, "Divergence measures based on the Shannon entropy" - Foundational paper detailing the properties, symmetry bounds, and metric qualities of JS divergence.