GAN Optimal Discriminator JS Identity

Worked Example

Consider a discrete setting with three atoms, where the real data distributionP and the generator distributionQ are given by:

P = [3/5, 3/10, 1/10] Q = [1/10, 3/10, 3/5]

The optimal discriminator D* is derived from the ratio of the real data density to the total density:

1. Calculate the sum of probabilities at each atom:
   P + Q = [7/10, 6/10, 7/10]
2. Compute the optimal discriminator probabilities D* = P / (P + Q):
   D* = [6/7, 1/2, 1/7] ≈ [0.857, 0.500, 0.143]
3. Evaluate the maximum GAN objective V(D*) = ∑ P ln(D*) + ∑ Q ln(1 - D*):
   V(D*) ≈ -0.990
4. Calculate the Jensen-Shannon divergence using the mixture M = (P + Q)/2:
   JS(P || Q) = 0.5 * KL(P || M) + 0.5 * KL(Q || M) ≈ 0.198 nats
5. Verify the exact identity V(D*) = 2 * JS(P || Q) - ln(4):
   2 * 0.198 - 1.386 ≈ -0.990

Interactive Experiment

You can interactively explore how changes in the generated distribution affect the discriminator's objective and its connection to the Jensen-Shannon divergence using theinteractive model.

1. Select the Distributions control to explore the "Shifted", "Disjoint" or "Same" boundary cases.
2. Adjust the Overlap mixing control to continuously interpolate between the base distribution and a perfectly overlapping distribution.
3. Observe the GAN divergence identity residual, confirming it remains exactly zero across all configurations.

Limitations

The exact identity V(D*) = 2 * JS - ln(4) holds only when the discriminator is optimal over all possible functions. In practice, discriminators are constrained to a parameterized family (e.g., neural networks), meaning the empirical GAN objective is only a lower bound on the true shifted JS divergence. Furthermore, optimization dynamics and finite sample sizes cause the empirical distribution to deviate from the true continuous distribution. This simulation computes the theoretical optimum for a finite set of atoms.

References

The original formulation of Generative Adversarial Networks establishes the theoretical link between the optimal discriminator objective and the Jensen-Shannon divergence:

• Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).https://arxiv.org/abs/1406.2661