Gaussian Product of Experts

Theory

Given two normalized Gaussian experts, A and B, defined by means \mu_A, \mu_B and variances \sigma_A², \sigma_B².

A mixture distribution with weight α for expert A is defined as: p_(mix)(x) = α · N(x; \mu_A, \sigma_A²) + (1-α) · N(x; \mu_B, \sigma_B²)

The mean of the mixture is \mu_(mix) = α\mu_A + (1-α)\mu_B, and its variance incorporates both within-expert variance and between-expert variance: \sigma_(mix)² = α\sigma_A² + (1-α)\sigma_B² + α(1-α)(\mu_A - \mu_B)²

A product of experts multiplies the unnormalized densities and then divides by a normalization constant Z (overlap normalization). The resulting distribution is exactly Gaussian: p_(prod)(x) = 1/Z N(x; \mu_A, \sigma_A²) N(x; \mu_B, \sigma_B²) = N(x; \mu_(prod), \sigma_(prod)²)

The precision (inverse variance) of the product is the sum of precisions: 1/(\sigma_(prod)²) = 1/(\sigma_A²) + 1/(\sigma_B²). The product mean is precision-weighted: \mu_(prod) = ((\mu_A)/(\sigma_A²) + (\mu_B)/(\sigma_B²)) \sigma_(prod)². The overlap normalization constant Z is equivalent to N(\mu_A; \mu_B, \sigma_A² + \sigma_B²).

Worked Example

Consider Disagree experts: Expert A has \mu_A = -1, \sigma_A = 1, and Expert B has \mu_B = 1, \sigma_B = 1. We evaluate a 50/50 mixture (α = 0.5) and the product of experts.

For the Mixture: Mean = 0.5(-1) + 0.5(1) = 0. Variance = 0.5(1) + 0.5(1) + 0.25(2)^2 = 1 + 1 = 2.

For the Product of Experts: Precision = 1/1 + 1/1 = 2. Variance = 1/2 = 0.5. Mean = (-1/1 + 1/1) * 0.5 = 0.

Evaluating raw densities at probe x=0: p_A(0) ≈ 0.24197 and p_B(0) ≈ 0.24197.

1. Mixture mean is 0.0 with variance 2.0.
2. Product mean is 0.0 with variance 0.5.
3. Overlap normalization Z = N(-1; 1, 1+1) = N(-1; 1, 2) ≈ 0.103777.
4. Raw density product at 0 is 0.24197 × 0.24197 ≈ 0.05855.
5. Normalized product density at 0 is 0.05855 / 0.103777 ≈ 0.56419.

Experiment

You can interact with different sets of experts in this module: Gaussian Product of Experts.

1. Set Experts to Disagree, Mixture weight A to 0.5, and Probe x to 0.
2. Observe that the Product mean is exactly 0.0 with Product variance 0.5, while the Mixture mean is 0.0 with Mixture variance 2.0. The product is sharper.
3. Set Experts to Agree. Notice that the Mixture variance becomes 1.0 but the Product variance becomes 0.5.
4. Set Experts to Unequal. Notice the product shifts toward the narrower expert.

Limitations

While the product of experts creates surprisingly sharp predictions by combining distinct views, there are notable pitfalls:

• Overconfidence: As shown when multiplying identical experts, the product assumes independent evidence. If the experts share dependencies or prior information, simply taking their product artificially inflates precision, making the model overly confident.
• Zero Overlap: If experts completely disagree and their variances are tiny, the overlap Z tends to zero. This leads to numerical instability, as the unnormalized product vanishes everywhere.
• Not A True Bayesian Posterior: Although mathematical PoE shares similarities with likelihood multiplication in Bayes' rule, without explicitly modeling the shared prior and dependencies, it cannot be strictly treated as a unified Bayesian posterior.

References

The Product of Experts framework highlights how simpler sub-models can be combined to form a complex, highly sharp overall density.

• Hinton, G. E. (1999). Training Products of Experts by Minimizing Contrastive Divergence.