Expectation-Maximization for Mixtures

Visualize Expectation-Maximization for a two-Gaussian mixture model on one-dimensional data.

Expectation-Maximization for Mixtures

Concept Overview

The Expectation-Maximization (EM) algorithm is an iterative method to find local maximum likelihood estimates of parameters in statistical models with unobserved latent variables. In this module, we fit a two-component Gaussian mixture model to one-dimensional data.

Mathematical Definition

We consider a fixed set of one-dimensional data points [-2, -1, 1, 2] and a two-Gaussian mixture model. The mixture weights are fixed at 0.5 each, and both components share a fixed known variance v. The component means are constrained to be symmetric: -m and +m.

In the E-step, we compute the responsibility of the positive component for each data point x_i:

r_i = 1 / (1 + exp(-2 * m * x_i / v))

In the M-step, we update the positive mean m using the responsibilities:

m = sum(r_i * x_i) / sum(r_i)

Worked Example

Let's verify the first iteration step manually with the following initial state:

Initial mean magnitude is 1
Known variance is 1
EM iterations is 1

For x = -2, the responsibility r_0 = 1 / (1 + exp(-2 * 1 * (-2) / 1)) = 1 / (1 + exp(4)) ≈ 0.017986. For x=1: 1 / (1 + exp(-2)) ≈ 0.880797. However, our algorithm does multiple points: r_0 ≈ 0.017986, r_1 ≈ 0.119203, r_2 ≈ 0.880797, r_3 ≈ 0.982014. Summing these responsibilities gives exactly 2.0. Summing r_i * x_i: 0.017986(-2) + 0.119203(-1) + 0.880797(1) + 0.982014(2) ≈ 2.689649. Updating the mean m = 2.689649 / 2 ≈ 1.344825. These rounded working values explain the mean; the implementation retains full precision. The displayed responsibilities are recomputed at the updated mean, approximately [0.004590, 0.063587, 0.936413, 0.995410], rather than the E-step responsibilities used to produce it. The summed log mixture density rises from -7.158187 to -6.855904.

Experiment

Follow these steps using the Expectation-Maximization for Mixtures controls to explore convergence:

Set Initial mean magnitude to 1, and Known variance to 1. Note the initial log-likelihood.
Set EM iterations to 1. Observe how the means shift towards the outer data points.
Set EM iterations to 2. The positive mean updates to 1.427233, improving the likelihood further.
Set Known variance to 4 and EM iterations to 1. The positive mean becomes 0.584576; the variance was held at 4 during this replay.
Set Initial mean magnitude to 0, Known variance to 1 and EM iterations to 5. Every responsibility stays 0.5, every history mean stays zero, and log-likelihood stays -8.675754.
Set Initial mean magnitude to 2, Known variance to 0.25 and EM iterations to 0. No update is performed: the means remain -2 and 2, and current and initial log-likelihood agree.

Limitations

This implementation is constrained to a mean-only update for a symmetric two-component mixture with fixed weights and a fixed shared variance. Unconstrained covariance and weight estimation are not included in this educational model. Furthermore, The usual EM ascent property does not guarantee a global optimum, nor does this finite 0–10 iteration prefix prove convergence. A stationary point need not be a local maximum. The responsibilities are latent component probabilities under the fitted mixture; they are not probabilities that a mean parameter is true. Setting the initial mean to exactly 0 demonstrates a coincident stationary state from which the algorithm cannot escape, regardless of iterations.

References

Learn more about GMMs and Expectation Maximization in statistical models:

scikit-learn: Gaussian mixture models

Expectation-Maximization for Mixtures