Maximum Entropy Moment Matching

Finding the maximum entropy distribution over a discrete state space subject to a given expectation (moment) constraint is a foundational problem in information theory and statistical machine learning. For states {-1, 0, 1}, we seek to maximize the Shannon entropy while ensuring the expected value matches a target mean.

Worked Example: Target Mean 0.5

Let the target mean be μ = 0.5. We want to find the probabilities p_(-1), p_0, p_1 that maximize the entropy H = - ∑ p_x ln p_x subject to E[X] = p_1 - p_(-1) = 0.5 and ∑ p_x = 1.

Calculate t = (μ + √(4 - 3μ²)) / (2(1 - μ)). For μ = 0.5: t = (0.5 + √(4 - 3(0.25))) / (2(0.5)) = (0.5 + √(3.25)) ≈ (0.5 + 1.8027756) = 2.302775637731995.
Calculate the natural parameter β = ln(t) ≈ 0.8341151943524012.
Calculate the normalization constant Z = 1/t + 1 + t ≈ 0.4342585 + 1 + 2.3027756 = 3.7370341.
Calculate the probabilities: p_(-1) = (1/t) / Z ≈ 0.11620406037800088
p_0 = 1 / Z ≈ 0.2675918792439982
p_1 = t / Z ≈ 0.6162040603780009
Check the moment: p_1 - p_(-1) = 0.6162040603780009 - 0.11620406037800088 = 0.5. (Expected result: 0.5)

Exploring the Constraint with the Interactive Module

You can interact with the Maximum Entropy Moment Matching module to see how the distribution changes with the target mean.

Set the Target mean to 0. Notice that the Maximum Entropy distribution assigns equal probability (1/3) to all three states, resulting in the maximum possible entropy of ln(3) ≈ 1.0986.
Change the Distribution to Feasible two point while keeping the Target mean at 0. Notice that it assigns 0.5 to states -1 and 1, and 0 to state 0. The entropy drops to ln(2) ≈ 0.6931. The entropy deficit is about 0.4055.
Set the Target mean to 0.5 and the Distribution to Maximum entropy. Observe the calculated probabilities and compare them with the worked example above.

Limitations and Boundary Conditions

The maximum entropy distribution takes the form p_x ∝ exp(β x). However, at the boundary where the target mean is exactly ± 1, the only feasible distribution is a point mass at that extreme state (e.g., p_1 = 1 for μ = 1). In these cases, the entropy and variance are exactly zero, and there is no finite natural parameter β (it diverges to ± ∞). The model gracefully handles these deterministic boundary states, reporting the parameter as unavailable.

References

For a foundational discussion of maximum entropy methods in machine learning, see the Stanford CS229 lecture notes: Stanford CS229 MaxEnt.

Maximum Entropy Moment Matching