Bernoulli Natural Gradient

When training probabilistic models, the parameter space is often better measured using information geometry rather than standard Euclidean distance. For a Bernoulli parameter p, this module compares the ordinary gradient descent update with the natural gradient step.

Worked Example

Consider a Bernoulli model with initial probability p = 0.5 and soft target q = 0.25. We want to take a step size eta = 0.5.

Calculate Cross Entropy Loss:
L = -q * ln(p) - (1-q) * ln(1-p) = -0.25 * ln(0.5) - 0.75 * ln(0.5) = -ln(0.5) ≈ 0.6931
Calculate Ordinary Gradient:
dL/dp = (p - q) / (p * (1 - p)) = (0.5 - 0.25) / (0.5 * 0.5) = 0.25 / 0.25 = 1
Calculate Ordinary Raw Probability Update:
p_new_ordinary = p - eta * dL/dp = 0.5 - 0.5 * 1 = 0
Calculate Fisher Information:
I(p) = 1 / (p * (1 - p)) = 1 / (0.5 * 0.5) = 4
Calculate Natural Gradient Update:
p_new_natural = p - eta * (p - q) = 0.5 - 0.5 * (0.5 - 0.25) = 0.5 - 0.125 = 0.375

In this example, both steps stay within the valid probability domain, but the natural gradient scales the update using the Fisher information matrix to navigate the curved probability space more effectively. This provides a substantive visible explanation for the calculation.

Experiment

Interact with the module at https://riano.app/module/bernoulli-natural-gradient to see how gradient scaling affects the probability update.

Set Initial probability to 0.1.
Set Target probability to 1.
Set Step size to 0.5.
Observe that the ordinary raw update jumps to 5.1, moving far outside the valid probability domain.
Notice the natural gradient step keeps the new probability at a valid 0.55.

This experiment demonstrates the stability of natural gradient updates.

Limitations and Assumptions

This simulation uses the exact closed decimal grid and scaled integer fractions to classify true endpoint updates. However, it's important to note that parameterization invariance describes the infinitesimal natural-gradient field, not exact equality of arbitrary finite Euler steps in p and logit coordinates. Large step sizes (like eta = 1) can still push parameters to exact boundary limits where cross-entropy becomes zero or mathematically unavailable. These assumptions form the boundaries of the model's predictive power.

References

The following external sources support the formulation of the natural gradient.

Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation. - Explains the fundamental theoretical basis for the natural gradient direction.
Martens, J. (2014). New Insights and Perspectives on the Natural Gradient Method. arXiv:1412.1193. - Provides a deep dive into the perspective of using Fisher information.

These external references are fundamental for understanding natural gradients and Fisher information matrices as applied in this visualization module.

Bernoulli Natural Gradient