Adam Moment Bias Correction
Inspect biased moving moments and the correction that changes Adam's early steps.
By Riano · Published · Updated · Report a correction
Adam Moment Bias Correction
The Adam optimizer computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients. However, because these moving averages are initialized as zero vectors, they are biased towards zero, especially during the initial timesteps. This bias correction step addresses this issue.
Worked Example
Let us trace the updates with a Constant gradient stream of g = 2.0, Beta 1 = 0.9, Beta 2 = 0.999, Step size = 0.1, up to t = 2 updates.
- Initialization: m0 = 0, v0 = 0, x = 0
- Update t = 1:
- Gradient g1 = 2.0
- m1 = 0.9 * 0 + 0.1 * 2.0 = 0.2
- v1 = 0.999 * 0 + 0.001 * (2.0)2 = 0.004
- mHat1 = 0.2 / (1 - 0.91) = 2.0
- vHat1 = 0.004 / (1 - 0.9991) = 4.0
- xCorrected1 = 0 - 0.1 * 2.0 / (sqrt(4.0) + 1e-8) ≈ -0.1
- xUncorrected1 = 0 - 0.1 * 0.2 / (sqrt(0.004) + 1e-8) ≈ -0.316
- Update t = 2:
- Gradient g2 = 2.0
- m2 = 0.9 * 0.2 + 0.1 * 2.0 = 0.38
- v2 = 0.999 * 0.004 + 0.001 * (2.0)2 = 0.007996
- mHat2 = 0.38 / (1 - 0.92) = 0.38 / 0.19 = 2.0
- vHat2 = 0.007996 / (1 - 0.9992) = 0.007996 / 0.001999 = 4.000000000000057
- xCorrected2 = -0.1 - 0.1 * 2.0 / (sqrt(4.0) + 1e-8) = -0.2
- xUncorrected2 ≈ -0.316 - 0.1 * 0.38 / (sqrt(0.007996) + 1e-8) ≈ -0.741
Experiment
You can interact with these dynamics directly using the Adam Moment Bias Correction module.
- Set the Gradient stream to "Constant", Beta 1 to 0.9, and Beta 2 to 0.999.
- Set Updates to 1. Notice how the Uncorrected position jumps much further than the Corrected position due to the uncorrected moving averages being close to 0.
- Increase Updates to 4. Observe that the Corrected position moves linearly as intended for a constant gradient, while the Uncorrected position trace starts aggressively and gradually slows down.
- Change the Gradient stream to "Alternating". Compare how the corrected moments adapt to the changing sign.
Limitations
This simulation models a simplified 1-dimensional, up to 4-step sequence with a fixed external gradient stream, rather than full network training over many epochs. It isolates the moving averages but does not show the effects on a real loss landscape or model weights. True loss functions have dynamic derivatives that change based on the updated position, whereas this module uses frozen gradients for precise arithmetic tracking. At 0 updates, corrected moments are unavailable (division by zero), so they are displayed as "Unavailable".
References
The equations for the moments and the bias correction are introduced in:
- Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. This is the original Adam paper which details the moving averages and why bias correction is necessary for early iterations.
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Adam Moment Bias Correction module.
Try Adam Moment Bias Correction on Riano →