Adaptive Decoupled Weight Decay
With an adaptive metric, L2 regularization and multiplying weights by a decay factor become different updates.
By Riano · Published · Updated · Report a correction
Adaptive Decoupled Weight Decay
In modern deep learning optimization, weight decay is often applied to regularize the model. When using standard stochastic gradient descent, L2 regularization (adding an L2 penalty to the loss) and weight decay (subtracting a fraction of the weight at each step) are equivalent. However, when an adaptive preconditioner like Adam is introduced, these two approaches lead to geometrically distinct updates, motivating the development of decoupled weight decay (e.g., AdamW).
Worked Example
Consider a fixed positive preconditioner P = diag(1, 0.1), quadratic data with Hessian H = diag(1, 4), and bias vector b = (1, 1). We evaluate the gradient at initial weights w = (1, 2) where g = Hw - b = (0, 7). For a step size η = 0.1 and decay strength λ = 1:
- Coupled L2 update:w - η P (g + λ w) = (1, 2) - 0.1 · diag(1, 0.1) · ((0, 7) + 1 · (1, 2)) = (1, 2) - 0.1 · diag(1, 0.1) · (1, 9) = (1, 2) - (0.1, 0.09) = (0.9, 1.91).
- Decoupled update:(1 - η λ)w - η P g = (1 - 0.1 · 1)(1, 2) - 0.1 · diag(1, 0.1) · (0, 7) = 0.9(1, 2) - (0, 0.07) = (0.9, 1.8) - (0, 0.07) = (0.9, 1.73).
- Difference:The difference is (0.9, 1.73) - (0.9, 1.91) = (0, -0.18).
Experiment
Use the interactive controls to see how updates change under different configurations.
- Select Second preconditioner to 0.1, Step size to 0.1, and Decay strength to 1.
- Observe that the Coupled second coordinate is 1.91, while the Decoupled second coordinate is 1.73, making the difference -0.18.
- Change the Second preconditioner to 1. You will see both updates agree with second coordinates of 1.1, showing that isotropic preconditioners eliminate this difference.
Limitations
This module uses a fixed preconditioning matrix to clearly illustrate the difference between coupled L2 regularization and decoupled weight decay. In practical algorithms like AdamW, the preconditioner is dynamically updated based on running moments of past gradients. Thus, this toy model captures the essence of the decoupled formulation without pretending to implement the full Adam moment estimation logic.
References
For more details on decoupled weight decay and its connection to adaptive gradient methods, see the original paper by Loshchilov and Hutter (2017):
- Loshchilov, I., & Hutter, F. (2017). Decoupled Weight Decay Regularization. https://arxiv.org/abs/1711.05101
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Adaptive Decoupled Weight Decay module.
Try Adaptive Decoupled Weight Decay on Riano →