AdaGrad Coordinate Learning Rates

Adaptive Gradient Algorithm (AdaGrad) dynamically adjusts the learning rate for each parameter by keeping a running sum of the squared gradients.

algorithm

The objective is to optimize a function f(x, y) = (x² + b · y²)/2. The gradient is g = (x, b · y).

At each step t, the gradient squared is added coordinate-wise to an accumulator G:

G.x = G.x + g.x^2\n G.y = G.y + g.y^2

The weights are updated using a global learning rate η divided by the square root of the accumulated gradients:

x = x - η · g.x / (√(G.x) + ε)\n y = y - η · g.y / (√(G.y) + ε)

worked-example

Let b=4, η=0.5, and initial position w_0=(2,2) with G_0=(0,0).

Step 1: Compute gradient g = (x, 4 · y) = (2, 8).
Update G: G.x = 0 + 2² = 4, G.y = 0 + 8² = 64.
Update weights (assuming ε=1e-8):
x = 2 - 0.5 · 2 / (√4 + 1e-8) ≈ 2 - 0.5 = 1.5.
y = 2 - 0.5 · 8 / (√64 + 1e-8) ≈ 2 - 0.5 = 1.5.

The scales are 0.5 / 2 = 0.25 and 0.5 / 8 = 0.0625 respectively. Notice how the parameter with the steeper gradient (y) receives a much smaller effective step size.

experiment

In the interactive AdaGrad Coordinate Learning Rates module, you can observe this behavior:

Set the Y curvature to 4 and the Learning rate to 0.5.
Observe the trajectory for different step counts (1, 2, 5).
Notice how the effective learning rates decrease over time, which can eventually lead to very slow progress.

limitations

A major limitation of AdaGrad is its continually decaying learning rate. Since the accumulator G is strictly increasing, the effective learning rate eventually shrinks so much that the model stops learning. Later methods like RMSProp and Adam address this by using an exponentially decaying average of squared gradients instead of a simple sum.

references

The original AdaGrad paper introduces adaptive subgradient methods for online learning and stochastic optimization.

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (Duchi et al., 2011)

AdaGrad Coordinate Learning Rates