FedAvg Quadratic Client Drift

Federated Averaging (FedAvg) is a standard algorithm in federated learning where participating clients perform multiple local training steps on their local data before communicating the updated weights to a central server. This allows for reduced communication frequency. However, when the client loss functions are heterogeneous—specifically when they have different curvatures (second derivatives)—FedAvg can suffer from client drift.

Worked Example

Consider two clients with deterministic quadratic loss functions:

f1(w) = 0.5 * (w + 1)^2 f2(w) = 0.5 * h2 * (w - 1)^2

Let the global objective be the weighted sum of these local functions, with a fixed weight `q = 0.5` for the first client. Suppose `h2 = 4`, local steps `E = 2`, and step size `eta = 0.25`. The global objective is minimized at `w* = 0.6`.

In a communication round, each client starts from the current server parameter `w` and performs `E` local steps. For Client 1, `h1 = 1`, and the contraction per step is `(1 - eta * h1) = 0.75`. For Client 2, `h2 = 4`, and the contraction per step is `(1 - eta * h2) = 0`.

Start at server parameter `w = 0`.
Client 1 local steps: `w1_1 = 0 - 0.25*(0+1) = -0.25`, `w1_2 = -0.25 - 0.25*(-0.25+1) = -0.4375`.
Client 2 local steps: `w2_1 = 0 - 0.25*4*(0-1) = 1`, `w2_2 = 1 - 0.25*4*(1-1) = 1`.
Server averages the client endpoints: `w_next = 0.5 * (-0.4375) + 0.5 * 1 = 0.28125`.

The parameter moves to `0.28125`. The limiting fixed point of this process is `~0.391`, which differs from the global minimum `0.6` by a bias of `-0.209`.

Experiment

In the interactive model, you can visualize how this drift occurs.

Observe the local trajectories of the two clients starting from the server parameter.
Notice how the server updates its parameter based on the averaged client endpoints.
Increase the local steps `E` and observe the discrepancy between the FedAvg fixed point and the global minimizer when the second client's curvature `h2` is unequal to `h1=1`.
Set local steps `E = 1` and verify that the fixed point perfectly aligns with the global optimum, recovering centralized gradient descent.

Limitations

This module uses a highly simplified deterministic scenario with exactly two quadratic clients and full-gradient descent to isolate the effect of curvature-induced drift. It does not model stochastic noise, partial client participation (where only a subset of clients communicate per round), varying network delays, or the privacy mechanisms often employed in real-world federated learning systems.

References

The following primary references provide the foundational theory of Federated Averaging and analyze the optimization challenges, including client drift, caused by heterogeneous data distributions across participating devices.

FedAvg Quadratic Client Drift