Elastic Weight Consolidation Toy

Worked Example

Consider an old task with minimizer a = [-1, 1] and importance weights F = [9, 1]. A new task has minimizer b = [1, -1] and importance G = [1, 9]. We set retention strength λ = 1. The combined objective balances the losses:

1. Old task loss: L_A = 1/2∑ F_i(\theta_i - a_i)².
2. New task loss: L_B = 1/2∑ G_i(\theta_i - b_i)².
3. Total objective: L_B + λ L_A.
4. The unique minimizer is computed as: \theta_i = (G_i b_i + λ F_i a_i)/(G_i + λ F_i).
5. For parameter 1: \theta_1 = (1(1) + 1(9)(-1))/(1 + 1(9)) = -0.8.
6. For parameter 2: \theta_2 = (9(-1) + 1(1)(1))/(9 + 1(1)) = -0.8.
7. We calculate L_A at θ = [-0.8, -0.8]: L_A = 1/2(9(0.2)² + 1(-1.8)²) = 1.8.

Experiment

You can interactively explore the effect of λ on the parameter optimization trajectory in the Elastic Weight Consolidation Toy module.

1. Set Tasks to "Complementary".
2. Set Retention strength (λ) to "4".
3. Observe how the consolidated minimizer shifts closer to the old task minimizer compared to λ = 1. The old task loss L_A decreases to 0.972.
4. Set Tasks to "Unidentified coordinate". Notice how if a parameter has zero importance in both tasks (when λ = 0), the minimizer is technically non-unique.

Limitations

This toy model illustrates the core mechanic of EWC but simplifies several real-world aspects. Real EWC uses the diagonal of the Fisher Information matrix approximated from task data rather than manually specified importance weights. Moreover, we assume the loss landscape is perfectly quadratic, whereas real deep learning models have highly non-convex loss surfaces where EWC acts as a local quadratic approximation around the previous minimizer.

References

The original Elastic Weight Consolidation methodology was introduced in the paper:Overcoming catastrophic forgetting in neural networks by Kirkpatrick et al., 2017.