Frisch-Waugh-Lovell Residual Regression

Worked Example

Consider four-element vectors where u = (-1, -1, 1, 1), v = (-1, 1, -1, 1), and z = (1, -1, -1, 1). Let our predictors be x1 = u and x2 = r * u + v, and response y = beta1 * x1 + beta2 * x2 + e * z.

If r = 1, beta1 = 2, beta2 = 3, and e = 1:

1. The full regression coefficient of x1 is exactly 2.0.
2. The marginal OLS of x1 alone yields 2.0 + 1 * 3.0 = 5.0.
3. Residualizing y and x1 against x2, and regressing ry on rx yields a coefficient of exactly 2.0, matching the full OLS.
4. The full residual sum of squares is 4 * e^2 = 4.0 (since norm-squared of z is 4).

Interactive Experiment

In the interactive model, you can independently vary the predictor mixing, the coefficients, and the residual scale to observe how the theorem holds:

1. Change the Predictor mixing to see how the correlation between predictors affects the marginal coefficient.
2. Adjust the First coefficient and Second coefficient to observe their independent effects on the full regression output.
3. Observe that regardless of these parameters, the Residual-regression coefficient always exactly matches the Full first coefficient.

Limitations

This exact mathematical identity only applies to ordinary least squares (OLS) linear regression without regularization. Non-linear models or regularized models (like Ridge or Lasso) do not preserve this exact geometric projection.

Primary References

For further reading on the Frisch-Waugh-Lovell theorem and its applications in econometrics and statistics, consider these resources:

• Foundations of Applied Regression (Ian Gow) - Regression Basics
• Frisch-Waugh-Lovell Theorem Notes