Hat Matrix and Regression Leverage

Worked Example

Consider an OLS regression with an intercept and one predictor for four observations. Suppose the predictor values are x = [0, 1, 2, 10] (the "Extreme" design) and responses are y = [2, 5, 8, 42] (the "Last outlier" responses).

1. Calculate the mean of x: x = (0 + 1 + 2 + 10) / 4 = 3.25.
2. Calculate the sum of squares S_xx: (0 - 3.25)² + (1 - 3.25)² + (2 - 3.25)² + (10 - 3.25)² = 62.75.
3. The hat matrix entries are given by H_ij = 1/4 + (x_i - x)(x_j - x) / S_xx.
4. For the last observation (i=3), the leverage is H₃₃ = 1/4 + (10 - 3.25)² / 62.75 = 0.9760956175298805.
5. The fitted value for the last observation is 41.7609561752988, with a residual of 0.23904382470119523.
6. The leave-one-out prediction (LOO prediction) for the last observation is (41.7609561752988 - 0.9760956175298805 * 42) / (1 - 0.9760956175298805) = 32.0.

Experiment

In the interactive Hat Matrix and Regression Leverage module, you can explore the effects of different design and response configurations on the leverage.

1. Set the Design control to Single distinct (where x = [0, 0, 0, 1]). Notice how the last observation has a leverage of exactly 1.
2. Set the Responses control to Last outlier and the Observation control to 3.
3. Observe that for Single distinct, the LOO prediction is Unavailable because removing the observation at x=1 leaves the model unidentifiable at that point.
4. Switch back to the Extreme design and observe the Leverage and LOO prediction values. High leverage points disproportionately affect the fitted regression line.

Limitations

The hat matrix relies on the invertibility of the matrix X^TX (or in the simple linear case, S_xx > 0). If all predictors are identical (e.g., the Repeated design where x = [1, 1, 1, 1]), then S_xx = 0. In this singular case, the fitted projection reduces to a simple mean prediction where every entry is 1/n and the rank is 1. We artificially compute a unique constant fitted projection H = 11^T/4 despite nonunique coefficients without using an inverse singular X^TX.

References

The following reference provides the basis for the calculations:

• Lecture 20: Leverage, Influence and Cook's Distance - Explains the geometric interpretation of the hat matrix, leverage, and LOO prediction formulas.