Principal Component Regression Shrinkage

Worked Example

Let us consider a specific configuration to calculate the coefficients manually.

1. Setup the inputs: Retained PCs = 1, Ridge lambda = 1, and Feature rotation degrees = 0.
2. The design matrix in PCA space has diagonal elements.
3. For PCR, we keep the first principal component and drop the second. The coefficients in PC space are 1 and 0 respectively.
4. For Ridge, the coefficients are shrunk based on lambda. The coefficients in PC space are 9/(9+1) = 0.9 and 2/(1+1) = 1.0.
5. Without rotation, the physical coefficients match the PC coordinates. PCR gives [1.0, 0.0] and Ridge gives [0.9, 1.0].
6. We evaluate the SSE (Sum of Squared Errors) by taking the squared difference of predictions from the target variable.

Experiment

You can interact with the module to see these concepts in action:

1. Set Retained PCs to 2. Notice how PCR coefficients perfectly match least squares.
2. Set Ridge lambda to 4. Observe the shrinkage in the ridge coefficients.
3. Change the Feature rotation degrees. This rotates the physical coordinate system relative to the PC axes, demonstrating how regression models respond to rotated inputs.

Limitations

Principal Component Regression (PCR) assumes that the directions with the highest variance in the design matrix are also the most predictive of the target response.

In cases where a low-variance principal component has a strong relationship with the target, PCR may discard it, causing a higher residual error. Ridge regression offers a smoother shrinkage where all components contribute somewhat, proportional to their variance and the lambda parameter.

References

For more details on PCR vs Ridge, see the following sources:

• PCR vs PLS - Scikit-Learn Documentation
• Ridge Regression - Scikit-Learn Documentation