PCA and ZCA Covariance Whitening

Covariance whitening transformations map a set of variables with a known covariance matrix into a new set of variables whose covariance is the identity matrix. When the data has zero mean, this transformation spheres the data. PCA and ZCA represent two such transformations with different mathematical properties and physical interpretations.

Worked Example

Consider a centered covariance matrix with a correlation ρ = 0.5. The matrix is C = [[1, 0.5], [], [0.5, 1]].

The eigenvalues are \lambda_1 = 1.5 and \lambda_2 = 0.5. With a fixed basis R = [[1/√2, -1/√2], [], [1/√2, 1/√2]], we compute the PCA whitening matrix:

PCA W = diag(1/sqrt(lambda)) * R^T
      ≈ [[ 0.577,  0.577],
         [-1.000,  1.000]]

The ZCA whitening matrix is calculated as ZCA = R × PCA:

ZCA W ≈ [[ 1.115, -0.299],
         [-0.299,  1.115]]

Applying ZCA to a diagonal point (1, 1) yields approximately (0.816, 0.816).

Experiment

We can visualize how PCA and ZCA whitening differ in transforming data distributions and single points.

Open the PCA and ZCA Covariance Whitening module.
Set the Correlation to 0.5 and observe the transformed ellipse for PCA vs ZCA.
Change the point to Opposed and note how ZCA preserves the spatial arrangement relative to the original point better than PCA, minimizing the mean squared displacement.
Set Correlation to 1. Notice that whitening becomes impossible due to a singular covariance matrix.

Limitations

This demonstration assumes a 2-dimensional feature space and a fixed eigenvalue ordering. When the correlation ρ is exactly 1, the covariance matrix becomes singular, meaning the variables are perfectly linearly dependent. In such cases, full 2D whitening is impossible without regularization or dropping a dimension, which we do not implement here to strictly illustrate the singular constraint.

References

The transformation of the data space by PCA and ZCA gives equivalent statistical decorrelation but different geometric orientations.

Kessy, A., Lewin, A., & Strimmer, K. (2018). Optimal Whitening and Decorrelation. arXiv:1512.00809.

PCA and ZCA Covariance Whitening