Centered Kernel Alignment Invariance

Worked Example

Let us calculate the Centered Kernel Alignment (CKA) between representations X and Y for three samples, where X = [[-1, 0], [0, 1], [1, -1]]. Under the Identity representation with translation 0 and noise 0.5 applied along the noise profile (1, -1, 0) in the first column, we add [[0.5, 0], [-0.5, 0], [0, 0]] to obtain Y = [[-0.5, 0], [-0.5, 1], [1, -1]].

Both X and Y already have column means (0, 0), so their centered matrices are X_c = X and Y_c = Y.

1. First Gram matrix: X_c X_c^T = [[1, 0, -1], [0, 1, -1], [-1, -1, 2]]. Its squared Frobenius norm is 1² + 0² + (-1)² + 0² + 1² + (-1)² + (-1)² + (-1)² + 2² = 10, so ||X_c X_c^T||_F = √10 ≈ 3.16227766.
2. Second Gram matrix: Y_c Y_c^T = [[0.25, 0.25, -0.5], [0.25, 1.25, -1.5], [-0.5, -1.5, 2]]. Its squared Frobenius norm is 0.0625 + 0.0625 + 0.25 + 0.0625 + 1.5625 + 2.25 + 0.25 + 2.25 + 4 = 10.75, so ||Y_c Y_c^T||_F = √10.75 ≈ 3.27871926.
3. Cross-covariance matrix: X_c^T Y_c = [[1.5, -1], [-1.5, 2]]. Its squared Frobenius norm is 1.5² + (-1)² + (-1.5)² + 2² = 2.25 + 1 + 2.25 + 4 = 9.5.
4. Linear CKA = 9.5 / (√10 × √10.75) = 9.5 / √107.5 ≈ 0.91626136.

Experiment

You can test different learned representations and invariances by adjusting the controls in theCentered Kernel Alignment Invariance module.

1. Change the Representation to "Rotate" to apply an orthogonal transformation to the data. Notice that the Centered Kernel Alignment remains 1.0 (perfectly similar).
2. Increase the Translation to shift the samples in space. CKA centers the data first, meaning it is perfectly invariant to uniform translations.
3. Add Noise to break the exact linear relationship between the samples, lowering the CKA score below 1.0 as the geometric similarity decreases.
4. Select the "Constant" representation, mapping all points to zero. The centered norm becomes zero, leaving CKA undefined rather than indicating perfect similarity.

Limitations

Centered Kernel Alignment requires the denominator to be strictly positive. If the centered norm of either representation is zero (e.g., a constant representation for all samples), the CKA becomes mathematically undefined rather than evaluating to 1 or 0. Moreover, linear CKA is invariant to orthogonal transformations and isotropic scaling, but it is not invariant to any arbitrary invertible linear transformation (like anisotropic scaling).

References

The following reference provides the mathematical foundation and properties of Centered Kernel Alignment:

• Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019). Similarity of Neural Network Representations Revisited.Proceedings of the 36th International Conference on Machine Learning (ICML). This paper discusses linear and kernel CKA and analyzes their invariances.