Neural Collapse Simplex Geometry

During the terminal phase of training deep neural networks, the features of instances belonging to the same class often collapse to their class means, and the class means arrange themselves as the vertices of a regular simplex.

Worked Example

Consider a simplified configuration where we have K = 3 classes. The regular simplex vertices v_c for c ∈ {0, 1, 2} can be derived by taking the standard basis vectors e_c in R³, subtracting their mean, and normalizing to length 1.

Let the center radius r = 1, making the class means \mu_c = 1 · v_c. The within-class spread is set to δ = 0.25.

Metric	Value
Between-class trace	r² = 1² = 1
Within-class trace	δ² = 0.25² = 0.0625
Collapse ratio	0.0625 / 1 = 0.0625
Off-diagonal center product	-r² / (K - 1) = -1 / 2 = -0.5

Experiment

By manipulating the parameters in the Interactive Module, you can observe how the geometry affects classification accuracy:

Select the Center radius to 1 and the Within-class spread to 1. This eventually leads to overlap between class clusters, dropping accuracy below 1.0 (specifically 0.6666666666666666) while Collapse ratio reaches 1.0.
Select the Center radius to 0. This causes all class means to collapse to the origin, leading to a random baseline Nearest-center accuracy of 0.3333333333333333.

Limitations

This module is a static geometric illustration of the terminal phase of training. It does not provide proof that any specific neural network will undergo neural collapse during its training dynamics. Real-world data, optimization paths, and network architectures introduce substantial complexity not captured by these idealized regular simplex formations.

References

For a comprehensive study of neural collapse, refer to the seminal paper introducing the phenomenon: Prevalence of neural collapse during the terminal phase of deep learning training.

Neural Collapse Simplex Geometry