ReLU Sharpness Rescaling Symmetry

Worked Example

Consider a tiny scalar two-layer network defined by f(x) = b · max(0, a · x). If we choose a positive constant c, we can set a = c and b = 1/c. Notice that for any c > 0, the network computes f(x) = (1/c) · max(0, c · x) = max(0, x), which is exactly the standard ReLU function.

Let our training data be two points: (1, 1) and (2, 2). The loss is the mean squared error L = 0.5 · 1/N ∑ (f(x_i) - y_i)². At our chosen solution, the training loss is exactly 0. The Hessian at this optimum can be computed as:

Experiment

To see this symmetry in action, open the module at https://riano.app/module/relu-sharpness-rescaling-symmetry.

Follow these steps:

1. Change the Rescaling control to adjust c. Observe that the Original Prediction always matches the probe value, meaning the network function is unchanged.
2. Set a small Perturbation (like 0.1) and watch how the First-weight perturbation loss drastically changes depending on the rescaling factor, even though the unperturbed model is identical.
3. Change the Probe input to verify the model acts as the identity for positive values and zero for negative values.

Limitations

This demonstration uses a trivial scalar two-layer network. In real deep neural networks, the geometry of the loss landscape is vastly more complex, involving millions of dimensions and complex inter-layer interactions. Additionally, we should not infer that identical functions with different parameterizations have different generalization properties; the generalization is a property of the function, while sharpness is a property of the parameterization. The Hessian formula used here is strictly valid at the zero-loss optimum.

References

This concept is explored in detail in the paper "Sharp Minima Can Generalize For Deep Nets" by Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. It formally shows that sharpness is a parameterization-dependent property. Read the paper at https://proceedings.mlr.press/v70/dinh17b.html.