Label Smoothing and Cross-Entropy

Worked Example

Let us calculate label smoothing cross entropy step-by-step for a classification task with three classes. Consider Logits z = [2, 0, 0], true class 0, and uniform smoothing mass distributed with ε = 1/4 (one quarter).

1. Determine Softmax Probabilities:
   First, subtract the maximum logit (2) for numerical stability.
   z' = [0, -2, -2]
exp(z') = [1, e-2, e-2] ≈ [1, 0.1353, 0.1353]
   Sum of exponentials ≈ 1.2707.
   Probabilities p = [0.787, 0.1065, 0.1065].
2. Determine Smoothed Targets:
   The one-hot target is [1, 0, 0]. The uniform smoothing mass distributes ε/3 to each class and weights the one-hot component by (1-ε).
   For class 0: q0 = (1 - 1/4) * 1 + 1/12 = 3/4 + 1/12 = 10/12 = 5/6 ≈ 0.833
   For class 1, 2: q1,2 = (1 - 1/4) * 0 + 1/12 = 1/12 ≈ 0.083
   The smoothed targets are q = [5/6, 1/12, 1/12].
3. Calculate Losses:
   Hard-label loss = -ln(p0) = -ln(0.787) ≈ 0.240 Nats.
   Cross-entropy loss = - (5/6 * ln(p0) + 1/12 * ln(p1) + 1/12 * ln(p2)) ≈ 0.573 Nats.
4. Calculate Logit Gradients:
   The gradient of the cross-entropy loss with respect to logit i is pi - qi.
   For class 0: 0.787 - 0.833 = -0.046
   For class 1: 0.1065 - 0.0833 = 0.023
   For class 2: 0.1065 - 0.0833 = 0.023
   The sum of all gradients is exactly 0.

Experiment

Explore the effects of varying epsilon on predictions and gradients in the interactive module:

1. Change the Logit 0 value and observe how the probabilities change. Notice how the gradient adjusts based on the discrepancy between the target q and the predicted probability p.
2. Change the True class to 1 and observe how the smoothed targets update, shifting the maximum target mass.
3. Set Smoothing quarters to 4 (which means ε = 1). Observe that the targets become completely uniform ([1/3, 1/3, 1/3]), independent of the true class.
4. Set Smoothing quarters to 0 (which means ε = 0). Observe that the targets recover the exact one-hot vector and the cross-entropy loss equals the hard-label loss.

Limitations

Label smoothing cross-entropy assumes that uniformly distributing mass among incorrect classes is beneficial. In practice, not all incorrect classes are equally plausible; for instance, classifying a dog as a cat might be less penalizing than classifying it as an airplane, but uniform label smoothing treats all errors equally. The hard assumption of a uniform prior does not provide optimal calibration across all datasets.

References

This module is based on the following primary resource detailing the uniform label smoothing target mixture distribution and its mathematical formulation.

• Szegedy et al., Rethinking the Inception Architecture, section 7