Multiclass Temperature Scaling

Adjust confidence without changing a classifier's ranking while tracking stable log loss.

Multiclass Temperature Scaling

Temperature scaling is a simple yet effective technique to calibrate the probabilities output by a neural network. By dividing the logits by a learned scalar temperature parameter before applying the softmax function, we can adjust the model's confidence without altering its predictions.

Worked Example: Moderate Logits

Let's manually compute the probabilities for the "Moderate" logits [2, 1, -1] with a temperature of T = 1. We will use the max-shifted exponential trick for numerical stability.

z = [2, 1, -1]
T = 1
scaled_z = [2, 1, -1]

# Find max scaled logit
max_scaled_z = 2

# Compute sum of exponentials of (scaled_z - max_scaled_z) for other indices
# Indices 1 and 2 are "other"
sum_exp_other = exp(1 - 2) + exp(-1 - 2)
              = exp(-1) + exp(-3)
              ≈ 0.367879 + 0.049787
              ≈ 0.417666

# Compute logNormalizerRelative
logNormalizerRelative = log(1 + 0.417666)
                      ≈ 0.349012

# Compute log probabilities: logp_i = scaled_z_i - max_scaled_z - logNormalizerRelative
logp_0 = 2 - 2 - 0.349012 = -0.349012
logp_1 = 1 - 2 - 0.349012 = -1.349012
logp_2 = -1 - 2 - 0.349012 = -3.349012

# Compute probabilities: p_i = exp(logp_i)
p_0 = exp(-0.349012) ≈ 0.70538
p_1 = exp(-1.349012) ≈ 0.25950
p_2 = exp(-3.349012) ≈ 0.03512

This matches the computed result. The negative log likelihood (NLL) for target class 0 is exactly -logp_0 ≈ 0.349012.

Experimenting with Temperature

Open the interactive module at Multiclass Temperature Scaling and observe the effect of changing the temperature.

Set Logits to "Moderate" and Temperature to "1". The probability for class 0 is approximately 0.705.
Change Temperature to "4". The probability for class 0 drops to 0.444, making the distribution closer to uniform. The predicted class remains 0.
Set Logits to "Sharp" and Temperature to "0.25". The model becomes extremely confident, with class 0 probability reaching 1.0 (within standard float precision) and the NLL dropping to near zero.
Set Target class to "2". The model was highly confident in class 0, but the target was 2. The NLL explodes to 48.0, penalizing the overconfident wrong prediction.

Limitations of Temperature Scaling

Temperature scaling preserves the original ranking of the classes, meaning it does not change the model's accuracy on the original dataset. It only affects the confidence of the predictions. In this module, we use fixed example logits. In practice, the temperature parameter T is optimized on a held-out validation set (calibration set) by minimizing the negative log likelihood (cross-entropy loss).

Additionally, while temperature scaling works well for many models, it assumes the miscalibration is consistent across all classes and examples. If a model has varying confidence issues for different classes (e.g., highly confident for class A, but underconfident for class B), a single global temperature scalar might not be sufficient to calibrate it perfectly.

References

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1321-1330. https://proceedings.mlr.press/v70/guo17a.html

Multiclass Temperature Scaling