Noisy-Label Loss Correction

Theory

In classification tasks, we assume a known binary symmetric label transition channel characterized by a flip probability η. The transition matrix T relates the clean probability p and the observed noisy probability q.

The standard clean loss vector is [-log(1-p), -log p]. The backward loss correction computes the corrected loss: l_(corr)(i) = ((1-η)l_i - η l_(1-i))/(1-2η).

Its expectation over observed labels conditional on the true label equals the clean loss, correcting the gradient updates in expectation, though individual corrected losses can be negative. At η = 0.5, the transition matrix becomes singular, q = 0.5 regardless of p, and the true labels cannot be distinguished.

Worked Example

Consider a setup with the following parameters:

1. Flip probability: η = 0.25
2. Predicted clean probability: p = 0.75
3. True label: 1
4. Observed label: 0

We calculate the expected losses:

η = 0.25, p = 0.75\n l_0 = -log(0.25) ≈ 1.38629\n l_1 = -log(0.75) ≈ 0.28768\n Naive observed loss (on label 0): l_0 ≈ 1.38629\n Determinant: 1 - 2(0.25) = 0.5\n Backward corrected observed loss (on label 0):\n ((1-0.25)(1.38629) - 0.25(0.28768))/0.5 = (0.75(1.38629) - 0.07192)/0.5 = 1.93560

The calculated Backward corrected observed loss matches the expected output of 1.9356005054539454.

Experiment

In the interactive module at https://riano.app/module/noisy-label-loss-correction, you can investigate how expected loss and corrected loss change under symmetric label noise.

1. Select a value from the "Flip probability" control to update the label noise transition matrix. Notice that choosing 0.5 demonstrates the singular matrix boundary case, rendering backward correction unavailable.
2. Select a different value from the "Predicted clean probability" control to observe the updated target loss and observe how the backward corrected expected loss relates to the clean loss.

Limitations

This correction assumes the label-noise channel is completely known and independent of the features. If the noise is instance-dependent or unknown, this backward correction method may fail or amplify errors. It differs from methods like label smoothing or PU risk. Furthermore, at exactly 50% noise, the transition matrix becomes singular, meaning the true labels are entirely lost.

References

The primary derivation for the backward loss correction, including the exact arithmetic and analysis of singular matrix cases discussed in this module, comes from the original foundational paper on making deep neural networks robust to label noise. By assuming that the symmetric noise rate is fully known, the expected loss can be modified to precisely recover the correct gradients required for standard optimization algorithms without bias.

• Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. Patrini et al., CVPR 2017.