Focal Loss

Worked Example

Consider a positive example (y = 1) predicted with high probability (p = 0.9). We use a modulating factor (1 - p_t)^gamma and an alpha-balancing factor alpha_t. Let alpha = 0.25 and gamma = 2.

1. p_t = p = 0.9 since y = 1.
2. alpha_t = alpha = 0.25 since y = 1.
3. Weighted Cross Entropy (WCE) = -alpha_t * ln(p_t) = -0.25 * ln(0.9) ≈ 0.02634
4. Modulation = (1 - p_t)^gamma = (1 - 0.9)^2 = 0.01
5. Focal Loss (FL) = Modulation * WCE = 0.01 * 0.02634012891445657 ≈ 0.00026340128914456557

The focal loss for this "easy" example is scaled down by a factor of 100 compared to the weighted cross entropy, significantly reducing its contribution to the overall loss gradient.

Experiment

You can interact with the Focal Loss module directly.

1. Try changing Gamma to 0. You will see that the Focal Loss curve perfectly aligns with the Weighted Cross Entropy curve, as the modulation factor becomes 1.
2. Next, increase Gamma to 2 and change the Predicted probability to 0.99. Observe how the Focal Loss approaches zero much faster than the Weighted Cross Entropy, effectively ignoring this highly confident example.

Explore these effects here: Focal Loss Interactive Module

Limitations

Focal Loss introduces two additional hyperparameters (alpha and gamma) that must be tuned for each specific task. The optimal values often depend on the severity of the class imbalance and the difficulty of the dataset. While effective for object detection, its benefits in tasks with less extreme imbalance may be marginal compared to well-tuned weighted cross-entropy.

References

• Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988). Available at: arXiv:1708.02002

This foundational paper introduces Focal Loss specifically to handle the extreme foreground-background class imbalance encountered during training of one-stage object detectors like RetinaNet.