Huber Loss and Robust Location
Compare bounded Huber influence with the pull of a large squared-error residual.
By Riano · Published · Updated · Report a correction
Huber Loss and Robust Location
The textbook Huber loss function provides a robust alternative to squared error when estimating location. It applies a quadratic penalty for small residuals and a linear penalty for large ones. This ensures that while small errors are treated efficiently like the mean, outliers do not exert unbounded influence on the estimate, as their gradients are clipped.
Worked Example
Let the fixed measurements be [0, 0, 1, 1, outlier] with an outlier = 10. Assume our candidate location estimate is 0.5, and the Huber threshold is delta = 1.
- Calculate residuals
r = y - locationfor each measurement:[-0.5, -0.5, 0.5, 0.5, 9.5]. - Determine Huber loss for each residual. For
|r| <= 1(the first four), the loss is0.5 * r^2 = 0.5 * 0.25 = 0.125. For the outlier (r = 9.5),|r| > 1, so the loss is1 * (9.5 - 0.5) = 9.0. Total Huber loss:0.125 * 4 + 9.0 = 9.5. Mean Huber loss:9.5 / 5 = 1.9. - Determine half-squared loss for each. For the first four:
0.5 * 0.25 = 0.125. For the outlier:0.5 * 9.5^2 = 45.125. Total squared loss:0.125 * 4 + 45.125 = 45.625. Mean half-squared loss:45.625 / 5 = 9.125. - Calculate Huber gradient with respect to location. The derivative is
-clip(r, -delta, delta). For the first four:-r = [0.5, 0.5, -0.5, -0.5]. For the outlier:-clip(9.5, -1, 1) = -1.0. Total gradient:0.5 + 0.5 - 0.5 - 0.5 - 1.0 = -1.0. Mean Huber gradient:-1.0 / 5 = -0.2. - Calculate squared gradient with respect to location. The derivative is
-r. Total gradient:0.5 + 0.5 - 0.5 - 0.5 - 9.5 = -9.5. Mean squared gradient:-9.5 / 5 = -1.9.
Experiment
Interact with the module at Huber Loss and Robust Location to see how outliers influence the loss and gradient.
- Set Outlier to
10. Notice that the Huber loss curve transitions from quadratic to linear, bounding the influence of the outlier on the gradient. - Change the Location to
1. Observe how the residuals shift and how the gradients adjust to reflect the new location. - Adjust Delta to
2. The region where the loss acts quadratically is widened, increasing the influence of points that were previously outside the threshold.
Limitations
This module demonstrates the unscaled textbook Huber objective, keeping the variance constant to illustrate the raw effect of the loss function. In practice, estimating location robustly often requires simultaneous scale estimation (e.g., using median absolute deviation) or a concomitant variance formulation. Furthermore, a capped gradient simply limits the influence of an outlier; it does not guarantee that the selected location is the global optimum.
References
- Robust Estimation of a Location Parameter. Peter J. Huber. Annals of Mathematical Statistics, 1964. Introduces the theoretical properties and the loss function now known as Huber loss.
This paper established the foundation for robust statistics by proving that clipping the influence function optimally bounds the effect of contamination in the underlying distribution.
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Huber Loss and Robust Location module.
Try Huber Loss and Robust Location on Riano →