Moreau Envelope and Huber Duality

The Moreau envelope provides a way to smooth non-differentiable convex functions while preserving their minimum. By adding a quadratic penalty scaled by a parameter λ, the proximal minimization operation yields a continuously differentiable envelope whose derivative represents a bounded slope, directly linking to the Huber loss.

Worked Example

Consider the absolute value function f(x) = |x|. We compute the Moreau envelope e_λ(x) for input x = 1 and smoothing parameter λ = 0.5. The proximal minimizer z^* minimizes the trial objective g(z) = |z| + (z - x)² / (2λ). This demonstrates the exact evaluation of the duality gap.

1. Calculate Proximal Minimizer:
   soft_threshold(x, lambda) = sign(x) * max(|x| - lambda, 0)
   prox_0.5(1) = 1 * max(1 - 0.5, 0) = 0.5

2. Calculate Moreau Envelope:
   Since |x| > lambda (1 > 0.5), use the outer formula:
   e_0.5(1) = |x| - lambda / 2 = 1 - 0.25 = 0.75

3. Verify Envelope Derivative:
   e'(x) = (x - prox) / lambda
   e'(1) = (1 - 0.5) / 0.5 = 1.0

Experiment

To visualize the smoothing effect and the duality gap in the interactive Moreau Envelope and Huber Duality module, follow these steps to adjust the parameters and observe the effect on the Moreau envelope.

Select Input x from the dropdown to set the evaluation point of the envelope.
Select Smoothing lambda to control the width of the quadratic bowl (Huber loss transition region). Observe how a larger λ creates a wider smoothed tip.
Select Trial z to evaluate the trial objective g(z) independently of the optimal proximal point.

Limitations

While the Moreau envelope guarantees continuous first derivatives (gradient Lipschitz), it does not guarantee second-order differentiability. At the boundary where |x| = λ, the second derivative of the Huber loss is discontinuous. The visualization focuses on the 1D absolute value case; generalized proximal operators in higher dimensions require more complex solvers beyond explicit soft-thresholding.

References

The following reference provides more rigorous theoretical analysis and context for the Moreau envelope, the proximal mapping, and related concepts in convex analysis.

Candès, E. (2020). Moreau-Yosida Regularization.View PDF

Moreau Envelope and Huber Duality