Proximal Soft Thresholding

Concept Overview

Proximal operators are a key tool for solving optimization problems with non-differentiable penalty terms. The proximal soft thresholding operator corresponds to the L1 penalty, yielding exact zero coefficients (sparsity). The L2 proximal operator instead scales each coordinate by 1/(1 + lambda). An initially zero coordinate stays zero; a nonzero coordinate remains nonzero for finite supported lambda.

Worked Example

Consider a 2D unregularized vector input v = (3, -1). We want to find the solution x that minimizes:
0.5 * ||x - v||² + λ * ||x||₁
with λ = 1.

For the first coordinate, v₁ = 3. Using the L1 thresholding rule: x₁ = sign(3) * max(|3| - 1, 0) = 1 * max(2, 0) = 2.
For the second coordinate, v₂ = -1. Using the L1 thresholding rule: x₂ = sign(-1) * max(|-1| - 1, 0) = -1 * max(0, 0) = 0.
The final solution is exactly x = (2, 0). The number of non-zero coordinates is 1.
The objective function value is 0.5 * ((2-3)² + (0-(-1))²) + 1 * (|2| + |0|) = 0.5 * (1 + 1) + 2 = 1 + 2 = 3.

Experiment

Use the Proximal Soft Thresholding module to observe the difference between L1 and L2 penalties.

Set Input v1 to 3 and Input v2 to -1.
Set Penalty lambda to 1.
Leave Penalty as L1. Observe that x₂ becomes exactly 0 (Sparsity).
Change Penalty to L2. Observe the solution shifts to (1.5, -0.5), meaning no coordinate is exactly zero.
In Penalty = L2, set Penalty lambda to 0.5. Read solution (2, -2/3) and objective 5/3 = 1.666667; L2 results are not restricted to halves or quarters.
Restore L1 and set lambda to 4: both coordinates are zero and the objective is 5. At lambda 0, the input and solution coincide and objective is zero.

Limitations

This proximal operator assumes an identity mapping for the data matrix (as in denoising). For arbitrary design matrices (like general Lasso regression), there is no coordinate-wise closed-form solution, and iterative methods such as ISTA or FISTA are required. L2 uses the penalty (lambda/2) times the squared Euclidean norm, not lambda times that norm. Exact fractions and the sparsity count use the rational solution before decimal rounding. Contour radii use 30 bisection iterations and 128 angular segments, so the drawing is a numerical approximation to exact objective levels. The viewport can be panned without changing the optimization problem.

References

The mathematical derivation of this operator is well documented in optimization literature:

Proximal Algorithms (Parikh & Boyd, 2014) - Defines the proximal soft thresholding operator in its separable-penalty examples and explains its connection to the L1 penalty.

Proximal Soft Thresholding