Sparsemax Simplex Projection

The sparsemax function maps an unnormalized vector of logits to a probability distribution, similar to softmax. However, unlike softmax, which always assigns a dense distribution with strictly positive probabilities, sparsemax computes the Euclidean projection of the logits onto the probability simplex. This Euclidean projection naturally hits the boundaries of the simplex, yielding exactly zero probability for logits that are too small relative to the leading candidates.

Worked Example

Consider the logits vector z = [0, 0.25, 0.5] with Temperature T = 1. We seek the probability vector p on the simplex (where components sum to 1 and are non-negative) that minimizes the squared Euclidean distance to z. The solution takes the form p_i = max(z_i - τ, 0), where the threshold τ is determined such that the active components sum to 1.

Sort the logits in descending order: z(1) = 0.5, z(2) = 0.25, z(3) = 0.
For k=1: τ = (0.5 - 1)/1 = -0.5. Since z(1) > -0.5, we proceed.
For k=2: τ = (0.5 + 0.25 - 1)/2 = -0.125. Since z(2) > -0.125, we proceed.
For k=3: τ = (0.5 + 0.25 + 0 - 1)/3 = -0.08333333333333333. Since z(3) > -0.08333333333333333, we keep k=3.
The final active set has 3 coordinates, and the threshold is τ = -0.08333333333333333.
Compute probabilities: p_i = max(z_i - (-0.08333333333333333), 0).
The resulting probabilities are [0.08333333333333333, 0.3333333333333333, 0.5833333333333334].

Experiment

You can interact with the projection operation by adjusting the logits spread and the temperature.

Open the Sparsemax Simplex Projection module.
Set the Logits control to "Separated". Observe how the smallest two logits are aggressively pushed below the threshold, resulting in exactly zero probabilities, leaving only one active coordinate.
Change the Logits to "Close" and lower the Temperature to 0.5 (which scales the logits to [0, 0.5, 1.0]). The increased variance pushes the lowest coordinate to zero.
Compare the Sparsemax probabilities to the traditional Softmax output. Softmax never assigns exactly zero probability, no matter how small the logit, whereas Sparsemax efficiently identifies a compact active set.

Limitations

Unlike the dense softmax, sparsemax is piecewise differentiable rather than smoothly differentiable everywhere. Its gradient is zero for coordinates outside the active set, which can stop gradient flow completely for unselected classes during training if the model becomes too confident too early. Furthermore, the active-set sorting algorithm requires an O(N log N) sorting step (or a randomized O(N) median-finding step), making it computationally slightly more expensive than the O(N) exponentiation and sum required by softmax.

References

The sparsemax transformation was introduced as a sparse alternative to softmax, offering exact zeros and a geometric interpretation via simplex projection.

Martins, A. F. T., & Astudillo, R. F. (2016). From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. Demonstrates the formulation and the O(N log N) sorting algorithm.

Sparsemax Simplex Projection