Separable Logistic Finite MLE

Separable Logistic Finite MLE

In a linearly separable classification dataset, binary outcomes can be perfectly predicted by a sufficiently large weight vector. When maximizing the logistic likelihood (or minimizing the cross-entropy loss) without regularization, the optimization attempts to push the predicted probabilities arbitrarily close to 0 or 1. This means the weights grow to infinity, and a finite Maximum Likelihood Estimator (MLE) does not exist.

We can formalize this with a symmetric two-point dataset where a single feature takes values x = ± a. Let the labels correspond perfectly with weight 1 - ε, and flip to incorrect labels with weight ε. The weighted logistic objective is:

L(w) = softplus(-a · w) + ε · a · w + (λ)/2 w²

The first term represents the expected logistic loss on correctly labeled points. As w → ∞, softplus(-a · w) → 0. If ε = 0 (perfect separation) and λ = 0 (no regularization), the objective infimum is 0 but it is not attained at any finite w.

However, when we add label noise (ε > 0) or apply ridge regularization (λ > 0), the optimization problem changes:

• Label Noise (ε > 0): A finite fraction of the data pulls the weights back. The optimal finite MLE occurs at w^* = 1/a ln((1-ε)/(ε)).
• Ridge Regularization (λ > 0): The penalty term (λ)/2w² forces the curvature to be strictly positive everywhere, yielding a unique, finite minimizer that balances fitting the points perfectly against keeping the weights small.

Experiment

1. Select Feature magnitude 1.
2. Select Flip fraction 0.
3. Select Ridge lambda 0.1.
4. Observe the Optimum weight result is 1.6335061701558464.

The interactive module allows you to explore this phenomenon using a single-feature binary classification setup.

Adjust the Flip fraction and Ridge lambda controls to see how the objective curve transforms from having an infimum at infinity to having a well-defined minimum. You can also probe the curve at specific weights to evaluate the gradient and curvature at that point.

You can try the module here: Separable Logistic Finite MLE

Worked Example

Consider a dataset where a = 1, label flip fraction ε = 0.1, no ridge penalty (λ = 0), and we probe at w = 1.

Since λ = 0 and ε > 0, the finite optimum is:

w^* = 1/a ln((1-ε)/(ε)) = ln(0.9/0.1) = ln(9) ≈ 2.1972245773362196

At our probe point w = 1, the objective is:

1. softplus(-1) = ln(1 + e⁻¹) ≈ 0.31326168751822286
2. ε · a · w = 0.1 · 1 · 1 = 0.1
3. Summing these gives the probe objective: 0.31326168751822286 + 0.1 = 0.4132616875182229

The gradient at w = 1 is:

1. s = σ(1) = 1/(1+e⁻¹) ≈ 0.7310585786300049
2. grad = a · (s - (1-ε)) = 1 · (0.7310585786300049 - 0.9) = -0.16894142136999513

Limitations

This module considers a simplified 1D binary scenario with symmetric data. In multi-dimensional spaces, a dataset may be only partially separable, causing some weights to diverge while others converge. The effects of L1 regularization (lasso) can also induce sparsity alongside finding a finite solution, which behaves differently from ridge regularization shown here.

References

The following references provide deeper insights into the behavior of logistic regression on separable datasets.

• The Surprising Behavior of Linear Separable Data in Logistic Regression - This paper analyzes how optimization behaves on separable data.