Jackknife+ Leave-One-Out Intervals

Combine leave-one-out fits and residuals with the finite-sample Jackknife+ quantile correction.

Jackknife+ Leave-One-Out Intervals

Jackknife+ is a predictive inference method that combines leave-one-out cross-validation residuals with finite-sample quantile corrections. It provides distribution-free marginal coverage guarantees by evaluating the model's stability when excluding one training data point at a time.

Worked Example

Suppose we have a dataset with 6 points and an outlier. We want to predict a test point with alpha = 0.25 (75% coverage). First, we fit the OLS model leaving each point out in turn.

Calculate the leave-one-out residuals r_i = |y_i - f_(-i)(x_i)|.
For the test point, calculate the candidates: f_(-i)(x) - r_i and f_(-i)(x) + r_i.
Sort the lower and upper candidates.
Calculate the indices: lower index = floor((N+1)*alpha) = floor(7*0.25) = 1. Upper index = ceil((N+1)*(1-alpha)) = ceil(7*0.75) = 6.
The Jackknife+ interval is bounded by the corresponding sorted candidates, yielding an interval of [-3.0, 5.0].

Interactive Experiment

Use the Jackknife+ Leave-One-Out Intervals module to explore how the interval changes.

Set "Data" to "Outlier".
Set "Training size" to 6.
Set "Alpha" to 0.25.
Set "Test x" to 0.
Observe the calculated "Lower order index" is 1 and "Upper order index" is 6.
The resulting "Jackknife lower" is -3.0 and "Jackknife upper" is 5.0.
Change "Alpha" to 0.1 and "Training size" to 4. Notice that the bounds become -Infinity and Infinity.

Assumptions and Limitations

The primary assumption is the exchangeability of the training and test data, and that the learning algorithm treats the data symmetrically. It guarantees marginal coverage of at least 1-2*alpha, but does not guarantee conditional coverage. Like split conformal prediction, small sample sizes compared to a stringent alpha can lead to infinite (uninformative) intervals due to the finite-sample correction. Unlike split conformal, it does not require a separate hold-out set, making it more data-efficient at the cost of requiring N re-fits of the model.

References

Read this paper for the foundation of the Jackknife+ method.

Predictive inference with the jackknife+— Discusses the jackknife+ algorithm and its distribution-free marginal coverage guarantees.

Jackknife+ Leave-One-Out Intervals