Split Conformal Prediction

Concept Overview

Split Conformal Prediction is a distribution-free method that takes an existing, fixed prediction model and endows it with valid, finite-sample marginal coverage guarantees. It works by computing absolute residual scores on a held-out calibration set and using their order statistics to construct prediction intervals for new test points, relying only on the exchangeability of the data.

Worked Example

Suppose we have a trained predictor that outputs a point prediction y₀ = 1. We have a calibration set of n=9 held-out absolute residual scores: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9. We want an interval with a miscoverage rate α = 20% (marginal coverage of 80%).

Calculate the adjusted rank k = ⌈(n+1)(1-α)⌉ = ⌈(9+1)(1-0.2)⌉ = ⌈10 × 0.8⌉ = 8.
Find the k-th smallest calibration score, q. The sorted scores are 0.1, ..., 0.9. The 8th score is q = 0.8.
Construct the prediction interval: [y₀ - q, y₀ + q] = [1 - 0.8, 1 + 0.8] = [0.2, 1.8].

Interactive Experiment

Use the Split Conformal Prediction module to explore how the interval changes with different parameters.

Set "Miscoverage alpha (%)" to 20.
Set "Prediction center" to 1.
Set "Calibration scores" to "Distinct".
Observe that the calculated rank k is 8, the calibration radius is 0.800000, and the interval is [0.200000, 1.800000].
Change "Miscoverage alpha (%)" to 20 and "Calibration scores" to "Tied". The rank stays 8, but its tied score is 1/5, giving interval [4/5, 6/5] and full width 2/5.
Change "Miscoverage alpha (%)" to 5. The required rank k becomes 10. Since n=9, k > n, and the interval becomes unbounded ("all real numbers").

Assumptions and Limitations

The primary assumption is exchangeability of the calibration and test data. The method guarantees marginal coverage (averaged over all possible test points and calibration sets), but not necessarily conditional coverage (coverage for a specific individual test point X=x). Furthermore, the finite-sample correction (n+1)(1-α) means that if the calibration set is too small relative to the desired α (e.g., n=9 and α=5%), the required rank k will exceed n, forcing the method to output an infinite uninformative interval to maintain the mathematical guarantee.

The prediction center is illustrative output from an already fixed predictor. Training must be separate from calibration, and calibration and future scores must be exchangeable conditional on that fitted predictor. The deterministic scores here illustrate the rank construction; they are not an empirical coverage experiment and do not establish exchangeability. The guarantee averages over calibration and future draws, not posterior probability for one fixed interval or conditional coverage at every input. Ties can make coverage conservative. The drawn step-free interval has closed endpoints; the finite screen viewport does not bound the all-real-line result.

References

Read this paper for the foundation of split conformal prediction. It provides the marginal coverage guarantee motivated in this illustration.

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification— Discusses the split conformal prediction algorithm and the finite-sample marginal coverage guarantee.

Split Conformal Prediction