Probability Calibration & Brier Score

Probability Calibration & Brier Score

Mathematical Definition

For observation i, let p_i be the forecast probability of an event and y_i its outcome, either 0 or 1. This page uses the single-event binary normalization: the score ranges from 0 to 1, and lower is better. It does not sum the squared errors of both event categories, which would double this value.

In a group of n equal forecasts p with s observed events, both event and non-event errors contribute. A forecast of 0.2 for a group with 2 events out of 10 has loss [2(0.8)² + 8(0.2)²]/10 = 0.16, even though its probability matches the observed event frequency exactly. Squaring just the difference between the forecast and the group frequency would miss this remaining outcome uncertainty.

Key Concepts

Exact empirical groups and the Murphy decomposition

Pool all observations sharing exactly the same forecast probability into a group k. Let n_k be its size, o_k its observed event fraction, w_k = n_k/N its weight, and o the overall event fraction. The decomposition is an exact algebraic identity for these groups, apart from floating-point display rounding:

• Reliability error: A nonnegative penalty for differences between issued probabilities and observed frequencies. Zero means an exact match in this sample, not proven population calibration.
• Resolution: Different forecasts identify groups whose event frequencies depart from the overall rate. With reliability held fixed, greater resolution reduces the score.
• Uncertainty: The score of always predicting this sample's overall event rate. It is largest at o = 0.5 and zero when all outcomes are the same.

To see the identity, a group's mean loss equals (p_k − o_k)² + o_k(1 − o_k). Average with the sample weights. The weighted outcome-variance term equals o(1 − o) minus the weighted squared deviations of group frequencies from o, giving REL − RES + UNC.

An independently worked comparison

The initial fictional sample has three groups of 10 observations, probabilities 0.2, 0.5 and 0.8, and event counts 2, 5 and 8. Its event rate is 15/30 = 0.5. The groups have losses 0.16, 0.25 and 0.16, giving BS = 0.19. All probabilities match their frequencies, so REL = 0. Resolution is (0.09 + 0 + 0.09)/3 = 0.06 and uncertainty is 0.25. Thus 0 − 0.06 + 0.25 = 0.19.

The overconfident example changes only the first and last forecasts to 0 and 1. Their observed frequencies stay 0.2 and 0.8. Reliability error becomes (0.04 + 0 + 0.04)/3 = 2/75; resolution and uncertainty stay fixed. The Brier score rises to 13/60, about 0.216667. More extreme probabilities can therefore worsen forecast quality without changing the ordering of these groups.

Predicting 0.5 for every observation instead pools everything into one forecast group. Its frequency is 0.5, so REL and RES both become zero and BS = 0.25. Retaining three separately named rows would create artificial resolution for a constant forecast; the visualization correctly merges them.

The “Perfect outcomes” preset refers only to its certain endpoints: p = 0 has no events, and p = 1 has all events. The middle group still has 5 events out of 10 at p = 0.5. Its score is therefore 1/12 overall, not zero. Zero reliability error still allows nonzero Brier loss. At the opposite extreme, confidently predicting every binary outcome incorrectly gives BS = 1.

Why squared loss encourages honest probabilities

If the true event probability is q, the expected loss of issuing p is q(1 − p)² + (1 − q)p² = (p − q)² + q(1 − q). The unique minimum is p = q, including q = 0 and q = 1. This is the strictly proper scoring property: optimizing expected loss rewards the true probability, rather than a more confident claim. A particular finite sample can still favor an inaccurate forecast by chance.

Sampling, weighting and limits

• Each observation has equal weight; forecast groups do not. Doubling just one group's size changes its contribution and usually the overall event rate.
• The bars compare exact forecast groups. Combining different probabilities into broad bins is a different summary and does not automatically preserve this score identity for the original forecasts.
• Many distinct probabilities with few observations each make empirical calibration estimates unstable. Matching a tiny sample cannot establish population calibration.
• Fit calibration methods using appropriate training/validation data, then evaluate on separate held-out observations. This module edits illustrative counts and does not fit a calibrator.
• Compare models on the same evaluation outcomes. Dataset difficulty and event prevalence change uncertainty, so scores from different populations need care.
• A valid empty group contributes no observations. With no observations anywhere, event frequencies and scores are undefined. Negative/fractional counts, impossible event counts, empty fields and out-of-range probabilities are explicit errors.

Historical Context

Glenn W. Brier introduced a probability forecast verification score in 1950 in meteorology. Allan H. Murphy's 1973 partition distinguished uncertainty, reliability and resolution. Proper scoring rules subsequently became a broader statistical framework for assessing probabilistic predictions; Gneiting and Raftery's 2007 review develops that framework. This page uses the modern single-event binary squared-loss convention and an exact empirical grouping calculation.

Real-world Applications

• Weather forecasts: Check issued event probabilities against recorded outcomes and compare a forecast system with a base-rate prediction.
• Machine learning: Evaluate classifier probabilities alongside ranking and threshold metrics, particularly when probabilities inform downstream decisions.
• Operational forecasting: Compare predicted event frequencies with observed frequencies while accounting for the number of observations behind each probability.

Related Concepts

• Confusion Matrix & Metrics — assesses hard decisions after a threshold, rather than the fidelity of the full probability values.
• ROC Curve & AUC — evaluates ranking discrimination; a monotone probability change can preserve ranking while altering calibration and Brier loss.
• Knowledge Distillation — uses softened teacher probabilities for student training, a different use of probability temperature.
• Entropy & Information Theory — studies logarithmic uncertainty; the Brier uncertainty term here is binary variance, not Shannon entropy.

References

1. Brier (1950), Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review 78, 1–3.
2. Murphy (1973), A New Vector Partition of the Probability Score, Journal of Applied Meteorology 12, 595–600.
3. Gneiting & Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association 102, 359–378. Author-hosted full text.