Brier Decomposition

Context

The scalar Brier score is easy to compute, but by itself it does not say whether forecast quality comes from honest calibration, useful separation of cases, or an easy base-rate environment. This note records the standard decomposition-level view that makes those distinctions explicit.

Formal Statement

For binary forecasts with probabilities $p_i$ and outcomes $o_i\in \{0,1\}$,

$$\mathrm{BS}=\frac{1}{N}\sum _{i=1}^N(p_i-o_i{)}^2.$$

When forecasts are grouped into bins indexed by $k$, Murphy’s decomposition writes

$$\mathrm{BS}=\mathrm{REL}-\mathrm{RES}+\mathrm{UNC},$$

where:

• $\mathrm{REL}$ measures how far the empirical event frequency in each bin is from the forecast probability in that bin.
• $\mathrm{RES}$ measures how much the event frequencies vary across bins relative to the base rate.
• $\mathrm{UNC}$ is the irreducible uncertainty of the outcome process at the base rate.

Relative to a baseline forecast, the Brier skill score is

$$\mathrm{BSS}=1-\frac{\mathrm{BS}}{{\mathrm{BS}}_{ref}}.$$

Derivation / Construction

• Start from the mean squared error form of the Brier score.
• Partition examples by forecast value or forecast bin.
• Add and subtract the empirical frequency of the positive class within each bin.
• Expanding the square and regrouping terms yields one component for within-bin mismatch (REL), one for between-bin discrimination (RES), and one base-rate term (UNC).
• In words: better calibration lowers REL, sharper discrimination increases RES, and the environment fixes UNC.

Implications

• Good calibration alone is not enough for a low Brier score; a forecaster that always predicts the base rate can have low reliability error but zero resolution.
• High resolution is valuable only when paired with enough calibration to avoid overconfident mistakes.
• BSS is useful when raw Brier scores are hard to compare across tasks because it normalizes against a chosen reference predictor.
• In modern ML evaluation, the decomposition is a clean way to explain why two models with similar calibration plots can still have different probabilistic usefulness.

Links

• Parent concept: Brier Score
• Related concept: Calibration
• Source anchors: Wikipedia2026 - Brier Score, PredictAddict2026 - Calibration vs Refinement Thread