Brier Score

Definition

The Brier score is a strictly proper scoring rule for probabilistic forecasts. For binary events with forecast probabilities $p_i\in [0,1]$ and outcomes $o_i\in \{0,1\}$, it is the mean squared error

$$\mathrm{BS}=\frac{1}{N}\sum _{i=1}^N(p_i-o_i{)}^2.$$

For multiclass forecasts with probabilities $f_{ik}$ and one-hot outcomes $o_{ik}$, the usual categorical form is

$$\mathrm{BS}=\frac{1}{N}\sum _{i=1}^N\sum _{k=1}^K(f_{ik}-o_{ik}{)}^2.$$

Why It Matters

It gives a simple way to evaluate probabilistic predictions while still reminding us that forecast quality is not only about top-line accuracy. The score rewards truthful probability estimates, but its value also depends on how informative or resolving the forecasts are.

Formalism / Key Objects

• Canonical definition: binary Brier score is $\frac{1}{N}{\sum }_i(p_i-o_i{)}^2$, with lower values better.
• Key decomposition: for binned binary forecasts, Murphy’s decomposition writes the score as $\mathrm{BS}=\mathrm{REL}-\mathrm{RES}+\mathrm{UNC}$, separating reliability, resolution, and base-rate uncertainty.
• Skill score: relative to a reference forecast, $\mathrm{BSS}=1-\frac{\mathrm{BS}}{{\mathrm{BS}}_{\mathrm{ref}}}$.
• Worked contrast: always predicting the base rate can look well calibrated while having poor resolution; see Brier Decomposition.

Connections

• Closely tied to Calibration but not reducible to it.
• Useful in AI Evaluation and Benchmarking when confidence quality matters alongside correctness.
• Lives inside the broader probabilistic-evaluation toolkit tracked by Probability and Statistics.
• Brier Decomposition holds the fuller formal payload for decomposition, skill scores, and calibration-versus-resolution tradeoffs.

Common Confusions

• A low Brier score does not mean calibration alone is good; refinement and base-rate structure also matter.
• The score is natural for binary and categorical events, not ordinal targets.
• The common modern binary range is $[0,1]$, but Brier’s original 1950 multicategory formulation used a scale up to $2$.
• Proper scoring rules evaluate forecasts, not just decisions taken from them.

Key Sources

• Wikipedia2026 - Brier Score
• PredictAddict2026 - Calibration vs Refinement Thread