Brier score

Summary

This partial ingest is based on a clipped encyclopedia page. It provides a broad reference overview of the Brier score as a strictly proper scoring rule for probabilistic predictions, along with the common calibration-oriented interpretation and the classic decomposition literature around reliability, resolution, and uncertainty.

Key Claims

• The Brier score is a mean-squared-error style loss over predicted probabilities and realized outcomes.
• Lower Brier scores indicate better probabilistic predictions, but the metric bundles together multiple desiderata rather than isolating calibration alone.
• The score is naturally suited to binary and categorical forecasting settings and is not appropriate for ordinal targets.
• Decomposition results matter because probabilistic usefulness depends on more than simple reliability.

Methods / Formalism

• Binary form: $\mathrm{BS}=\frac{1}{N}{\sum }_i(p_i-o_i{)}^2$ for probabilities $p_i$ and outcomes $o_i\in \{0,1\}$.
• Categorical form: $\mathrm{BS}=\frac{1}{N}{\sum }_i{\sum }_k(f_{ik}-o_{ik}{)}^2$ with one-hot outcomes and probabilities summing to one.
• Decomposition view: the clipped reference points toward reliability, resolution, and uncertainty terms rather than treating calibration as the whole story.
• Relative interpretation: the Brier skill score compares a forecast against a reference baseline; see Brier Decomposition.

Evidence / Experiments

• The clip is a reference summary rather than an original experiment.
• It points to canonical sources from forecasting and statistical decision theory, making it useful as an orienting note but not a substitute for the original decomposition papers.

Connections

• Core reference source for Brier Score.
• Supports Calibration notes by making explicit that proper scoring rules evaluate forecasts more richly than raw accuracy alone.
• Connects naturally to Phan2026 - Humanity’s Last Exam, where calibration error is reported alongside accuracy, and to PredictAddict2026 - Calibration vs Refinement Thread, which emphasizes the calibration-versus-resolution distinction.
• Brier Decomposition stores the reusable equations and decomposition-level interpretation extracted from this reference.

Open Questions

• Which decomposition is most useful for modern ML settings with multiclass and structured predictions?
• When should the Brier score be preferred over log loss or selective-prediction metrics in AI evaluation?
• How do public benchmark reports communicate calibration without collapsing reliability and resolution into one number?

Citation

Wikipedia contributors. (2026). Brier score. Wikipedia. The clipped page cites Glenn W. Brier (1950) and later decomposition work as the primary historical references.