Calibration
Definition
Calibration describes how well predicted probabilities line up with observed frequencies, so that events assigned a confidence level occur at roughly that rate over repeated cases.
Why It Matters
Confidence quality matters for decision-making, risk communication, selective prediction, and benchmark interpretation. A model that is accurate but miscalibrated can still be unreliable, while a perfectly calibrated model can still be uninformative.
Formalism / Key Objects
- predicted confidence scores or probabilities
- empirical event frequencies conditioned on those scores
- reliability diagrams, calibration error summaries, and proper scoring rules such as the Brier Score
Connections
- Directly connected to Brier Score and other probabilistic evaluation tools.
- Important in AI Evaluation and Benchmarking, where benchmark dashboards increasingly report confidence quality next to accuracy.
- Part of the uncertainty toolkit tracked in Probability and Statistics.
- Guan2026 - Open Sourcing Monitorability Evaluations shows that filtering noisy positive-effect instances is also a calibration-adjacent reporting problem: metric values need reliability conditions, not just point estimates.
Common Confusions
- Calibration is not the same thing as usefulness or discriminative power.
- Better calibration after post-processing can come with worse refinement or resolution.
- A single calibration number can hide important subgroup or confidence-range failures.