Start a project
Model behaviour Reviewed 24 Sep 2026 · 17 sources

calibration

/ˌkæl.əˈbreɪ.ʃən/ noun Plural calibrations · verb calibrate · adjectives calibrated, well-calibrated
Definition

The agreement between an AI model’s confidence in its answers and how often they are right, or an adjustment that brings the two into line.

Examples
  • A model whose 80%-confidence answers are right 80% of the time
  • Temperature scaling applied to an image classifier after training
  • An assistant saying it is unsure on questions it often gets wrong
A row of ten answer cards from a model, each stamped 90% sure. Nine are marked with a tick and one with a cross.
Inkling · Answers stamped 90% sure, of which nine in ten are rightFig. C1
Earliest AI use
2001
Machine learning: calibrated probability estimates from classifiers.
Current meaning
2022
Language models: confidence in their own answers, even stated in words.
Opposite
Miscalibration
Confidence that runs above accuracy (overconfidence) or below it.
§01

In plain words

In AI, calibration is how well a model’s confidence matches its accuracy: answers given with 80% confidence should be right about 80% of the time.

A model that reports how sure it is can be checked against its record. Its answers are grouped by the confidence it gave, and the right answers in each group are counted. If about 80% of the answers given at 80% confidence are right, and likewise at other levels, the model is well calibrated. Forecasters are judged the same way: a forecaster’s 70% calls should come true about 70% of the time.1

Confidence matters because it decides what happens next. A calibrated score can tell a system which answers to trust and which to pass to a person. People also judge confidence from how an answer reads. In a 2025 study, users overestimated how often a large language model was right, and longer explanations raised their confidence without improving accuracy.2

Calibration varies by model and shifts with training. Modern deep neural networks, unlike those of a decade earlier, proved overconfident. Language models fresh from pretraining are often well calibrated on multiple-choice questions. The later training that turns them into assistants can reduce this, as OpenAI reported for GPT-4.3 In tests on OpenAI’s 2024 SimpleQA benchmark, models asked for a confidence percentage consistently overstated it.

Calibration is separate from accuracy. A model that is right half the time can still be well calibrated if it signals that doubt. Calibration also does not prevent hallucination. A calibrated model still makes false statements, but its lower confidence marks where they are more likely.

§02

How it happens

Calibration is measured by comparison. A model’s predictions are sorted into bins by confidence, such as 70 to 80%. Each bin’s average confidence is then set against the share it got right. Plotted together, the bins form a reliability diagram, on which a calibrated model follows the diagonal. The expected calibration error condenses the chart into one number, a weighted average of the gaps across bins.4

Overconfidence can come from training itself. A classifier is trained to lower its loss, an error score that falls as it puts more probability on the right label. Once nearly every training example is right, the loss can still fall if it grows more confident.4 Guo and colleagues found a simple repair, temperature scaling, which divides the model’s raw scores by one number fitted on held-out data. That softens the probabilities without changing which answer ranks first, so accuracy is unchanged.

Language models add two complications. Reinforcement learning from human feedback pulls a model toward answers that earn reward, which distorts its probabilities. In Anthropic’s tests, a single temperature adjustment largely repaired this.5 A model can also be asked to state its confidence, in words or as a percentage. For ChatGPT, GPT-4 and Claude, stated confidence was typically better calibrated than the underlying probabilities,6 though it still tends to run high.

Fig. C2 · Stated confidence against answers that proved rightIllustrative, not measured
  1. Stated 60% 54%
  2. Stated 70% 58%
  3. Stated 80% 63%
  4. Stated 90% 67%
  5. Stated 99% 71%
An overconfident model’s answers, grouped by the confidence it stated. Each bar shows the share that were right. A calibrated model’s bars would reach their labels. The numbers are invented.
§03

Origin and first known use

From calibrate, formed from caliber. In 1839 the verb meant measuring a caliber, such as the bore of a gun. By 1869 it also meant working out the values of the parts of a scale. Calibration, the act of calibrating, followed in the 1850s. Forecasters use the word for probability judgments that match outcomes, so that events given a 70% chance happen about 70% of the time. Machine learning researchers applied it to classifiers’ probability estimates by 2001, borrowing tools from forecast evaluation. From 2022, research on language models used it for a model’s confidence in its own answers, including confidence stated in words.

  1. 1850s Calibration, the act of calibrating, enters English. The verb calibrate, formed from caliber, dates from 1839.
    Online Etymology Dictionary, “calibration” and “calibrate” Etymonline →: source for 1850s
  2. 2001 Zadrozny and Elkan present methods for obtaining calibrated probability estimates from decision tree and naive Bayes classifiers, for decisions whose costs vary by example.
    Zadrozny and Elkan, “Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers”, ICML 2001 ucsd.edu →: source for 2001
  3. 2005 Niculescu-Mizil and Caruana chart classifiers’ calibration with reliability diagrams, a tool from forecast evaluation. Neural nets prove well calibrated; boosted trees and SVMs do not.
    Niculescu-Mizil and Caruana, “Predicting Good Probabilities With Supervised Learning”, ICML 2005 cornell.edu →: source for 2005
  4. 2017 Guo and colleagues find modern deep networks, unlike those of a decade earlier, poorly calibrated and overconfident, and recommend temperature scaling as a simple fix.
    Guo, Pleiss, Sun and Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017 arXiv 1706.04599 →: source for 2017
  5. 2022 Lin, Hilton and Evans teach a GPT-3 model to state its confidence in words, such as “90% confidence”, and find the stated levels well calibrated.
    Lin, Hilton and Evans, “Teaching Models to Express Their Uncertainty in Words” arXiv 2205.14334 →: source for 2022
  6. 2023 OpenAI reports that GPT-4’s pretrained model is highly calibrated on multiple-choice questions and that post-training reduces its calibration.
    OpenAI, “GPT-4 Technical Report” arXiv 2303.08774 →: source for 2023
  7. 2024 OpenAI’s SimpleQA benchmark asks models to state their confidence as a percentage. The models tested consistently overstate it.
    OpenAI, “Introducing SimpleQA”, 30 October 2024 openai.com →: source for 2024
  8. 2026 Anthropic’s new constitution for Claude lists being calibrated as part of honesty, including conveying no more or less confidence than Claude has.
    Anthropic, “Claude’s new constitution”, 22 January 2026 anthropic.com →: source for 2026
§04

In the wild

9 sightings · 2017–2026 · oldest first
  1. Paper14 Jun 2017

    “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.”

    Guo et al., ICML 2017 arXiv 1706.04599 →: Guo et al., ICML 2017, 14 Jun 2017Archived →: copy of Guo et al., ICML 2017, 14 Jun 2017
  2. Talk14 Sep 2020

    “This tutorial introduces fundamental concepts in classifier calibration and gives an overview of recent progress in the enhancement and evaluation of calibration methods.”

    Peter Flach and colleagues, ECML-PKDD 2020 tutorial classifier-calibration.github.io →: Peter Flach and colleagues, ECML-PKDD 2020 tutorial, 14 Sep 2020Archived →: copy of Peter Flach and colleagues, ECML-PKDD 2020 tutorial, 14 Sep 2020
  3. Paper28 May 2022

    “To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own answers in natural language.”

    Lin et al., arXiv preprint arXiv 2205.14334 →: Lin et al., arXiv preprint, 28 May 2022Archived →: copy of Lin et al., arXiv preprint, 28 May 2022
  4. Paper11 Jul 2022

    “We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format.”

    Kadavath et al., Anthropic anthropic.com →: Kadavath et al., Anthropic, 11 Jul 2022Archived →: copy of Kadavath et al., Anthropic, 11 Jul 2022
  5. Blog14 Mar 2023

    “Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct).”

    OpenAI, GPT-4 research announcement openai.com →: OpenAI, GPT-4 research announcement, 14 Mar 2023Archived →: copy of OpenAI, GPT-4 research announcement, 14 Mar 2023
  6. Paper24 May 2023

    “A trustworthy real-world prediction system should produce well-calibrated confidence scores… enabling deferral to an expert in cases of low-confidence predictions.”

    Tian et al., EMNLP 2023 arXiv 2305.14975 →: Tian et al., EMNLP 2023, 24 May 2023Archived →: copy of Tian et al., EMNLP 2023, 24 May 2023
  7. Blog30 Oct 2024

    “A factuality benchmark like SimpleQA also allows us to measure the scientific phenomenon known as calibration, or whether language models “know what they know.”

    OpenAI, SimpleQA announcement openai.com →: OpenAI, SimpleQA announcement, 30 Oct 2024Archived →: copy of OpenAI, SimpleQA announcement, 30 Oct 2024
  8. Paper21 Jan 2025

    “To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct.”

    Steyvers et al., Nature Machine Intelligence nature.com →: Steyvers et al., Nature Machine Intelligence, 21 Jan 2025Archived →: copy of Steyvers et al., Nature Machine Intelligence, 21 Jan 2025
  9. Docs22 Jan 2026

    “Claude tries to have calibrated uncertainty in claims… and avoids conveying beliefs with more or less confidence than it actually has.”

    Anthropic, Claude’s constitution anthropic.com →: Anthropic, Claude’s constitution, 22 Jan 2026Archived →: copy of Anthropic, Claude’s constitution, 22 Jan 2026
§05

Not to be confused with

  • Accuracy

    Accuracy is the share of answers that are right. Calibration asks whether stated confidence matches that share. A model right 60% of the time is well calibrated if it reports about 60% confidence.

  • Confidence score

    A number a product shows beside an answer or prediction. It is calibrated only if, tested against outcomes, items shown at 80% prove right about 80% of the time. Scores from some methods, such as boosted trees, distort probabilities.

  • Instrument calibration

    Adjusting a device, such as a scale, against a known standard. The AI sense keeps the idea of checking readings against reality: the readings are a model’s confidence levels, checked against how often it is right.

  • Hallucination

    A hallucination is a false statement presented as fact. Calibration concerns the confidence attached to statements. A calibrated model still makes errors, but it attaches lower confidence where errors are more likely.

    See hallucination →
§06

Questions

  • What does calibration mean in AI?

    Calibration is how closely a model’s confidence matches how often it is right. If a model gives many answers at 80% confidence and about 80% of them are correct, it is well calibrated. Forecasters are judged the same way: events given a 70% chance should happen about 70% of the time. Machine learning researchers applied it to classifiers by 2001 and to language models from 2022.

  • Are large language models well calibrated?

    Partly. Anthropic researchers reported in 2022 that larger models were well calibrated on multiple-choice and true/false questions in the right format. OpenAI found GPT-4 highly calibrated after pretraining, but post-training raised its expected calibration error on multiple-choice questions from 0.007 to 0.074. In OpenAI’s 2024 SimpleQA tests, models asked for a confidence percentage consistently overstated it.

  • How is model calibration measured?

    The model’s predictions are grouped into bins by the confidence it gave. Each bin’s average confidence is then compared with the share of its answers that were right. Plotted together, the bins form a reliability diagram, on which a perfectly calibrated model follows the diagonal. The expected calibration error sums up the chart in one number: a weighted average of the gaps across bins.

  • Why are neural networks overconfident?

    Training lowers an error score, the loss, which falls as a network puts more probability on the right label. In 2017 Guo and colleagues showed that a network right on almost every training example can still lower its loss by growing more confident. Their remedy, temperature scaling, divides the network’s raw scores by one number fitted on held-out data. It softens the probabilities without changing accuracy.

  • What is the difference between calibration and accuracy?

    Accuracy is how often a model is right. Calibration is whether its confidence matches that rate. A model that is right half the time can be well calibrated if it reports about 50% confidence. A more accurate model can be poorly calibrated if it claims near certainty on every answer. Guo and colleagues found modern deep networks, unlike those of a decade earlier, poorly calibrated.

§07

Watch

§09

Cite this entry

“Calibration.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/dictionary/calibration.

First published 24 Sep 2026 · Last reviewed 24 Sep 2026 · Report an error →