Calibration of Language Models

Overview & Motivation

Score Topic 3’s Kneser–Ney trigram on $23{,}219$ tokens of new text from the generator that wrote its training corpus. For every token, write down the probability the model gave its single likeliest next word, and whether that word is the one that came. Sort the tokens into fifteen bins by that probability, and in each bin compare the average probability with the fraction of hits. The fifteen points lie almost on the diagonal: the model’s mean confidence is $0.522$ and its accuracy $0.527$. Now do the same on $830$ tokens of real MD&A text from EDGAR filings. The model’s mean confidence is $0.144$; its accuracy is $0.059$. It reads as overconfident by a factor of more than two, and most of that reading, we will see, is one token.

That comparison is a calibration check, and this page makes it precise for language models. The sampling topic treated temperature as a decoding choice that trades quality against diversity. Here the same operation is a repair: the temperature that makes the model’s probabilities honest. We develop it in order:

1. What calibration means for a 3,427-way forecast. The strong form, that the whole distribution is right, is the same as being the true conditional, and no corpus can check it. The weak form, that the top probability is right, any corpus can check.
2. What miscalibration costs. Every log loss splits into three exact parts, and only one of them can be recalibrated away.
3. How to measure it honestly. The standard estimator is biased in both directions, and a calibrated twin of the model measures one of the biases.
4. One dial. Temperature scaling is a convex fit that matches one moment; it keeps every argmax but not the order of confidences.
5. Why $\tau = 1$. Maximum likelihood leaves the training-data temperature at exactly $1$ when the model can scale its own logits, and below $1$ when it is penalized.
6. Calibrated sets. Calibration is what makes a nucleus a prediction set.

Then we measure both corpus engines on three test sets and return to the filings.

---

What calibration means for a 3,427-way forecast

We write $X$ for the context a model conditions on (the trigram’s last two words, or the decoder’s whole prefix), $Y$ for the next token, and $F = f(X) \in \Delta^{\mathcal V}$ for the model’s forecast, a point of the probability simplex over the vocabulary $\mathcal V$. The truth is $P = P(Y \mid X)$, the next-token distribution given everything the model sees. The recalibration of $f$ is

$$C \;=\; P(Y \mid F) \;=\; \mathbb{E}[P \mid F],$$

the distribution of the next token among all contexts that receive the same forecast. Geometrically, $f$ sends contexts to points of the simplex; $C$ averages the truth over each point’s preimage.

The formalML uncertainty-quantification topic defines calibration for a binary outcome, where there is only one probability to check. A next-token forecast has $3{,}426$ free coordinates, and the two notions above sit at the two ends of what one could ask of them.

Part (c) is why canonical calibration is out of reach for a language model. Most contexts the trigram has seen have a row of their own ($2{,}950$ seen contexts sit on $2{,}480$ distinct rows in the test text below), so checking canonical calibration means checking that nearly every row is the true conditional over $3{,}427$ outcomes, from a few occurrences of each context. What a corpus can check is top-label calibration, one number per token, and that is what every corpus number on this page measures. Canonical calibration we test exactly, on small finite worlds where every expectation is a sum.

---

What miscalibration costs

The log loss of a forecaster is its cross-entropy, $\mathbb{E}[-\log F_Y]$. Calibration is one part of it, and the decomposition is clearest first for the Brier score $\mathbb{E}\lVert F - e_Y \rVert^2$, where $e_Y$ is the corner of the simplex at the token that occurred. Three points sit in the simplex for each context: the forecast $F$, its recalibration $C$, and the truth $P$. Conditional on $F$, the outcome $e_Y$ averages to $C$; conditional on $X$, it averages to $P$. So the squared distance from $F$ to the outcome splits by Pythagoras, twice.

The three terms answer three questions. The irreducible term is what the text cannot tell any forecaster that sees the same context, the floor that only a longer context lowers. The refinement gap is what $f$ throws away by sending different contexts to the same forecast. The reliability term is what a recalibration of the forecast, a function applied to $F$ alone, could recover. Bröcker (2009, Eq. 10) gives the split into $\mathbb{E}\,H(C)$ and the reliability term for any proper score, and his §3 on sufficiency gives the second split; for binned binary forecasts under the Brier score it is Murphy’s (1973) partition, which the formalML topic proves as its Theorem 3. DeGroot and Fienberg (1983) named the two halves calibration and refinement.

So the drop in held-out log loss that any fitted map of the forecast achieves estimates a lower bound on how miscalibrated the model is. It is only an estimate, from a sample, and it needs to be an honest one, since a temperature fit and scored on the same tokens flatters itself. We cross-fit: fit $\tau$ on the even sentences and score the odd ones, then swap, and average the two by their token counts. And it is only a lower bound, for the one map we tried. Temperature’s can be far from tight, as the finance case study will show. The harness checks Theorem 1 and Corollary 1 to $10^{-12}$ on $50$ random finite worlds, against $41$ temperatures each.

---

Measuring calibration honestly

On a corpus we see tokens, not conditional distributions, and the standard estimator bins them.

The formalML topic notes that ECE depends on the bin count, biased toward zero with too few bins and upward with too many. Both directions can be made exact, and the second can be measured.

Part (c) is the correction Bröcker (2012) and Ferro and Fricker (2012) derived for the Brier score’s reliability term; Kumar, Liang, and Ma (2019) state it for binned models (their Definition 5.2), where every token in a bin shares one $\rho$, $D = 0$, and the estimator is exactly unbiased, and they prove it needs far fewer samples than the plug-in. For a model whose confidences vary inside a bin, the remaining bias is the small negative heterogeneity term. Roelofs and colleagues (2022) find the binned estimator’s bias most severe for perfectly calibrated models, which is part (b).

Part (b) also suggests a measurement. Keep the model’s confidences and redraw every token’s hit as a coin with that confidence: this calibrated twin is calibrated by construction, so its estimated ECE is pure estimator noise. On the trigram’s in-distribution tokens, at $15$ bins, $100$ twins average $0.006$ with a standard deviation of $0.0012$, while the trigram itself scores $0.018$, about ten standard deviations above the floor. The miscalibration is real, and small. The twin’s floor rises with the bin count, from $0.002$ at one bin to $0.012$ at $60$; it grows like $\sqrt{B}$, at about $0.25\sqrt{B/n}$. The trigram’s own plug-in ECE rises too, from $0.005$ to $0.024$, but that is not part (a) at work. Part (a) is about the population ECE and nested bins, and the plug-in also carries the noise of part (b): the twin’s floor alone rises $0.010$ of that $0.019$. Bin counts of $10$ and $15$ are not nested in any case. The harness checks part (a) along $B = 1, 2, 4, \dots, 64$ on $50$ worlds, and parts (b) and (c) by exact Poisson-binomial expectations on $200$ random bins.

---

One dial: temperature scaling

A recalibration map has to be fit, and the simplest has one parameter. With $z$ the log-probabilities of a row and $\beta = 1/\tau$, temperature scaling replaces the row by $\operatorname{softmax}(\beta z)$, exactly the operation of the sampling topic, now chosen to minimize the held-out log loss

$$L(\beta) \;=\; \frac{1}{N} \sum_{\text{tokens}} \Big[\log \sum_v e^{\beta z_v} \;-\; \beta z_y\Big].$$

Part (a) is the formalStatistics exponential-families topic’s Theorem 7.1 for a one-parameter family whose natural parameter is $\beta$, and part (b) is its Theorem 7.4: the maximum-likelihood fit matches the sufficient statistic’s mean. Guo and colleagues (2017, Supplement S2) characterize temperature scaling the same way, as the maximum-entropy distribution subject to that one moment. The harness fits $\beta$ by Newton’s method from $\beta = 1$ and checks that $\lvert L'\rvert \lt 10^{-10}$ at the result on every engine and test set (by convexity, the global optimum); on the two EDGAR sets it also agrees with a bounded scalar search to $10^{-6}$.

Part (c) separates temperature scaling from the recalibrations of the formalRAG significance-testing topic, whose Theorem 2 shows that a monotone map of a score preserves every ranking. Temperature is not a function of the confidence: two rows with the same top probability can land on different ones. On the trigram’s in-distribution rows, Kendall’s $\tau_b$ between the confidences at $\tau = 1$ and at $\tau = 2$ is $0.860$ over distinct rows and about $0.81$ over tokens. A selective-prediction rule that ranks contexts by confidence changes when the temperature does, though not by much at the top: a rule that keeps the fifth of the tokens with the highest confidence keeps $91.5\%$ of the same tokens from $\tau = 1$ to $\tau = 2$, and its accuracy goes from $1.000$ to $0.997$.

KN trigram (V = 3,427)decoder (V = 512)

continuation (same generator)held-out splitEDGAR filings

temperature τ1.00

bins B1510153060

the engine’s ECEcalibrated twin, meantwin ± 2 SD (100 twins)

ECE at 15 bins

0.018

calibrated twin’s ECE (noise floor)

0.006 ± 0.001

mean confidence / accuracy

0.522 / 0.527

log loss, nats per token

2.538

nucleus(0.9): coverage / kept mass

0.926 / 0.939

NLL-optimal τ* (its NLL)

1.003 (2.538)

cross-fitted gain from τ*, nats

0.000

tokens scored

23,219

Fewer bins average miscalibration away and more bins invent it (Theorem 2): the twin, calibrated by construction, shows how much of an ECE is noise at this sample size. Temperature moves tokens between bins, so the circles can jump, and only the overall accuracy readout stays fixed (Proposition 2). In distribution both engines sit near the diagonal. On the EDGAR filings the trigram’s coverage can’t pass its share of known words at any temperature, because the unknown-word token is last in every row (Proposition 3); the decoder’s comes close only at high τ.

temperature τ1.00

seen twice or more (30 probes)seen once (33 probes)never seen (48 probes)all 111 probes

seen twice or more: confidence / accuracy

0.785 / 1.000

seen once: confidence / accuracy

0.332 / 0.697

never seen: confidence / accuracy

0.303 / 0.062

all probes: log loss at this τ

1.623

log loss, in sample: τ = 1 / one τ* / τ* per regime

1.623 / 1.613 / 1.119

Accuracy does not move with τ; confidence does. Facts seen twice or more are always right but underconfident, so their loss falls as τ falls; facts never seen are overconfident, so theirs falls as τ rises toward the uniform forecast. The pooled optimum (1.179) serves neither.

---

Why τ = 1 at all

In distribution, the trigram’s NLL-optimal temperature is $1.003$, as if the model came out of training exactly where temperature scaling would put it. That number is a balance, not a law: on the words the trigram knows, $\tau^*$ is $0.96$, and the <unk> tokens pull it back up. A Kneser–Ney trigram is no likelihood optimum and has no direction that scales its logits, so nothing pins its temperature. For some families, though, something does.

This is the score equation of the formalStatistics maximum-likelihood topic (its Theorem 1) read along one direction of parameter space. A log-linear bigram, whose logits are a matrix $W$ indexed by the previous token, has the direction exactly: $s_\beta(W) = \beta W$, and the L2 penalty $\Omega = \tfrac12\lVert W \rVert^2$ scales as required. Fit on the shared filings over the decoder’s $512$ tokens, at ten penalties from $10^{-2}$ to $10^{-7}$, the derivative at $\beta = 1$ and $-\lambda\lVert W \rVert^2$ agree to six decimals every time. Seven of the ten:

penalty $\lambda$	$10^{-2}$	$10^{-3}$	$10^{-4}$	$3 \cdot 10^{-5}$	$10^{-5}$	$10^{-6}$	$10^{-7}$
derivative at $\beta = 1$	$-0.277$	$-1.092$	$-1.456$	$-0.960$	$-0.581$	$-0.155$	$-0.029$
$\tau^*$, training	$0.234$	$0.520$	$0.744$	$0.822$	$0.871$	$0.943$	$0.984$
$\tau^*$, continuation	$0.232$	$0.516$	$0.744$	$0.829$	$0.889$	$1.001$	$1.107$
$\tau^*$, held	$0.210$	$0.472$	$0.711$	$0.784$	$0.828$	$0.885$	$0.917$
$\tau^*$, EDGAR	$0.285$	$0.758$	$0.938$	$1.038$	$1.130$	$1.355$	$1.647$

The training row is the theorem: the more penalty, the more underconfident. Without the penalty there is no finite optimum to be stationary at: the logits of word pairs never seen in training run off to $-\infty$, and with them the held-out likelihood of any text that contains one. The penalty that keeps the model usable is what makes it underconfident on its training data. The other rows are text the theorem says nothing about, and the temperatures there are not pinned. On the continuation $\tau^*$ crosses $1$ near $\lambda = 10^{-6}$, on EDGAR between $10^{-4}$ and $3 \cdot 10^{-5}$, and on held it never crosses on this grid. (The test sets, continuation, held, and EDGAR, are defined below.)

Topic 14’s decoder has no such direction. Its final layer normalization has no learned gain, its output matrix is its input embedding scaled by a fixed $1/\sqrt{d}$, and scaling the embedding moves the inputs as well as the outputs. Nor is it a stationary point: its $1{,}200$ steps of Adam end with a nonzero gradient over the training windows. Neither hypothesis of Theorem 3 holds, so nothing pins its temperature. On every second training sentence the NLL-optimal $\tau$ is $0.943$, and it is $0.937$ to $0.943$ across Topic 14’s four training seeds.

The same dial appeared in the sampling topic for the opposite reason. There, temperature reshaped a model deliberately, to trade predictability for diversity.

---

Calibrated sets

The sampling topic’s nucleus keeps the shortest list of likeliest tokens whose mass reaches $p$. It is also a prediction set: a claim that the next token is in the list. Calibration is what makes the claim honest.

In distribution the trigram’s nucleus of mass $0.9$ keeps $0.939$ of the mass on average and covers $0.926$ of the tokens; the decoder’s keeps $0.945$ and covers $0.951$. Both are close, in opposite directions. The harness checks the proposition exactly on $30$ calibrated worlds, and on the trigram’s own rows with outcomes drawn from the model, where coverage matches kept mass within four standard errors. A guarantee of coverage that does not need calibration at all comes from choosing the threshold on held-out data, which is split conformal prediction.

---

On the corpus

Both engines are the sampling topic’s. Topic 3’s interpolated Kneser–Ney trigram has $3{,}427$ outcomes, $3{,}425$ word types plus </s> and <unk>. Topic 14’s two-block decoder has $512$ tokens, $509$ words plus <s>, </s>, and <unk>, and <s> is never scored. Here both score every token of a text, so the unknown-word token <unk> is an outcome like any other; the decoder sees only the first $32$ positions of each sentence. Three test sets:

• continuation: the generator that wrote the training corpus, run on for $100$ more documents. The same source: a best case, in which $486$ of its $2{,}043$ sentences repeat a training sentence word for word.
• held: Topic 3’s held-out split, a fresh run of the generator whose word frequencies differ from the training text’s.
• EDGAR: the public MD&A excerpts, real filings, $33$ sentences and $830$ trigram tokens.

The table gives mean confidence and accuracy, the ECE at $15$ bins beside the calibrated twin’s floor, the NLL-optimal temperature $\tau^*$, and the coverage and kept mass of the nucleus of mass $0.9$.

engine, test set	conf. / acc.	ECE / twin	$\tau^*$	nucleus cover / kept
trigram, continuation	$0.522$ / $0.527$	$0.018$ / $0.006$	$1.003$	$0.926$ / $0.939$
trigram, held	$0.512$ / $0.500$	$0.023$ / $0.006$	$1.055$	$0.876$ / $0.938$
decoder, continuation	$0.653$ / $0.661$	$0.016$ / $0.006$	$0.957$	$0.951$ / $0.945$
decoder, held	$0.649$ / $0.707$	$0.064$ / $0.006$	$0.824$	$0.967$ / $0.944$

In distribution, both engines are close to calibrated, and a cross-fitted temperature recovers almost nothing: under $0.0001$ nats for the trigram, $0.002$ for the decoder. The trigram’s reliability diagram has a shape, though. Bins with confidence between $1/3$ and $0.6$ are overconfident and every bin above $2/3$ is underconfident. Kneser–Ney’s discounts, which take mass from what was seen and give it to what was not, explain half of that. Refit the trigram with scaled discounts and score the same tokens in every column (gaps are accuracy minus confidence):

discount, $\times$ Ney’s	$0.1$	$0.25$	$0.5$	$1$	capped at $0.99$
$\tau^*$	$1.542$	$1.339$	$1.176$	$1.003$	$0.979$
gap, $1/3$ to $0.6$	$-0.070$	$-0.062$	$-0.048$	$-0.026$	$-0.023$
gap, above $2/3$	$-0.006$	$-0.004$	$+0.001$	$+0.012$	$+0.016$

The gap above $2/3$ flips sign with the discount, so that underconfidence is the discounts’ doing. The overconfidence between $1/3$ and $0.6$ is there at every discount and shrinks as the discount grows, so it is not their signature. The top bin holds $7{,}652$ of the tokens at accuracy $0.999$: the generator’s fixed phrases, like “cost → of” and “from its head → office”, which turn up anywhere in a sentence. They are not sentence openers. Of that bin’s tokens, $55\%$ fall at positions $0$ to $5$, against $53\%$ of all tokens, and only $24\%$ of a sentence’s first three tokens land in it, against $33\%$ overall.

Under the held-out split’s mild shift the two engines move in opposite directions: the trigram becomes slightly overconfident, and the decoder reads as underconfident. Nearly all of the decoder’s gap sits on rows whose top label is <unk> ($3{,}649$ tokens, confidence $0.456$, accuracy $0.785$), as <unk> rises from $10.3\%$ of its tokens to $16.0\%$. Rows with a word on top read $0.686$ and $0.693$. Shift does not have a sign.

---

---

Where this sits

Everything on this page is the calibration of one next-token distribution at a time. The calibration of whole generations is a different property: Braverman and colleagues (2020) show that the entropy rate of a model’s samples drifts upward with length, the long-horizon face of Corollary 2, and correct it with a one-parameter fit of their own. Desai and Durrett (2020) find pre-trained transformers calibrated in domain and less so out of domain, the pattern of the table above, and Ovadia and colleagues (2019) show that calibration measured on in-distribution data does not survive dataset shift. For large models the question moves to the confidence of a chosen answer: Kadavath and colleagues (2022) measure it on multiple-choice questions and on the model’s own judgments of its answers, and the GPT-4 report (OpenAI 2023, Figure 8) finds the pre-trained model well calibrated and the post-trained one much less so. Neither finding is a case of Theorem 3, whose hypotheses ask for a stationary point of a penalized likelihood and a direction that scales the logits. What to do with a calibrated confidence, answering or abstaining, is the formalRAG selective-generation topic.

---

Honest caveats

---

Implementation

The companion notebook (notebookPath) imports language_model_calibration.py, which owns every number on this page. That module imports the sampling topic’s module for temperature, the nucleus, and the fact model, and through it the beam-search topic’s trigram and trained decoder. It implements finite worlds with exact recalibration, the plug-in and debiased estimators and the calibrated twin, a Newton temperature fit on deduplicated rows, the log-linear bigram fit by L-BFGS, and per-temperature statistics for both engines on all three test sets. A harness of $41$ assertion-based tests runs in about two and a half minutes on a laptop CPU. The tests cover:

• Proposition 1, Theorem 1, and Corollary 1 exactly on random finite worlds, and Proposition 1’s counterexample.
• Theorem 2(a) on nested bins, (b) and (c) by exact Poisson-binomial expectations, and the calibrated twin’s floor on the corpus.
• Proposition 2’s derivatives against finite differences, the Newton fit’s stationarity on every set and against a scalar search on EDGAR, the invariance of accuracy, and the reordering of confidences by rows and by tokens.
• Theorem 3’s identity on the log-linear bigram at ten penalties, where held-out temperatures land on that grid, and the decoder’s temperature across four seeds with its nonzero gradient.
• Proposition 3 exactly on calibrated worlds and on the trigram’s rows, Corollary 2, and the fast nucleus against the sampling topic’s own.
• The in-distribution, held-out, and EDGAR findings, the Kneser–Ney reliability shape and the discount refits, the known-word cut, <unk>’s rank and the coverage ceiling, the map onto <unk>, and the fact regimes, including that no row mixes them.

The laboratory reads its data from a JSON file that one of those tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.