intermediate decoding-generation 40 min read

Sampling: Temperature, Top-k, and Nucleus

Ancestral sampling draws exactly from the model; temperature tilts it toward a Gibbs distribution and truncation cuts its tail, each at a cost we can compute exactly, and neither can add knowledge the model lacks

Part of the Decoding & Generation track · View full curriculum →

Prerequisites: Decoding as Search: Greedy and Beam Search

Overview & Motivation

Go back to the beam-search topic’s small example. After “revenue” the model says “rose” with probability $0.45$ , “was” with $0.35$ , and “fell” with $0.20$ . The most probable sentence, the mode, is “revenue was flat” at $0.238$ ; greedy decoding finds “revenue rose sharply” at $0.0864$ . Neither is what the model says about revenue. The model says that thirteen different sentences are possible, and how often each should turn up. To hear that, we draw from it.

That topic left a gap that motivates this one. On the shared filings, the trigram’s exact modes cost $0.835$ nats per word and the held-out sentences cost $2.576$ . This page adds a third number: the model’s own samples cost $2.348$ (all pooled per word, the end mark included). The mode is far more predictable than what the model usually writes. Sampling closes most of that gap, and we develop it in order:

Sampling is exact. Drawing one token at a time returns every string with exactly its probability, and a draw is the argmax of noisy log-probabilities, so greedy decoding is sampling with the noise turned off.
Temperature is a Gibbs distribution. Dividing logits by $\tau$ trades expected score against entropy. The entropy only rises with $\tau$ , and the log–log slope of the distribution divides by $\tau$ .
Per-step temperature is not a tempered model. Its cold limit is greedy decoding, not the mode, and above $\tau = 1$ the tempered model may not exist.
Truncation has an exact price. A cut that keeps mass $m$ costs $1 - m$ in total variation and $-\log m$ in KL, and over a sentence the KL adds up exactly.
Truncation is desmoothing, imperfectly. A Kneser–Ney model knows which words it saw, but no setting of any truncation rule recovers them on every prompt.

Then we measure every rule on two models trained on the shared filings, and ask what temperature does to a model’s facts.

Sampling is exact; search is not

We write $\mathrm{eos}$ for the end mark </s>, and $h_t$ for the prefix before step $t$ .

Definition 1 (Ancestral sampling).

Given a prompt $x$ , draw $Y_1 \sim p(\cdot \mid x)$ , then $Y_t \sim p(\cdot \mid x, Y_{\lt t})$ for $t = 2, 3, \dots$ , until $\mathrm{eos}$ is drawn.

Theorem 1 (Sampling is exact).

(i) For every finite string $y$ ending in $\mathrm{eos}$ , ancestral sampling returns $y$ with probability exactly $p(y \mid x)$ .

(ii) It halts with probability $\sum_y p(y \mid x)$ , so it halts almost surely if and only if the model is consistent.

Proof.

(i) Given the prefix $y_{\lt t}$ , the draw $Y_t$ has law $p(\cdot \mid x, y_{\lt t})$ . So the probability of drawing $y_1, \dots, y_n$ in order is the product of these conditional probabilities, $\prod_t p(y_t \mid x, y_{\lt t}) = p(y \mid x)$ , by the chain rule.

(ii) The events “the sampler returns $y$ ” are disjoint for different finite $y$ , and their union is the event that it halts. By (i), $\Pr(\text{halt}) = \sum_y p(y \mid x)$ .

∎

The neural-language-models topic proves that a softmax recurrent model with bounded state is consistent (its Theorem 4), so for such models (ii) is never an issue. The harness draws $20{,}000$ samples from the lab tree and from $30$ random trees, and tests each tree’s fit by $\chi^2$ . It also checks every leaf with an expected count of at least $5$ against $p(y)$ ; on the random trees the other $1{,}849$ of $3{,}630$ leaves are pooled into one $\chi^2$ cell. On the beam-search topic’s stationary model it checks that lengths are geometric.

Greedy decoding and sampling look like different algorithms. One fact about extreme-value noise shows that they differ only in what is added to the scores before taking a maximum.

Proposition 1 (Gumbel-max).

Let $z \in \mathbb{R}^V$ and let $G_w$ be independent standard Gumbel variables, with CDF $\exp(-e^{-g})$ . Then $\arg\max_w (z_w + G_w) \sim \operatorname{softmax}(z)$ , and $\max_w (z_w + G_w)$ is Gumbel with location $\log \sum_w e^{z_w}$ , independent of the argmax.

Proof.

Write $Z = \sum_w e^{z_w}$ . The density of $z_i + G_i$ at $g$ is $e^{z_i - g}\exp(-e^{z_i - g})$ , and $\Pr(z_j + G_j \le g) = \exp(-e^{z_j - g})$ . So

\Pr\big(\arg\max = i,\ \max \le m\big) = \int_{-\infty}^{m} e^{z_i - g} \exp\Big(-\sum_j e^{z_j - g}\Big)\, dg = \frac{e^{z_i}}{Z} \exp\big(-Z e^{-m}\big),

which factors into $\operatorname{softmax}(z)_i$ times the Gumbel CDF with location $\log Z$ at $m$ .

∎

So a draw from $p(\cdot \mid h)$ is the argmax of $\log p(\cdot \mid h)$ plus noise, and greedy decoding is the same argmax with the noise set to zero. The argmax half goes back to Yellott (1977); Maddison, Tarlow, and Minka (2014) state both halves. Kool, van Hoof, and Welling (2019) push the idea to whole sequences: perturb every prefix consistently, run beam search on the perturbed scores, and the top $k$ strings are an exact sample without replacement (their Theorem 1, Gumbel-top- $k$ ). So sampling and greedy decoding are one loop with a dial for per-step noise. That dial does not reach the mode: noise off gives greedy’s string, which is Theorem 3’s point.

Temperature as a Gibbs distribution

Definition 2 (Temperature).

For a next-token distribution with log-probabilities $z = \log p(\cdot \mid h)$ and $\tau \gt 0$ ,

p_\tau(w \mid h) \;=\; \frac{p(w \mid h)^{1/\tau}}{Z_\tau(h)} \;=\; \operatorname{softmax}(z / \tau)_w, \qquad Z_\tau(h) = \sum_w p(w \mid h)^{1/\tau}.

Figure A of the laboratory below shows one such distribution as $\tau$ slides. Near $\tau = 0$ the mass sits on the likeliest token; at large $\tau$ it spreads evenly over the support. Everything between is a single curve, and it has three exact properties.

Theorem 2 (Temperature is a Gibbs distribution).

For any $z \in \mathbb{R}^V$ and $\tau \gt 0$ :

(a) $p_\tau$ is the unique maximizer over the simplex of $\langle q, z\rangle + \tau H(q)$ , and the maximum is $\tau \log \sum_w e^{z_w/\tau}$ .

(b) $\dfrac{d}{d\tau} H(p_\tau) = \dfrac{\operatorname{Var}_{p_\tau}(z)}{\tau^3} \ge 0$ , with equality only when $z$ is constant on the support. The expected score $\mathbb{E}_{p_\tau}[z]$ is nonincreasing in $\tau$ .

(c) As $\tau \to 0$ , $p_\tau$ tends to the uniform distribution on $\arg\max z$ ; as $\tau \to \infty$ , to the uniform distribution on the support.

Proof.

(a) Write $\mathcal{L} = \log \sum_w e^{z_w/\tau}$ , so that $\log p_\tau(w) = z_w/\tau - \mathcal{L}$ . For any $q$ , $\langle q, z\rangle + \tau H(q) = \tau \mathcal{L} - \tau\, \mathrm{KL}(q \,\|\, p_\tau)$ . Gibbs’ inequality makes the KL term non-negative, and zero only at $q = p_\tau$ .

(b) Put $\beta = 1/\tau$ and $A(\beta) = \log \sum_w e^{\beta z_w}$ . This is a log-partition function, so $A'(\beta) = \mathbb{E}_\beta z$ and $A''(\beta) = \operatorname{Var}_\beta z$ . Since $H = A(\beta) - \beta A'(\beta)$ , we get $dH/d\beta = -\beta \operatorname{Var}_\beta z$ , and $d\beta/d\tau = -1/\tau^2$ turns this into $dH/d\tau = \operatorname{Var}(z)/\tau^3$ . Also $d\,\mathbb{E}_\beta z / d\beta = \operatorname{Var}_\beta z \ge 0$ .

(c) Divide numerator and denominator of $p_\tau(w)$ by $e^{\beta \max z}$ and let $\beta \to \infty$ or $\beta \to 0$ .

∎

Part (a) is the maximum-entropy principle of Jaynes (1957) with $-z$ as the energy: temperature prices entropy against expected score. Part (b) is formalStatistics’ log-partition theorem (its Theorem 7.1) applied to the family $\{p_\tau\}$ , an exponential family with sufficient statistic $\log p$ and natural parameter $1/\tau$ . Annealing walks along that one curve (Kirkpatrick, Gelatt, and Vecchi 1983), and Ackley, Hinton, and Sejnowski (1985) brought the word “temperature” into neural networks. The harness checks (a) against a numerical optimizer on $50$ random score vectors, and (b) as monotone entropy on a $200$ -point grid of $\tau$ for every trigram prompt.

Temperature also has a picture on the Zipf topic’s log–log plot.

Proposition 2 (Temperature is the Zipf slope).

For any distribution, the least-squares slope of $\log p_\tau$ against $\log(\text{rank})$ is the slope of $\log p$ divided by $\tau$ . A Zipf( $s$ ) distribution becomes a Zipf( $s/\tau$ ) distribution.

Proof.

Tempering preserves ranks, and $\log p_\tau = \tfrac{1}{\tau} \log p - \log Z_\tau$ . Least squares is linear in the response and ignores an added constant.

∎

So temperature divides the slope of the rank–frequency line, exactly, for one distribution. It moves the intercept too, because $\log Z_\tau$ shifts the whole line: on a Zipf( $1.1$ ) row over $60$ ranks, $\tau = 0.5$ takes the log-probability at rank $1$ from $-1.374$ to $-0.395$ , and the two lines cross near rank $2.4$ . Sampled text mixes many contexts, so its own rank–frequency slope only roughly follows $s/\tau$ .

Local against global temperature

In practice temperature is applied at every step. That is not the same as tempering the model.

Theorem 3 (Local temperature is not global temperature).

Let $q_\tau(y) = \prod_t p_\tau(y_t \mid h_t)$ be the per-step tempered sampler, and $\pi_\tau(y) = p(y)^{1/\tau}/\mathcal{Z}_\tau$ , with $\mathcal{Z}_\tau = \sum_y p(y)^{1/\tau}$ , the tempered sequence distribution, when that sum is finite.

(a) $q_\tau(y) = p(y)^{1/\tau} \big/ \prod_t Z_\tau(h_t)$ . So $q_\tau = \pi_\tau$ if and only if $\prod_t Z_\tau(h_t)$ takes the same value for every string in the support.

(b) With $\alpha = 1/\tau$ and $H_\alpha$ the Rényi entropy of order $\alpha$ ,

\log \frac{q_\tau(y)}{\pi_\tau(y)} \;=\; \log \mathcal{Z}_\tau + (\alpha - 1) \sum_t H_\alpha\big(p(\cdot \mid h_t)\big).

For $\tau \lt 1$ , local tempering upweights the strings whose prefixes carry more accumulated Rényi entropy.

(c) On a finite tree, if the mode and the argmaxes along greedy’s path are unique, then as $\tau \to 0$ , $q_\tau$ concentrates on greedy’s string and $\pi_\tau$ on the mode.

Proof.

(a) Substitute Definition 2 into the product; the numerators multiply to $p(y)^{1/\tau}$ .

(b) By definition $H_\alpha(p) = \frac{1}{1 - \alpha} \log \sum_w p_w^\alpha$ , so $\log Z_\tau(h) = (1 - \alpha) H_\alpha(p(\cdot \mid h))$ . Divide (a) by $\pi_\tau$ and take logarithms.

(c) Along greedy’s path, each $p_\tau(\cdot \mid h)$ tends to the point mass on that prefix’s unique argmax. The path has finitely many steps, so $q_\tau$ of greedy’s string tends to $1$ . Likewise $\pi_\tau \propto p(y)^{1/\tau}$ over finitely many strings tends to the point mass on the unique mode.

∎

The condition in (a) holds when, for instance, every string ends in a forced end mark at one fixed depth and every row before that is a permutation of one vector: those rows share one $Z_\tau$ , and the forced row contributes $1$ . The forced row has to be excepted, because the last row of a same-length string puts all its mass on $\mathrm{eos}$ . The harness checks the equality on such trees, and checks the identity in (b) on $30$ random trees at three temperatures. Part (c) asks only about greedy’s path because the lab tree has ties elsewhere, at “rose on” and “rose in”. On the lab tree, Figure C shows the two samplers parting as the dial cools. At $\tau = 0.1$ they are $0.925$ apart in total variation: the per-step sampler puts $0.705$ on greedy’s “rose sharply” while the tempered model puts $0.999$ on the mode “was flat”. Caccia and colleagues (2020, §3) and Shih, Sadigh, and Ermon (2023, §2.1) observe that per-token temperature differs from tempering the joint distribution, and Shih and colleagues write out the per-step normalizer (their Eqs. 2–3). What this section adds is the reading of that normalizer as accumulated Rényi entropy, and the remark that follows. The gap is a relative of the label bias of locally normalized models (Andor and colleagues 2016): both come from normalizing step by step, though label bias limits what a model can express, while this gap is in how a sampler reshapes a fixed one.

The tempered model need not exist. For $\tau \gt 1$ , $\mathcal{Z}_\tau$ can be infinite. Take the beam-search topic’s stationary model: $k$ words of probability $(1 - \varepsilon)/k$ each and $\mathrm{eos}$ of probability $\varepsilon$ , after every prefix. Then

\mathcal{Z}_\tau = \sum_{t \ge 0} k^t \Big(\frac{1 - \varepsilon}{k}\Big)^{t/\tau} \varepsilon^{1/\tau},

a geometric series with ratio $k^{1 - 1/\tau}(1 - \varepsilon)^{1/\tau}$ . For $k = 5$ and $\varepsilon = 0.05$ it diverges once $\tau \ge 1.032$ . The per-step sampler exists at every temperature; “the model at temperature $1.1$ ” does not.

Truncation: top-k, nucleus, min-p, η, and typical

Definition 3 (Truncation).

A truncation rule keeps a set $S(h)$ at each prefix and samples from $p_S(w \mid h) = p(w \mid h)\, \mathbf{1}[w \in S(h)] / m(h)$ , where $m(h) = p(S(h) \mid h)$ is the kept mass. With the probabilities sorted as $p_{(1)} \ge p_{(2)} \ge \cdots$ :

top- $k$ keeps the $k$ largest;
nucleus( $p$ ) keeps the shortest prefix of the sorted order whose mass reaches $p$ ;
min- $p$ ( $\rho$ ) keeps every $w$ with $p_w \ge \rho\, p_{(1)}$ ;
$\eta$ keeps every $w$ with $p_w \gt \min(\eta, \sqrt{\eta}\, e^{-H})$ , where $H$ is the row’s entropy;
typical sorts tokens by $|{-\log p_w} - H|$ and keeps them until their mass reaches the typical mass.

Ties go to the lower token id.

Fan, Lewis, and Dauphin (2018) popularized top- $k$ ; Holtzman and colleagues (2020) introduced the nucleus; Nguyen and colleagues (2025) min- $p$ ; Hewitt, Manning, and Liang (2022) $\eta$ ; and Meister and colleagues (2023) locally typical sampling, which we implement in the greedy form of their erratum.

Theorem 4 (What truncation costs).

(a) The nucleus has the fewest tokens of any set with mass at least $p$ , and the most mass among sets of its size. Top- $k$ has the most mass among sets of $k$ tokens.

(b) At one step, $\mathrm{TV}(p_S, p) = 1 - m$ and $\mathrm{KL}(p_S \,\|\, p) = -\log m$ , while $\mathrm{KL}(p \,\|\, p_S) = \infty$ whenever $m \lt 1$ .

(c) Over a sequence, for the truncated sampler $q$ ,

\mathrm{KL}(q \,\|\, p) = \mathbb{E}_q\Big[\sum_t -\log m(h_t)\Big], \qquad \mathrm{TV}(q, p) \le \mathbb{E}_q\Big[\sum_t \big(1 - m(h_t)\big)\Big],

and the probability that pure sampling never leaves the kept sets is $\mathbb{E}_q\big[\prod_t m(h_t)\big]$ .

(d) If every kept mass is at most $1 - \varepsilon$ , pure sampling reaches step $n$ inside the kept sets with probability at most $(1 - \varepsilon)^n$ . That is, the first $n$ draws happen and are all kept.

Proof.

(a) Any $k$ entries sum to at most the $k$ largest, so a set of mass at least $p$ has at least as many tokens as the shortest sorted prefix reaching $p$ .

(b) On $S$ , $p_S - p = p\,(1/m - 1) \ge 0$ , so $\mathrm{TV} = m\,(1/m - 1) = 1 - m$ , and $\mathrm{KL}(p_S \,\|\, p) = \sum_{w \in S} p_S(w) \log(1/m) = -\log m$ . In the other direction $p_S$ is zero where $p$ is not.

(c) The KL identity is the chain rule for KL: step $t$ contributes $-\log m(h_t)$ , averaged over the prefixes $q$ visits. For TV, couple the two samplers step by step with a maximal coupling; they agree until the first decoupling, which happens at step $t$ with probability $1 - m(h_t)$ given agreement so far, and a union bound finishes. For the last identity, $\sum_{y \in \operatorname{supp} q} p(y) = \sum_y q(y)\, p(y)/q(y)$ with $p(y)/q(y) = \prod_t m(h_t)$ .

(d) Let $A_n$ be the event in the statement. Given $A_{n-1}$ and its prefix $h_n$ , the $n$ -th draw is kept with probability $m(h_n) \le 1 - \varepsilon$ , so $\Pr(A_n) \le (1 - \varepsilon) \Pr(A_{n-1})$ . A string that halts before step $n$ is not in $A_n$ , and it has to be left out: if the root gives $\mathrm{eos}$ probability $0.9$ and only $\mathrm{eos}$ is kept, $0.9$ of draws never leave the kept sets.

∎

The harness checks (a) against every subset of $200$ random eight-token distributions, and (c) by exact enumeration on $30$ random trees under every rule. Parts (c) and (d) make Holtzman and colleagues’ “unreliable tail” exact, and over a whole sentence it is not rare. On the trigram, a pure sample stays inside the $0.95$ -nucleus at every step with probability $0.783$ , measured directly, or $0.775$ through the identity in (c) under nucleus sampling. So about one continuation in five draws at least one tail token. That figure comes from (c): a row’s nucleus can keep nearly all of its mass, so (d) gives no useful bound here.

A · The temperature dial — one next-token distribution as τ slides

temperature τ1.00

entropy H(p_τ), nats

1.461

log–log (Zipf) slope

-0.954

log–log slope at τ = 1

-0.954

expected log-probability E[log p]

-1.461

Entropy only rises as τ rises (Theorem 2), and the log–log slope is always the τ = 1 slope divided by τ (Proposition 2): temperature tilts Topic 1’s rank–frequency line, and its intercept moves too. Near τ = 0 the mass sits on the likeliest token; at large τ it spreads over all 3426 tokens.

B · Where each rule cuts — kept tokens, the cost of the cut, and the seen set

p0.9

tokens kept

4 of 3426

kept mass m

0.907

TV = 1 − m

0.093

KL(p_S ‖ p) = −log m

0.098

precision vs seen set

0.750

recall vs seen set

1.000

words seen after this context

Every cut costs exactly 1 − m in total variation and −log m in KL (Theorem 4). The Kneser–Ney row knows which words it saw after this context (Proposition 4), but the rules see only probabilities: no single setting of any rule keeps exactly the seen words on every prompt.

C · Local against global temperature — #14’s lab tree

temperature τ1.00

per-step sampler q_τ (temper every row)tempered model π_τ ∝ p(y)^(1/τ)

TV(q_τ, π_τ)

0.000

q_τ on greedy’s “rose sharply”

0.086

π_τ on the mode “was flat”

0.238

At τ = 1 the two coincide. Cool the dial and they part (Theorem 3): the per-step sampler collapses onto greedy’s string, the tempered model onto the mode.

D · One frontier — 188 prompts × 8 draws per setting

τ1

temperaturetop-knucleusmin-pηtypical

NLL under the model

2.348

NLL under the oracle

2.500

distinct-2

0.384

a training sentence verbatim

0.125

KL(q ‖ p) per word

0.000

mean words

7.900

Every rule lands on nearly the same curve: how much it cuts matters more than how. The held-out text (2.576 under the model, 1.968 under the oracle) is out of reach of these settings. At its own model NLL the curve runs well above it, partly because of rare words the oracle lumps together, and partly through the model’s error.

E · Boilerplate against fabrication — what lowering τ does to a fact the model half knows

temperature τ1.00

seen twice or more (30 probes)seen once (33 probes)never seen (48 probes)

seen twice or more: correct / wrong

0.727 / 0.198

seen once: correct / wrong

0.240 / 0.544

never seen: correct / wrong

0.064 / 0.699

Ashcombe Controls operates in the …seen twice or more (5×)KB: industrials

banking 0.008energy 0.008healthcare 0.008industrials 0.881retail 0.008technology 0.050

P(“industrials”) at τ = 1.00: 0.881

Copperleaf Markets operates in the …seen once (1×)KB: retail

banking 0.038energy 0.038healthcare 0.038industrials 0.283retail 0.161technology 0.248

P(“retail”) at τ = 1.00: 0.161

Brackenbury Lending operates in the …never seen (0×)KB: banking

banking 0.043energy 0.043healthcare 0.043industrials 0.323retail 0.043technology 0.283

P(“banking”) at τ = 1.00: 0.043

Lowering τ sharpens whatever the model believes. For contexts it saw twice or more, that removes the smoothing mass and with it every fabrication. For a context it saw once, the discounted count competes with the prior, and where the prior wins, a cold sampler states the wrong answer with certainty; on the way there, a little cooling raises fabrication before a lot of it lowers it. For contexts it never saw, cooling moves the mass on non-answers onto answers, and fabrication rises.

Truncation as desmoothing

Hewitt, Manning, and Liang (2022) read truncation as undoing smoothing. They model a language model as a mixture of a true distribution with a near-uniform smoothing distribution (their Eq. 3), argue for absolute and relative probability thresholds (§3.4), and describe recovering the true support (§4.4, App. A.2). The uniform case is exact.

Proposition 3 (Uniform desmoothing).

If $p = (1 - \lambda)p^\star + \lambda/|V|$ with $\lambda \lt 1$ , then $\{w : p_w \gt \lambda/|V|\} = \operatorname{supp} p^\star$ .

Proof.

$p_w = \lambda/|V|$ exactly when $p^\star_w = 0$ , and $p_w \gt \lambda/|V|$ otherwise.

∎

A Kneser–Ney model wears its smoothing openly, which lets us test the reading on a real decomposition.

Proposition 4 (Kneser–Ney desmoothing).

For interpolated Kneser–Ney with discount $D \lt 1$ and a context $h$ with $c(h) \gt 0$ ,

p(w \mid h) \;=\; \frac{\max(c(hw) - D, 0)}{c(h)} + \gamma(h)\, p_{\mathrm{lower}}(w \mid h^-),

and the seen set $\{w : c(hw) \gt 0\}$ is exactly $\{w : p(w \mid h) \gt \gamma(h)\, p_{\mathrm{lower}}(w \mid h^-)\}$ . The backoff term varies from word to word, so no single threshold on $p(w \mid h)$ recovers the seen set in general.

Proof.

This is the n-gram topic’s recursion. A seen word has $c(hw) \ge 1 \gt D$ , so its first term is positive; an unseen word has first term zero.

∎

On the $177$ prompt rows whose context the trigram has seen, the seen set averages $5.345$ words. On $0.847$ of those rows every seen word outranks every unseen one, so a threshold chosen for that row would recover it. But no setting of any of the five rules recovers the seen set on every row: each trades precision against recall, as Figure B shows on five prompts. Truncation rules see only probabilities, and probability alone does not say which mass is smoothing.

Typicality

Proposition 5 (Samples are typical of the model).

Under the model, $\mathbb{E}_p[-\log p(Y \mid x)] = \mathbb{E}_p\big[\sum_t H(p(\cdot \mid h_t))\big]$ , and $M_n = \sum_{t \le n} \big(-\log p(Y_t \mid h_t) - H(h_t)\big)$ is a martingale.

Proof.

Given $h_t$ , the surprisal $-\log p(Y_t \mid h_t)$ has conditional mean $H(h_t)$ , so every increment of $M_n$ has conditional mean zero. Taking expectations gives the identity.

∎

A sample’s NLL tracks its own path’s entropy, and on a finite vocabulary with a probability floor the increments are bounded, so formalStatistics’ Azuma–Hoeffding inequality (its Theorem 7) makes the concentration exponential. For i.i.d. and stationary ergodic sources this is the asymptotic equipartition property (Cover and Thomas 2006, §3.1 and §16.8). The harness confirms $\mathbb{E}[M] = 0$ within four standard errors on both engines.

Proposition 5 places the samples. It says nothing about the text, which is a cross-entropy: the text’s own entropy plus a penalty for the model’s error. On the trigram the three per-word NLLs are $0.835$ for the exact mode, $2.348$ for pure samples, and $2.576$ for the held-out text, and they come from three different pools. The modes cover all $188$ prompts, the samples are $8$ draws on each, and the text is the $417$ of $443$ sentences the trigram can score, on $182$ prompts at $1$ to $16$ sentences each. Give the samples the text’s weights and, over five random streams, they cost $2.26$ – $2.39$ nats per word against the text’s $2.576$ , and $20.1$ – $22.7$ nats per string on average against its $22.17$ . That mean hides the shape. The median string costs $12.2$ – $14.2$ nats against the text’s $20.0$ , and strings of $6$ to $12$ tokens cost $1.93$ – $2.07$ per word against its $2.54$ . At matched length the model finds the real text more surprising than its own output. The mode is nothing like a typical sample.

Definition 4 (Locally typical sampling).

Sort tokens by $|{-\log p_w} - H(h)|$ and keep them until their mass reaches the typical mass (Meister and colleagues 2023, Algorithm 5.3, greedy form).

Unlike the other rules, typical sampling can drop the head of a distribution: the likeliest token is also the most typical one on only $0.617$ of prompt rows, and on about $0.48$ of the steps along pure samples. So tightening this rule need not make text more predictable. On the trigram, on all five streams, cost is U-shaped in the typical mass, with means of $1.65$ , $1.50$ , and $1.74$ nats per word at masses $0.2$ , $0.5$ , and $0.9$ . On the decoder, masses $0.2$ and $0.5$ tie on the page’s training seed, and on three other seeds $0.2$ is a little cheaper, so the U is the trigram’s and not a law of the rule.

Quality against diversity

We now sample both of the beam-search topic’s engines, the Kneser–Ney trigram over $3{,}427$ words and the toy causal transformer over $509$ , $8$ times on each of the same $188$ held-out MD&A prompts, under $25$ settings of the six families. Both are renormalized at each step to never emit the unknown-word token. Each setting becomes one point in Figure D:

the samples’ per-word NLL under the model that generated them, which measures how far into its own tail the sampler goes;
their per-word NLL under an oracle, a Kneser–Ney model fit to five times more text from the generator that wrote the training corpus (the device Caccia and colleagues 2020 use on synthetic data).

Lower oracle NLL means more like the generator’s text, though it also rewards blandness: the trigram’s exact modes score $1.073$ . As $\tau$ rises from $0.4$ to $1.5$ , both NLLs and the share of distinct bigrams rise on both engines, and copying falls: over five random streams the mean copy rate falls at every step, though a single stream can tick up. On the lab’s stream the trigram copies a training sentence verbatim in $0.148$ of its finished samples at $\tau = 0.5$ and $0.024$ at $\tau = 1.5$ ; the decoder in $0.226$ and $0.059$ . “Finished” matters at the hot end: at $\tau = 1.5$ only $0.76$ – $0.78$ of the trigram’s draws finish within its $40$ -token cap, and about $0.94$ of the decoder’s, and the NLLs and copy rates count those alone. As any truncation rule loosens, its KL to the model falls.

The rules do not differ much. Every truncation setting lies within $0.1$ nats of the temperature curve at its own model NLL, on both engines. How much a rule cuts matters more than which rule cuts, which is Zhang and colleagues’ (2021) finding on large models seen in a small one. None of the settings reaches the held-out text, at $2.576$ under the trigram but only $1.968$ under the oracle. At that model NLL the temperature curve reads about $2.7$ , so there the text lies about three-quarters of a nat below it. Cold settings do go below $1.968$ on the oracle axis, but the curve reaches that cost at a model NLL of about $1.77$ . On each of the $25$ settings, oracle NLL minus model NLL is larger than the text’s $-0.607$ , and the smallest margin on a stream is $0.25$ – $0.29$ nats across five streams. That is a fact about the settings run, not about decoding in general: past the grid the margin reaches zero near $\tau = 1.75$ , and at $\tau = 2$ , where only about a third of draws finish, it is negative on every stream.

Not all of that distance is the model’s error. $6.4$ percent of the text’s tokens are outside the oracle’s vocabulary, against about $2$ percent of the samples’. The oracle scores them all as one unknown word, at $3.04$ nats each, and the trigram charges them $10.31$ . On the other tokens the text costs $2.05$ under the trigram and $1.90$ under the oracle, and it lies about $0.44$ nats below the temperature curve redrawn on those tokens, a mean over five streams against $0.75$ on all tokens. That part is the model’s error, and none of these settings removes it.

§ Finance case study: boilerplate against fabrication

An analyst who asks a model to continue a filing wants two things at once: language that reads like a filing, and facts that are true. Temperature is usually tuned for the first. Here is what it does to the second.

Boilerplate. Cooling the sampler makes it copy: at $\tau = 0.5$ the trigram reproduces a training sentence word for word in $0.148$ of its samples and the decoder in $0.226$ . That is the mild end of the beam-search topic’s finding that the mode is boilerplate.

Fabrication. We probe what a model knows with the corpus’s own fact templates: “Halvorsen Industrial operates in the …”, ”… is classified under …”, and ”… is headquartered in …”, scored against the synthetic knowledge base. The beam-search topic’s decoder does little better than chance: among the answers a relation allows, it ranks the right one first on $0.189$ of $74$ sector probes, against chance $0.167$ , and on $0.115$ of $26$ headquarters probes, against chance $0.125$ . The trigram cannot even see the company. An order-5 Kneser–Ney model, the n-gram topic’s model with a longer memory, can. We sort its $111$ probes by how often the probe’s four-word context occurred in training. So “seen” counts the context, not the fact: on $14$ of the $34$ never-seen sector probes, the same fact is in training under the sibling frame, and an n-gram cannot carry it across. Every number here is an exact sum over one next-token distribution; nothing is sampled.

Seen twice or more ( $30$ probes). The argmax is the right answer every time. At $\tau = 1$ the model still puts $0.198$ on wrong answers: that is its smoothing mass. Ashcombe Controls, seen five times, gets “industrials” with $0.881$ . As $\tau$ falls, or the nucleus narrows, fabrication falls to zero.
Seen once ( $33$ probes). One sighting adds about $0.124$ to the backoff’s prior, and the prior wins on $0.303$ of these probes. All ten failures are sectors, $10$ of $21$ , because the sectors’ prior has a favorite near $0.26$ ; the cities’ prior is flat, with a favorite near $0.08$ , and they fail on $0$ of $12$ . Copperleaf Markets is a retailer, but after one sighting the model gives “retail” $0.161$ against $0.283$ for “industrials”. At $\tau = 1$ the regime’s answers are right with probability $0.240$ and wrong with $0.544$ . Cooling to $\tau = 0.040$ raises the right answers to $0.698$ , but on the probes the prior won, fabrication goes to certainty: the wrong sector, stated every time. The way there is not monotone. Wrong answers rise from $0.544$ at $\tau = 1$ to a peak of $0.607$ near $\tau = 0.75$ , as mass on words that are no answer moves onto answers, are back to $0.551$ at $\tau = 0.5$ , and fall to $0.302$ at $\tau = 0.040$ . Copperleaf’s right answer peaks at $0.178$ near $\tau = 0.79$ before it collapses. So a little cooling raises fabrication here before a lot of it lowers it.
Never seen ( $48$ probes). Right with probability $0.064$ at $\tau = 1$ . No temperature, and no setting of top- $k$ , nucleus, min- $p$ , or $\eta$ , lifts that above $0.1$ . Typical sampling at mass $0.2$ reaches $0.191$ , with wrong answers at $0.692$ , only by dropping the favorite on every probe, and on contexts seen once the same setting cuts the right answers from $0.240$ to $0.064$ . Cooling is not neutral here either. Wrong answers rise from $0.699$ at $\tau = 1$ to $0.951$ at $\tau = 0.040$ , as the $0.238$ on words that are no answer at all moves onto answers.

Figure E traces all three regimes as the dial turns. Temperature and truncation sharpen what the model believes. Where it saw a context often, that removes the noise around the fact; where it saw the context once, sharpening can turn a hedge into a confident error; where it never saw the context, sharpening turns a hedge into a guess. Fluent, cold output is not evidence of knowledge.

Where this goes next

The calibration topic, next in this track, asks the question the finance case study raises: does a model’s probability mean anything? There temperature reappears with a different job, fitted to held-out data so that predicted probabilities match frequencies (Guo and colleagues 2017). The BLEU and ROUGE topic supplies the reference-based scores this page avoided.

Several decoding methods are outside our scope. Minimum Bayes risk decoding picks the sample with the highest expected similarity to the others. Contrastive search, classifier-free guidance, and speculative decoding change the objective or the speed. Mirostat (Basu and colleagues 2021) adjusts truncation online to hold a target perplexity.

Honest caveats

⚠ Rigor flag

Proved and cited. Theorems 1–4 and Propositions 1–5 are proved here. Kool and colleagues’ stochastic beam search, the asymptotic equipartition property, and the motivations for $\eta$ and typical sampling are cited.
Thresholds are heuristics. No value of $k$ , $p$ , $\rho$ , $\eta$ , or the typical mass is derived from the model’s error, and the desmoothing reading assumes a mixture form a neural model need not have.
Temperature at each step is not a tempered model (Theorem 3), and for $\tau \gt 1$ the tempered model may not exist.
Small, renormalized models. The trigram has $3{,}427$ words and the decoder $509$ ; both are sampled under a per-step renormalization that never emits the unknown-word token.
A proxy for quality. The oracle is a Kneser–Ney model fit to more synthetic text, with smoothing of its own, and it rewards blandness: the exact mode scores best of all.
A narrow test of fabrication. Three relation frames against a synthetic knowledge base, on an order-5 Kneser–Ney model, not a neural model’s factual recall.

Implementation

The companion notebook (notebookPath) imports sampling_temperature_nucleus.py, which owns every number on this page. That module imports the beam-search topic’s module for the lab tree, the trigram, the trained decoder, and the prompts; the Zipf topic’s module for the log–log slope; and the n-gram and perplexity topics’ Kneser–Ney code for the oracle and the order-5 fact model. It implements temperature, the five truncation rules, exact enumeration on finite trees, and a sampler that draws $1{,}504$ strings per setting from either engine, on five random streams and, for the decoder, four training seeds. The lab shows stream $0$ on seed $0$ . A harness of $36$ assertion-based tests runs in about two and a half minutes on a laptop CPU. The tests cover:

Theorem 1 by goodness of fit on $31$ trees, with the pooled leaves counted, and Proposition 1’s three claims on $200{,}000$ draws.
Theorem 2 against a numerical optimizer, its entropy derivative against a finite difference, and its limits on every trigram prompt; Proposition 2 exactly on every trigram prompt, and the moving intercept.
Theorem 3’s identity on $30$ random trees, its equality case on permutation trees, its limits and ties on the lab tree, and the divergence threshold.
Theorem 4 by brute force and exact enumeration under every rule, its tail identity on the trigram’s samples, and part (d) with its counterexample.
Propositions 3 and 4, including that no rule recovers the seen set; Proposition 5 on both engines, and the three pools on the text’s weights.
The frontier’s orderings, how often the head is typical, typical sampling’s cost on both engines, the copy rate and finished share over five streams, the text’s distance from the curve and its rare-word split, and every sampled string’s score against the engine.
The three fabrication regimes, the typical-sampling exception, the once-seen peak, cooling on never-seen contexts, what “seen” counts, and the decoder on the same probes.

The laboratory reads its data from a JSON file that one of those tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.

References & Further Reading

paper The Curious Case of Neural Text Degeneration — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2020) ICLR 2020. Nucleus sampling: the top-p vocabulary (their Eq. 2) and its rescaled distribution (Eq. 3), and the unreliable tail of Theorem 4(c)
paper Hierarchical Neural Story Generation — Angela Fan, Mike Lewis, and Yann Dauphin (2018) ACL 2018, 889–98. Sampling from the k = 10 most likely candidates (§5.4), the use that popularized top-k sampling
paper Truncation Sampling as Language Model Desmoothing — John Hewitt, Christopher D. Manning, and Percy Liang (2022) Findings of EMNLP 2022, 3414–27. The smoothing model (their Eq. 3), the absolute and relative probability principles (§3.4), η-sampling and support recovery (§4.4, App. A.2)
paper Locally Typical Sampling — Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell (2023) Transactions of the Association for Computational Linguistics 11: 102–21. The locally typical set (their Definition 3.5) and the sampler of Algorithm 5.3, in the greedy form of their erratum
paper Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv (2025) ICLR 2025. The threshold p_base · p_max of their §3.1
paper Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity — Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney (2021) ICLR 2021. A feedback controller that targets a perplexity, outside this page's scope
paper The Relationship between Luce's Choice Axiom, Thurstone's Theory of Comparative Judgment, and the Double Exponential Distribution — John I. Yellott Jr. (1977) Journal of Mathematical Psychology 15 (2): 109–44. The argmax half of Proposition 1
paper A* Sampling — Chris J. Maddison, Daniel Tarlow, and Tom Minka (2014) NeurIPS 27. The Gumbel-max trick and the independence of the maximum and the argmax (their Eqs. 1–2)
paper Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement — Wouter Kool, Herke van Hoof, and Max Welling (2019) ICML 2019, PMLR 97: 3499–508. Gumbel-top-k (their Theorem 1) and stochastic beam search
paper Information Theory and Statistical Mechanics — E. T. Jaynes (1957) Physical Review 106 (4): 620–30. The maximum-entropy reading of Gibbs distributions behind Theorem 2
paper Optimization by Simulated Annealing — S. Kirkpatrick, C. D. Gelatt Jr., and M. P. Vecchi (1983) Science 220 (4598): 671–80. Annealing: lowering the temperature of a Gibbs distribution toward its minimum
paper A Learning Algorithm for Boltzmann Machines — David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski (1985) Cognitive Science 9 (1): 147–69. Temperature in neural networks
paper Language GANs Falling Short — Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin (2020) ICLR 2020. Temperature sweeps as quality–diversity curves, oracle NLL on synthetic data, and the observation that per-token temperature differs from tempering the joint (§3)
paper Long Horizon Temperature Scaling — Andy Shih, Dorsa Sadigh, and Stefano Ermon (2023) ICML 2023, PMLR 202: 31422–34. Myopic temperature scaling is not the tempered joint distribution (their §2.1), with its per-step normalizer written out (their Eqs. 2–3)
paper Globally Normalized Transition-Based Neural Networks — Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins (2016) ACL 2016, 2442–52. Locally normalized models are strictly less expressive than globally normalized ones (their Theorem 3.1), the label-bias relative of Theorem 3
book Elements of Information Theory — Thomas M. Cover and Joy A. Thomas (2006) 2nd ed., Wiley. The asymptotic equipartition property (§3.1) and the Shannon–McMillan–Breiman theorem (§16.8) behind Proposition 5
paper Trading Off Diversity and Quality in Natural Language Generation — Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan (2021) Proceedings of the Workshop on Human Evaluation of NLP Systems, 25–33. Temperature, top-k, and nucleus compared along one quality–diversity curve
paper On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger (2017) ICML 2017, PMLR 70: 1321–30. Temperature scaling as calibration, the subject of the next topic