Neural Scaling Laws and Compute-Optimal Training

Overview & Motivation

Train a small transformer on the shared filings with a fixed budget of compute, and choose how to spend it: a narrow model trained on many tokens, or a wide one trained on few. At each budget the losses form a valley. At $10^{10}$ FLOPs the valley’s floor sits near $7{,}100$ weights. At $2.5$ times that budget it sits near $10{,}800$ , and at $6.3$ times near $22{,}700$ . On log–log axes the floors rise with the budget, at a slope of $0.46$ and then $0.80$ , so three noisy points only suggest a line. Kaplan and colleagues (2020) found loss falling as a power law, with some trends spanning more than seven orders of magnitude. Hoffmann and colleagues (2022) measured the valleys on far larger models, and both papers turned the result into rules for sizing real models.

Lines on log–log axes are power laws. The usual form is $L \approx E + A/N^\alpha$ , a line only while the loss stays well above its floor $E$ , and this topic asks two questions about it. Where do the exponents come from? And how should a budget be split? We develop the answer in order:

Data scaling is Zipf’s tail. A memorizer must err when the next word is one it has not seen. Under the Pitman–Yor generator behind our corpus, that probability is an exact ratio of Gamma functions, which falls as a power of the data.
Model-size scaling is a spectrum’s tail. A linear model whose spectrum and target decay as powers of the coordinate index leaves a tail it cannot represent, a power of its size.
Training time has its own exponent. Gradient flow learns the large-eigenvalue directions first, and what remains decays as a power of time. For Gaussian data we compute SGD’s expected loss exactly.
Compute is $6N$ per token, plus attention. We count it by instrumenting the arrays of a real training step.
The optimal split has a closed form. A Lagrange multiplier on the budget gives the compute-optimal size, and the solvable model predicts its exponent, which the fitting procedures recover.
Kaplan and Chinchilla measured one law in two units. Counting only non-embedding parameters at small scale can turn an exponent near Chinchilla’s $0.5$ into one near Kaplan’s $0.73$ .

Then the finance case. A firm that trains on its own filings has a fixed amount of text. A few repeated epochs cost almost nothing, and the cap moves the optimum only when the corpus is far smaller than the law asks for.

Data scaling from Zipf: the missing mass

The simplest learner is a lookup table. It predicts a token’s type only if that type has occurred before, and it is wrong whenever the next type is new. That error has a name.

Definition 1 (Memorizer and missing mass).

After $n$ tokens $x_1, \dots, x_n$ , the missing mass is the probability that token $x_{n+1}$ has a type not among $x_1, \dots, x_n$ . A memorizer is any predictor that can be right only on types it has seen, so its error rate is at least the missing mass.

The shared corpus draws its content words from Pitman–Yor restaurants (the Zipf–Heaps topic, its Definition 4) with discount $d = 0.6$ and strength $\theta = 20$ . Customer $n + 1$ opens a new table, which is a new word type, with probability $(\theta + d K_n)/(\theta + n)$ , where $K_n$ is the number of tables after $n$ customers.

Theorem 1 (The missing mass under Pitman–Yor).

For a $\mathrm{PY}(d, \theta)$ sequence with $0 \le d \lt 1$ and $\theta \gt 0$ ,

\Pr[x_{n+1} \text{ is new}] \;=\; \frac{\theta + d\, \mathbb{E} K_n}{\theta + n} \;=\; \frac{\Gamma(\theta + 1)\, \Gamma(\theta + d + n)}{\Gamma(\theta + d)\, \Gamma(\theta + n + 1)} \;\sim\; \frac{\Gamma(\theta + 1)}{\Gamma(\theta + d)}\; n^{-(1 - d)}.

Proof.

Conditioned on the first $n$ draws, the probability is $(\theta + d K_n)/(\theta + n)$ , which is linear in $K_n$ ; taking expectations gives the first equality. For the second, $\mathbb{E} K_{n+1} = \mathbb{E} K_n + (\theta + d\, \mathbb{E} K_n)/(\theta + n)$ , so $m_n = \mathbb{E} K_n + \theta/d$ satisfies $m_{n+1} = m_n (\theta + n + d)/(\theta + n)$ with $m_0 = \theta/d$ . The product telescopes to $m_n = (\theta/d)\, \Gamma(\theta + n + d)\, \Gamma(\theta) / (\Gamma(\theta + d)\, \Gamma(\theta + n))$ , which is the Zipf–Heaps topic’s Theorem 3. Then $\theta + d\, \mathbb{E} K_n = d\, m_n$ , and dividing by $\theta + n$ gives the Gamma form. The asymptote is $\Gamma(z + d)/\Gamma(z + 1) \sim z^{d - 1}$ as $z = \theta + n \to \infty$ . At $d = 0$ the same argument gives $\theta/(\theta + n)$ .

∎

Over $1000$ simulated restaurants the share of sequences whose draw $1001$ is new is $0.194 \pm 0.013$ , against the exact $0.209$ , and the agreement holds within three standard errors at every checkpoint from $n = 10$ to $2000$ . The corpus’s text mixes seven restaurants with template words that stop growing early. Past ten thousand tokens its rate of new types falls with log–log slope $-0.407$ , against the theorem’s $-(1 - d) = -0.4$ .

The exponent is Zipf’s. A Pitman–Yor measure’s ranked frequencies decay as $i^{-1/d}$ , so its Zipf exponent is $s = 1/d$ . Hutter (2021, §3) shows that a lookup learner on data with Zipf exponent $s$ errs at a rate proportional to $n^{-(1 - 1/s)}$ , which is $n^{-(1 - d)}$ here. At $d = 0$ , the one-parameter Chinese restaurant of a Dirichlet process, the rate is $\theta/(\theta + n)$ and the exponent is $1$ . Hutter’s §6 remarks that the two-parameter process also leads only to exponent $1$ , in his continuous-feature extension. For the missing mass itself, Theorem 1 gives $1 - d$ . The n-gram topic estimates the same quantity without a model, by Good and Turing’s $n_1/n$ (its Theorem 2), and Theorem 1 is its Bayesian counterpart under the generator.

So the heavier the tail of the type distribution, the slower a memorizer learns. The power is the limit of an exact Gamma ratio: at $n = 100$ the exact value is $7\%$ below the asymptote, and at $n = 1000$ it is under $1\%$ below. With $d = 0.6$ , a tenfold increase in data cuts the missing mass by about $10^{0.4} \approx 2.5$ ( $2.49$ from $n = 1000$ to $10{,}000$ ), not $10$ .

Model-size scaling: the tail of a spectrum

A memorizer has no size: it remembers everything it sees. For a model of size $N$ we need a learner that can only represent part of the target. The cleanest one is linear.

Definition 2 (The solvable model).

Features are $x \sim \mathcal{N}(0, \Lambda)$ in $\mathbb{R}^M$ with $\Lambda = \mathrm{diag}(\lambda_k)$ and $\lambda_k = k^{-a}$ . The target is $y = \langle w^\ast, x \rangle$ with $\lambda_k (w^\ast_k)^2 = k^{-b}$ , for exponents $a \gt 1$ and $b \gt 1$ . A model of size $N$ fits $f(x) = \langle w, x_{1:N} \rangle$ , and its population loss is

L(w) \;=\; \sum_{k \le N} \lambda_k (w_k - w^\ast_k)^2 \;+\; \sum_{k = N+1}^{M} k^{-b}.

We take $a = 1.5$ , $b = 2.5$ , and $M = 8192$ .

Coordinate $k$ is a direction of the input with variance $\lambda_k$ , and the target puts $k^{-b}$ of its variance there. The first coordinates are both the loudest and the most important. Language need not be so tidy: a word like the is as loud as any and predicts little. The second sum is what a size- $N$ model cannot touch however long it trains.

Theorem 2 (The approximation tail).

For $1 \le N \lt M$ ,

\int_{N+1}^{M+1} x^{-b}\, dx \;\le\; \sum_{k = N+1}^{M} k^{-b} \;\le\; \int_{N}^{M} x^{-b}\, dx,

so as $M \to \infty$ the tail lies between $(N + 1)^{-(b-1)}/(b - 1)$ and $N^{-(b-1)}/(b - 1)$ .

Proof.

$x^{-b}$ is decreasing, so $(k + 1)^{-b} \le \int_k^{k+1} x^{-b}\, dx \le k^{-b}$ for every $k$ . Summing the right inequality over $k = N+1, \dots, M$ gives the lower bound, and summing the left one over $k = N, \dots, M - 1$ gives the upper.

∎

This is formalCalculus’ integral test (its Theorem 8) with the remainder kept. The harness checks the bracket on $50$ random $(b, N, M)$ , and the tail of our model falls with exponent $b - 1 = 1.5$ . A size exponent is therefore the exponent of a spectrum’s tail.

Data scaling in the same model: gradient flow and exact SGD

Now train the model from $w = 0$ . Gradient flow on half the population loss, $L/2$ , moves each coordinate on its own: $w_k(t) - w^\ast_k = -w^\ast_k\, e^{-\lambda_k t}$ . We take the half because it is the scale the SGD step below descends; flow on $L$ itself would run twice as fast. Coordinate $k$ is learned on the time scale $1/\lambda_k = k^a$ , so at time $t$ the directions with $k \lesssim t^{1/a}$ are done and the rest have barely moved. A frontier sweeps through the spectrum, and what is left behind it is a tail again.

Theorem 3 (The gradient-flow residual).

With every coordinate kept ( $N = M = \infty$ ) and $s = (b - 1)/a$ ,

\sum_{k \ge 1} k^{-b}\, e^{-2t k^{-a}} \;=\; \frac{\Gamma(s)}{a}\, (2t)^{-s}\, \big(1 + o(1)\big) \qquad (t \to \infty).

Proof.

Let $\varphi(x) = x^{-b} e^{-2t x^{-a}}$ . Substituting $u = 2t x^{-a}$ gives

\int_0^\infty \varphi(x)\, dx \;=\; \frac{1}{a}\, (2t)^{-s} \int_0^\infty u^{s - 1} e^{-u}\, du \;=\; \frac{\Gamma(s)}{a}\, (2t)^{-s}.

$\varphi$ rises and then falls, with maximum $(b/(2ta))^{b/a} e^{-b/a}$ at $x^a = 2ta/b$ . By the Zipf–Heaps topic’s lemma on sums of unimodal functions (its Lemma 1), the sum over $k \ge 1$ is within twice that maximum of the integral over $[0, \infty)$ . The maximum is $O(t^{-b/a})$ , which is $o(t^{-s})$ because $b/a \gt (b - 1)/a$ .

∎

At $t = 10$ the ratio of the sum to the asymptote is already $0.9999$ , and in our model the asymptote is $1/(3t)$ . One-pass SGD takes one fresh sample per step with step size $\eta$ . Its expected step is $-\eta \Lambda (w - w^\ast)$ , a gradient step on $L/2$ , so it follows this flow at time $t = \eta D$ after $D$ samples, plus noise. So the data exponent is $(b - 1)/a$ , and in our model it is $1$ . For Gaussian features the noise can be computed exactly.

Proposition 1 (One-pass SGD in closed form).

Let $e = w - w^\ast_{1:N}$ and $S_k = \mathbb{E}[e_k^2]$ , and let $\tau_N = \sum_{k \gt N} k^{-b}$ . One SGD step $w \leftarrow w - \eta\, x_{1:N}(\langle w, x_{1:N} \rangle - y)$ on a fresh sample gives

S_k \;\leftarrow\; \big(1 - 2\eta\lambda_k + 2\eta^2\lambda_k^2\big)\, S_k \;+\; \eta^2 \lambda_k \Big( \sum_{j \le N} \lambda_j S_j + \tau_N \Big),

and the expected loss after the step is $\sum_{k \le N} \lambda_k S_k + \tau_N$ .

Proof.

The residual is $\langle e, x_{1:N} \rangle - \xi$ , where $\xi = \sum_{k \gt N} w^\ast_k x_k$ is independent of $x_{1:N}$ with mean $0$ and variance $\tau_N$ . Expanding $e \leftarrow e - \eta\, x_{1:N}(\langle e, x_{1:N}\rangle - \xi)$ and taking expectations, the cross terms in $\xi$ vanish, and Isserlis’ theorem for Gaussian $x$ gives $\mathbb{E}[x x^\top A x x^\top] = 2\Lambda A \Lambda + \mathrm{tr}(\Lambda A)\, \Lambda$ for symmetric $A$ . With $A = \mathbb{E}[e e^\top]$ , the diagonal of the result depends on $A$ only through its diagonal $S$ and through $\mathrm{tr}(\Lambda A) = \sum_j \lambda_j S_j$ , which is the stated recursion.

∎

The off-diagonal entries of $\mathbb{E}[e e^\top]$ are not zero, but the loss never needs them. The recursion costs $O(N)$ per step, so the expected loss of every size at every step is cheap to compute, with no sampling. Against $400$ Monte Carlo runs it agrees within three standard errors. With $\eta = 0.25$ , a step costs $N$ multiply-adds, so here compute is $C = ND$ . That step is not small: $1/\mathrm{tr}\, \Lambda = 0.386$ , so $\eta\, \mathrm{tr}\, \Lambda = 0.65$ . The recursion’s fixed point shows what the noise does. There $S_k = \eta X / (2(1 - \eta \lambda_k))$ , with $X$ the loss, so $X = \tau_N / (1 - c_N)$ , where $c_N = \sum_{k \le N} \eta \lambda_k / (2(1 - \eta \lambda_k))$ . The recursion settles there while $c_N \lt 1$ and diverges past that, which for the full model means $\eta\, \mathrm{tr}\, \Lambda \approx 1.36$ , not $2$ . At our step the floor is $1.41$ times the tail at $N = 8$ and $1.58$ times at $N = 1024$ . So the noise rescales the size term and barely moves its exponent. Paquette and colleagues (2024) and Bordelon, Atanasov, and Pehlevan (2024) map the phases where it does more. We cite that analysis rather than prove it, and the harness’s recovered exponents are the evidence that we are in the simple phase.

Counting compute

The budget of a real model is counted in floating-point operations. A product of an $m \times k$ and a $k \times n$ matrix costs $2mkn$ FLOPs, one multiply and one add per term. Per token, a weight matrix is used once in the forward pass and twice in the backward pass: once to send the gradient back to its input, and once to form its own gradient.

Proposition 2 (The FLOPs of a training step).

Take the toy decoder of this topic: $L$ pre-LN blocks of width $d$ (the transformer-block topic, its Definition 1) with causal attention over a context of $n$ tokens, and a tied $V \times d$ embedding that is also the unembedding. Let $N = 12 L d^2 + V d$ count the weights that enter matrix products. One training step on $B$ windows costs exactly

B n \,\big( 6N + 12 L n d \big)

matrix-multiply FLOPs.

Proof.

Each block’s weights are $12 d^2$ : the query, key, value, and output maps ( $4 d^2$ ) and the MLP’s $d \times 4d$ and $4d \times d$ maps ( $8 d^2$ ). The tied unembedding adds one $d \times V$ product per token. Each weight costs $2$ FLOPs per token forward and $4$ backward, which is $6N$ . The scores $QK^\top$ and the weighted sum $AV$ cost $2nd$ each per token per layer forward, and the backward pass takes four products of the same size: $4nd + 8nd = 12nd$ per layer. The embedding lookup and its gradient are a gather and a scatter, not products.

∎

We check this by counting, not by reading the code. The harness wraps every parameter in an array type that adds $2k$ FLOPs for each output entry of each matrix product it takes part in, then runs one real forward and backward pass. At all seven widths the count equals $Bn(6N + 12Lnd)$ to the integer. $N$ leaves out the biases and the position table, which take part in no matrix product: at $d = 12$ it is $9{,}600$ of $10{,}152$ parameters. The attention term is not small at toy scale. Its ratio to $6N$ is exactly $2Ln/(V + 12Ld)$ , here $128/(512 + 24d)$ , so it falls as the model widens: $19.5\%$ at $d = 6$ and $7.7\%$ at $d = 48$ . Of all the compute, those are $16.3\%$ and $7.1\%$ .

Kaplan and colleagues’ Table 1 and Eq. 2.2 write the forward cost as $2N + 2 n_{\text{layer}}\, n_{\text{ctx}}\, d$ . That is one context-sized term per layer, where the scores and the weighted sum cost that much each, and $N$ is the non-embedding count, which leaves out the unembedding’s $2Vd$ . At small widths the unembedding is most of the compute. Porian and colleagues (2024) list this last-layer cost as the first of three corrections that move Kaplan’s exponent toward Chinchilla’s.

Compute-optimal allocation

Fix a budget $C$ and a loss law $L(N, D)$ . A model too small for the budget saturates at its own tail, and a model too large sees too few tokens to learn anything. Between them is a valley, and its floor is the compute-optimal size.

Theorem 4 (The compute-optimal split).

Let $L(N, D) = E + A N^{-\alpha} + B D^{-\beta}$ with $A, B, \alpha, \beta \gt 0$ . On the budget $6ND = C$ the unique minimizer is

N^\ast = G \Big(\frac{C}{6}\Big)^{\beta/(\alpha + \beta)}, \qquad D^\ast = G^{-1} \Big(\frac{C}{6}\Big)^{\alpha/(\alpha + \beta)}, \qquad G = \Big(\frac{\alpha A}{\beta B}\Big)^{1/(\alpha + \beta)},

the frontier is $L^\ast(C) = E + K C^{-\alpha\beta/(\alpha+\beta)}$ for a constant $K$ , and the budget’s Lagrange multiplier is $\nu = -dL^\ast/dC = \frac{\alpha\beta}{\alpha + \beta}\, (L^\ast - E)/C$ .

Proof.

Put $D = C/(6N)$ and $u = \log N$ . Then $f(u) = E + A e^{-\alpha u} + B (C/6)^{-\beta} e^{\beta u}$ is a sum of a constant and two strictly convex exponentials, so it has at most one critical point. $f'(u) = 0$ reads $\alpha A N^{-\alpha} = \beta B D^{-\beta}$ . At the optimum, the two reducible terms stand in the ratio $\beta : \alpha$ . Substituting $D = C/(6N)$ and solving gives $N^\ast$ and $D^\ast$ . Then $A N^{\ast -\alpha}$ and $B D^{\ast -\beta}$ are both constant multiples of $(C/6)^{-\alpha\beta/(\alpha+\beta)}$ , which gives the frontier, and differentiating it gives $\nu$ . By the envelope theorem $\nu$ is the multiplier of the constraint.

∎

The exponent $a = \beta/(\alpha + \beta)$ splits each extra order of magnitude of compute between size and data. With $a = 0.5$ they grow together, and the number of tokens per parameter is constant. Read through formalML’s Lagrangian-duality topic, $\nu$ is the sensitivity theorem’s shadow price (its Theorem 7): the loss one more FLOP buys at the optimum. The harness checks the closed form against a numeric minimum and $\nu$ against a numeric derivative on $30$ random laws.

For the solvable model, Theorems 2 and 3 suggest $\alpha = b - 1$ and $\beta = (b - 1)/a$ , and Theorem 4 then predicts

a^\ast \;=\; \frac{\beta}{\alpha + \beta} \;=\; \frac{1}{1 + a},

whatever $b$ is. This is a prediction, not a corollary, because the model’s loss is not exactly additive. With $a = 1.5$ it says $N^\ast \propto C^{0.4}$ : the spectrum’s decay alone decides how compute should be split. (Here $C = ND$ , since a step costs $N$ , and the constant factor does not change the exponent.)

Hoffmann and colleagues estimated $a$ three ways. Approach 1 takes, at each budget, the size whose training curve is lowest. Approach 2 fits a parabola in $\log N$ to each budget’s valley and takes its vertex. Approach 3 fits the five parameters of the law to every run at once, minimizing a Huber loss of the log-residuals with $\delta = 10^{-3}$ by L-BFGS from a grid of starting points. That is a robust M-estimator, the subject of formalStatistics’ method-of-moments topic (its Theorem 11). On the solvable model’s exact loss over $15$ sizes and $49$ data sizes, the three approaches give $a = 0.403$ , $0.408$ , and $0.405$ , against $0.4$ . Approach 3’s exponents are $\hat\alpha = 1.523$ and $\hat\beta = 1.035$ , against $1.5$ and $1$ .

Other spectra test the prediction. Approach 1 gives $0.403$ at $(a, b) = (1.5, 2)$ and $0.382$ at $(1.5, 3.5)$ , against $0.4$ ; $0.440$ at $(1.25, 2.5)$ , against $0.444$ ; $0.340$ at $(2, 3)$ , against $0.333$ ; and $0.250$ at $(3, 4)$ , against $0.25$ , once the data grid runs to $2^{20}$ . On our usual grid that last spectrum gives $0.374$ , because its optima run off the end. All of them land within $0.02$ .

The law recovers the exponents, but it is not the model’s loss surface. Its root-mean-square log error over the grid is $0.128$ , and its worst point, at $N = 8$ and $D = 64$ , is off by a factor of $e^{0.400} \approx 1.49$ . The fit overstates the loss there, $0.073$ against $0.049$ , for two reasons. About $55\%$ of the gap is the data term: a size- $N$ model’s data term sums only the $N$ coordinates it keeps, so a small model finishes learning sooner than the additive form allows. The rest is the fitted size term, $0.0377$ against a true tail of $0.0268$ . It sits on SGD’s floor at this size, $0.0378$ , which the run has not reached at $D = 64$ . A law that recovers its exponents can still misdescribe the region it was not built for.

The published fits. Hoffmann and colleagues print $E = 1.69$ , $A = 406.4$ , $B = 410.7$ , $\alpha = 0.34$ , and $\beta = 0.28$ (their Eq. 10), so $a = 0.452$ . Besiroglu and colleagues (2024) recovered the unrounded values from the paper’s source, with $\alpha = 0.3392$ and $\beta = 0.2849$ , giving $a = 0.4565$ . At Gopher’s budget of $5.76 \times 10^{23}$ FLOPs, the printed values give $N^\ast = 32.2$ B parameters. The unrounded ones give $40.3$ B, which rounds to the paper’s $40$ B for Approach 3. Besiroglu and colleagues then refit the law to data reconstructed from the paper’s figures and found $a = 0.513$ . That puts $N^\ast$ at $72.2$ B and about $18$ tokens per parameter, beside Approaches 1 and 2 (67B and 63B) and Chinchilla’s own ratio of $20$ .

Toy transformers on the filings

The theorems above live in models we chose. To ask whether text behaves the same way, we train transformers. The decoders have two pre-LN blocks of width $d$ , a learned position table, and a tied embedding over the $511$ commonest types of the training text plus [UNK]. They read windows of $32$ tokens from $3500$ synthetic filings, about $736$ thousand tokens, of which the last $65{,}536$ are held out. Adam’s peak rate is $1.28/d$ , a one-over-width rule fixed on a small sweep, with $10\%$ warmup and a linear decay that ends at each run’s own horizon. Hoffmann and colleagues stress that the schedule must match the horizon, and Kaplan and colleagues’ did not. Each of three budgets, $C = 6ND \in \{10^{10}, 2.5 \times 10^{10}, 6.3 \times 10^{10}\}$ , trains five widths between $6$ and $48$ .

Every budget’s losses form a valley with its minimum inside the grid. The parabola vertices sit at $N^\ast \approx 7{,}130$ , $10{,}830$ , and $22{,}720$ weights, so Approach 2 gives $a = 0.630$ . Approach 3 on all fifteen runs gives $a = 0.694$ , with $\hat\alpha = 0.403$ , $\hat\beta = 0.914$ , and an irreducible $\hat E = 0.772$ nats.

One seed says little. We reran all fifteen with four more seeds. Each valley stays interior, but a point’s standard deviation runs from $0.017$ to $0.105$ nats, with median $0.031$ . That is small against the valleys’ depths but not against the differences between neighboring widths near a floor. (Retraining the middle run at $d = 12$ with two seeds gives only $0.010$ , which understates it.) The published Approach 3 exponent is the highest of the five, which run from $0.47$ to $0.69$ , and $\hat E$ runs from $0.77$ to $1.25$ . Approach 2 is steadier, from $0.53$ to $0.63$ , and on the five-seed mean losses it gives $0.58$ , with slopes of $0.49$ and $0.67$ between budgets.

Three budgets spanning a factor of $6.3$ cannot pin down an exponent. In total parameters ours is about $0.58$ , against Chinchilla’s $0.5$ . In non-embedding parameters, Kaplan’s count, it is about $0.66$ , against Kaplan’s $0.73$ . The honest reading is that the grid cannot separate the two.

A · Power laws — Zipf’s tail for data, a spectrum’s tail for size

model size N64

exact expected loss (Proposition 1)fitted E + A/N^α + B/D^β (Approach 3)

exact loss at D = 65,536

0.00196

fitted α (Theorem 2: b − 1 = 1.5)

1.523

fitted β (Theorem 3: (b − 1)/a = 1)

1.035

Against D, each size’s curve falls with slope near −1 until it reaches the tail its N coordinates cannot touch, then flattens; bigger models flatten later and lower. The fitted law tracks the large models closely and overstates the small ones: a size-N model’s data term sums only its N kept coordinates.

B · Splitting a budget — IsoFLOP valleys, Theorem 4, and Kaplan’s units

budget C = N·D2.62 × 10⁵

loss at each size on this budgetlowest size (Approach 1)parabola vertex (Approach 2)

lowest size (Approach 1)

128

parabola vertex (Approach 2)

119.9

Theorem 4: N* ∝ C^a, a = 1/(1 + a_λ)

0.400

a from Approaches 1 / 2 / 3

0.403 / 0.408 / 0.405

Left of the floor the model is too small and sits on its tail; right of it the model is too big for its data. Every factor of 16 in compute moves the floor right by about 16^0.4 ≈ 3.

C · A finite corpus — repeated epochs and the capped optimum

repeated, three-seed meanwhole stream, D′ tokens readtoy fit at Muennighoff’s effective data D′ (R* = 15.4)

Four epochs cost 0.019 nats, and sixty-four cost half a nat. The top row draws random windows from the whole stream, 0.65 passes of it. At 16 and 64 epochs the repeated run beats a run that reads D′ tokens of the whole stream, so here a repeated token is worth more than D′ says, though that run also takes fewer steps.

unique tokens U126.2B

optimal parameters N*

46.2B

epochs over the corpus

4.94

predicted loss (Besiroglu et al.’s fit)

2.0162

unconstrained loss

2.0121

Above the unconstrained D* the cap does not bind. Just below it, a few repeated epochs are nearly free. Far below it, more passes stop paying, and the compute goes into a bigger model.

Kaplan against Chinchilla

Kaplan and colleagues found $N^\ast \propto C^{0.73}$ (their Eq. 6.1), and Hoffmann and colleagues found $C^{0.5}$ . The two papers differ in schedules, data, and scale, but one difference is a matter of definition. Kaplan counted only non-embedding parameters $N_{\text{ne}}$ , and compute as $6 N_{\text{ne}} D$ .

Proposition 3 (The same optimum in Kaplan's units).

Suppose $N^\ast \propto C^a$ in total parameters along a family of models, and let $\rho = d \log N_{\text{ne}} / d \log N$ along the family. Then the non-embedding optimum, measured against $C_{\text{ne}} = 6 N_{\text{ne}}^\ast D^\ast$ , has local exponent

\frac{d \log N_{\text{ne}}^\ast}{d \log C_{\text{ne}}} \;=\; \frac{\rho a}{\rho a + 1 - a}.

If the embedding grows as $N_{\text{ne}}^{1/3}$ , then $\rho$ rises from $1$ toward $3$ as the embedding’s share grows, and the exponent rises toward $3a/(1 + 2a)$ .

Proof.

Along the optimum, $d \log N^\ast = a\, d\log C$ and $d \log D^\ast = (1 - a)\, d \log C$ , since $N^\ast D^\ast \propto C$ . So $d \log N_{\text{ne}}^\ast = \rho a\, d \log C$ , and $d \log C_{\text{ne}} = d \log N_{\text{ne}}^\ast + d \log D^\ast = (\rho a + 1 - a)\, d \log C$ . Divide. For $N = N_{\text{ne}} + c N_{\text{ne}}^{1/3}$ and $x = c N_{\text{ne}}^{-2/3}$ , $\rho = (1 + x)/(1 + x/3)$ , which increases from $1$ at $x = 0$ to $3$ as $x \to \infty$ , and the exponent is increasing in $\rho$ .

∎

A model family of fixed shape has $d_{\text{model}} \propto N_{\text{ne}}^{1/3}$ , so its embedding, $(V + n_{\text{ctx}})\, d_{\text{model}}$ , grows as $N_{\text{ne}}^{1/3}$ . Take GPT-2’s vocabulary of $50{,}257$ and context of $1024$ , Kaplan’s setting, with $d_{\text{model}} = 64\, n_{\text{layer}}$ , and size the models by Besiroglu and colleagues’ fit ( $a = 0.513$ ). Below $10^{18}$ FLOPs, where the embedding is between $40\%$ and $99\%$ of the model, the non-embedding exponent is $0.705$ . By $10^{23}$ FLOPs the embedding share has fallen below $1\%$ , and the exponent is back to $0.514$ . The local exponent agrees with Proposition 3 to $2 \times 10^{-3}$ at every budget, and the limit is $3a/(1 + 2a) = 0.759$ . This is a calculation about units, not a measurement. It extrapolates Besiroglu and colleagues’ fit far below its range, since Hoffmann and colleagues’ IsoFLOP runs start at $6 \times 10^{18}$ FLOPs, and its smallest rows are not real models: at $10^{15}$ FLOPs the family has $0.71$ of a layer. Still, this is Pearce and Song’s (2024) reconciliation, with the closed form behind it: most of Kaplan’s experiments ran where embeddings dominated, so Kaplan’s $0.73$ and Chinchilla’s $0.5$ can be the same law. Proposition 3 says nothing about which count is the right one. On our toy grid the same reading moves Approach 2’s exponent from $0.58$ to $0.66$ on the five-seed mean losses, and from $0.630$ to $0.706$ on the published seed.

§ Finance case study: a finite filing corpus

A bank or an exchange that trains a model on its own documents cannot buy more of them. The filings, transcripts, and research notes that exist are its unique tokens $U$ , and if the law wants more data than that, the model must see some of it twice. Muennighoff and colleagues (2023) fit how much a repeated token is worth. $D$ tokens drawn from $U$ unique ones count as

D' \;=\; U + U R^\ast \big(1 - e^{-R/R^\ast}\big), \qquad R = D/U - 1,

fresh tokens, with $R^\ast = 15.4$ . A few epochs are nearly free ( $D' \approx D$ while $R \ll R^\ast$ ), and value saturates at $U(1 + R^\ast)$ however long training runs.

On our filings. We take the middle budget’s $d = 12$ model and spend the same compute, $434{,}176$ tokens read, five ways. The first draws random windows from all $670{,}082$ training tokens, which is $0.65$ passes, not a corpus of $434{,}176$ tokens. (A true corpus of that size does no worse: $2.268$ nats against $2.287$ , over three seeds.) The others restrict the windows to $4$ , $16$ , $64$ , and $256$ times fewer unique tokens than that. The last column reads $D'$ tokens of the whole stream, a control with no fit in it. The losses are three-seed means, except the column labeled seed $0$ .

epochs	unique	seed $0$	mean	fit at $D'$	$D'$ tokens read
$0.65$	$670{,}082$	$2.298$	$2.287$	$2.301$	—
$4$	$108{,}544$	$2.315$	$2.306$	$2.326$	$2.312$
$16$	$27{,}136$	$2.357$	$2.371$	$2.471$	$2.449$
$64$	$6{,}784$	$2.827$	$2.777$	$3.236$	$3.417$
$256$	$1{,}696$	$4.806$	$4.832$	$6.499$	$4.961$

Four epochs cost $0.019$ nats, under one run’s seed deviation of $0.022$ . Sixty-four cost half a nat, because the model sees the same $6{,}784$ tokens about sixty-four times, and two hundred fifty-six cost two and a half. The toy fit at Muennighoff’s $D'$ overstates every repeated run, and at $256$ epochs it says $6.499$ , worse than a uniform guess at $\ln 512 = 6.238$ , so it has left its range.

The control agrees at four epochs and is behind at sixteen and sixty-four, so on these filings a repeated token is worth more than $D'$ says. It is not a clean control. It takes fewer optimizer steps, $107$ against $424$ at sixty-four epochs, and at this scale a repeated token still buys a step. Its random windows also repeat some tokens, $25\%$ of those read at $395$ steps and $8\%$ at $107$ . One explanation is that the filings come from templates, so even their unique text repeats its phrasing, and a repeated window may lose less than a repeated real document would. We have not tested that.

At the scale of a finance model. BloombergGPT (Wu and colleagues 2023) was sized by Chinchilla’s Approaches 1 and 2. At their planned compute budget, assuming one pass, its authors found their roughly $700$ billion tokens “too small for a ‘Chinchilla optimal’ configuration”. They trained $50.6$ B parameters and stopped at $569$ B of the $709$ B tokens available, about $80\%$ of an epoch, when the held-out loss stopped improving. So those two numbers are the compute spent, not the plan. Spend that compute, $6 \times 50.6 \times 10^9 \times 569 \times 10^9$ FLOPs, under Besiroglu and colleagues’ fit. Unconstrained, the optimum is $38.97$ B parameters on $739$ B tokens, slightly more data than exists. With the cap and Muennighoff’s decay, the optimum runs $1.04$ epochs and grows to only $39.10$ B parameters. At the $1111$ B tokens Approach 1 asked for, about twice that compute, the same fit gives $54.9$ B parameters free and $56.6$ B capped on $1.40$ epochs. The cap binds when the corpus is far smaller than the law’s $D^\ast$ . With $126$ B unique tokens the optimum is a $46$ B-parameter model on five epochs. With $20$ B it is a $77$ B-parameter model on nineteen, because once more passes over the data stop paying, the compute goes into the model. The lab’s last figure runs this sweep.

What the law cannot tell a firm is whether its documents are worth more than general text. Muennighoff’s decay treats every repeated token alike, and BloombergGPT’s mixture of $363$ B financial and $345$ B general tokens is a separate decision that no scaling law here prices.

Where this goes next

The losses on this page are cross-entropies, so everything in the perplexity topic about what a nat buys applies to the frontier: a loss gain of $0.1$ nats cuts perplexity by a factor of $e^{-0.1}$ , about $9.5\%$ , whatever the scale. The decoding topics take a trained model as given and ask how to generate from it. The fine-tuning topic asks how few new parameters adapt a pretrained one, which is a scaling question in a regime this page does not touch.

Three things are outside our scope. Emergent abilities on downstream tasks are not losses and need not follow a power law. Compute spent at inference, by sampling longer or more, has its own trade-offs. And the variance-limited regimes of Bahri and colleagues (2024), where a model is large enough to interpolate, belong with formalML’s double-descent topic, whose risk curves have the peak that our monotone laws leave out.

Honest caveats

⚠ Rigor flag

Theorems inside toy models only. Theorems 1–3 hold for a Pitman–Yor memorizer and a Gaussian linear model with power-law spectrum. Theorem 1’s exact form is only asymptotically a power, and $a^\ast = 1/(1 + a)$ is a prediction, tested on six spectra. For real language models the law $E + A/N^\alpha + B/D^\beta$ is a fit.
The law is misspecified even where it fits. On the solvable model it recovers the exponents within $4\%$ and misses the loss by $13\%$ RMS, up to a factor of $1.49$ at small sizes.
SGD’s noise is cited, not proved. Proposition 1 is exact for every step size, but the claim that our step leaves the exponents alone rests on Paquette and colleagues’ and Bordelon and colleagues’ analyses and on the recovered exponents.
The published fits disagree. Hoffmann and colleagues’ approaches give $0.46$ to $0.50$ ; their printed parameters give 32B at Gopher’s budget where their text says 40B; Besiroglu and colleagues’ refit gives $0.51$ , and they argue the original intervals are far too narrow.
Units matter, and Proposition 3 does not pick them. Kaplan’s $0.73$ is consistent with Chinchilla’s law read in non-embedding units at small scale. That explains the gap, not which count predicts better.
6ND drops attention. The attention term is $8\%$ to $20\%$ of $6N$ at our widths, about $7\%$ to $16\%$ of all the compute, and Kaplan’s accounting drops the unembedding as well.
The toy grid is small. Three budgets span a factor of $6.3$ , a point’s seed deviation runs from $0.017$ to $0.105$ nats, and one seed’s Approach 3 exponent runs from $0.47$ to $0.69$ . The grid cannot separate Kaplan from Chinchilla.
Repetition on synthetic text. Muennighoff and colleagues’ $R^\ast$ was fitted at large scale on real text. On our filings a repeated token is worth more than it predicts, by the fit and by a control with no fit in it, and we have not tested why.
Extrapolation is a bet. Every law here is supported only over the range it was fitted on.

Implementation

The companion notebook (notebookPath) imports scaling_laws.py, which owns every number on this page. That module imports the transformer-block topic’s module for the pre-LN block, attention, LayerNorm, and their backward passes. Through it, it also gets the neural-language-models topic’s Adam constants and the shared finance corpus. It imports the contextual-embeddings topic’s module for the faster MLP backward pass, and the perplexity topic’s module, which brings the Zipf–Heaps topic’s Pitman–Yor sampler and closed form. It computes the solvable model’s SGD losses exactly, trains $102$ decoders from scratch in NumPy with hand-written gradients, and runs a harness of $24$ assertion-based tests in about two and a half minutes on a laptop CPU. The tests cover:

Theorem 1 against $1000$ simulated restaurants, its Dirichlet case, its asymptote at three $(d, \theta)$ , and the corpus’s own learning curve.
Theorem 2’s bracket on $50$ random instances, and Theorem 3’s bracket against the exact incomplete-gamma integral on $20$ .
Theorem 3’s clock against small-step SGD, and Proposition 1 against $400$ Monte Carlo SGD runs.
Proposition 1’s fixed point and its divergence past $\eta\, \mathrm{tr}\, \Lambda \approx 1.36$ .
Theorem 4 against a numeric minimum on $30$ random laws, with its multiplier against a numeric derivative.
The three approaches’ recovery of the predicted exponent, Approach 1 on five more spectra, and the law’s misfit split into its two causes.
Proposition 2 by instrumented counting at seven widths, and the decoder’s gradient against central differences.
The published fits’ exponents and optima, Proposition 3 against finite differences, the toy grid’s valleys at all five seeds, the seed spread, the repetition runs with their no-fit control, and the capped optimum.

The laboratory reads its data from a JSON file that one of those tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.

Overview & Motivation

Data scaling from Zipf: the missing mass

Model-size scaling: the tail of a spectrum

Data scaling in the same model: gradient flow and exact SGD

Counting compute

Compute-optimal allocation

Toy transformers on the filings

Kaplan against Chinchilla

Where this goes next

Honest caveats

Implementation

References & Further Reading