Positional Encodings: Sinusoids, Rotations, and RoPE

Overview & Motivation

The attention topic read attention as a kernel smoother: a weighted average over a set of keys. A set has no order, and that topic’s Proposition 3 drew the consequence. At the last word, one layer of attention reads the dog bit the man today exactly as it reads the man bit the dog today. A positional encoding supplies the order. The question this topic asks is what kind of object it supplies.

The answer we develop is that every positional encoding is a kernel over offsets: a rule for how much the distance between two positions should change their score. Then length generalization, the ability to read sequences longer than any seen in training, becomes a question about what that kernel does past the training window. Here is the experiment the topic builds toward. We train single attention heads, with each of five ways of adding position, to answer which marked token came last? in sequences of $32$ tokens, and read them on sequences of $256$ :

scheme	accuracy at 32	at 256
no position at all	$0.542$	$0.237$
sinusoidal code, added	$0.996$	$0.142$
RoPE (rotations)	$1.000$	$0.466$
RoPE, positions interpolated	$1.000$	$0.917$
ALiBi (a linear penalty on distance)	$0.999$	$0.978$

Each row is taken up below.

Sinusoids are a stationary kernel. The sinusoidal code turns the hands of $d/2$ clocks. Shifting every position is one rotation, and the inner product of two positions depends only on their offset. As $d$ grows it becomes a ruler that is logarithmic in distance.
Adding a stationary code does not make attention relative. Only a $d$ -dimensional family of score matrices sees offsets alone, and trained heads sit as far from it as random matrices. The added code collapses at twice its training length, because absolute positions it never saw enter the score, through this term and through the content–position terms.
ALiBi is a product kernel. Its weights are the attention topic’s smoother with a content kernel times an exponential kernel in time. One inequality says both why it extrapolates and why it cannot look far.
RoPE is relative by construction, and up to the learned projections it is the only linear encoding that is. Its score is a trigonometric polynomial in the offset, and past the training window the polynomial does what training never constrained it to do.

Then we take the heads to text. Synthetic company filings name a parent and its subsidiaries and then refer back to “the company”. Asked at every position which company was named most recently, heads trained on $32$ -token windows read whole filings. Past the training window, RoPE with interpolated positions reads $0.986$ of the answers, and ALiBi, which held at $0.978$ through the length tournament, reads $0.580$ .

Order, and how a mask leaks it

The attention topic’s Proposition 3 says bidirectional attention is permutation-equivariant, and that one causal layer, which reads only earlier positions, depends on its prefix only through the prefix’s multiset. A multiset has no order, but it has a size.

Proposition 1 (A causal layer counts but does not order).

(i) Suppose a causal attention layer’s scores are all equal, the value at position $0$ is a vector $e$ (a beginning-of-sequence token’s), and every other value is $0$ . Then its output at position $t$ is $o_t = e/(t+1)$ , so $t = 1/\langle o_t, e\rangle - 1$ is recovered exactly from $o_t$ when $\lVert e \rVert = 1$ .

(ii) In general, given the token at $t$ , one causal layer’s output at $t$ is a function of the multiset of its prefix. The multiset’s size is the position, and its order is invisible.

Proof.

For (i), equal scores make the weights uniform over the $t + 1$ positions $0, \dots, t$ , and only the value at position $0$ is nonzero. For (ii), the attention topic’s Proposition 3 (ii) gives the dependence on the multiset, given the token at $t$ , and the number of its elements is a function of it.

∎

So the causal mask itself leaks position, as Haviv and colleagues (2022) observed: a causal stack without any positional encoding can write absolute position into its first layer’s output and use it in the second (Kazemnejad et al. 2023 give a construction). A single causal head cannot. On the task of the overview, where the target is the symbol on the most recent marked token, a head without positions sees a bag of tokens. Every ordering of a bag is equally likely, so the best it can do is guess the most frequent marked symbol. That reader’s accuracy, computed exactly by counting, is $0.547$ at length $32$ and $0.234$ at $256$ ; the trained heads without positions reach $0.542$ and $0.237$ , matching it within sampling error at every length.

Position as a set of clocks

Let $d$ be even and split $\mathbb{R}^d$ into $d/2$ coordinate planes $j = 0, \dots, d/2 - 1$ . Give plane $j$ the frequency $\omega_j = b^{-2j/d}$ , for a base $b$ that is $10^4$ unless stated, so the frequencies fall geometrically from $\omega_0 = 1$ to nearly $1/b$ . Write $R(\varphi) = \bigl(\begin{smallmatrix} \cos\varphi & -\sin\varphi \\ \sin\varphi & \cos\varphi \end{smallmatrix}\bigr)$ for the rotation of a plane by the angle $\varphi$ , $J = R(\pi/2)$ , and $R_\omega(k) = \bigoplus_j R(k\omega_j)$ for the block-diagonal rotation that turns plane $j$ by $k\omega_j$ .

Definition 1 (Absolute additive encoding).

An absolute additive encoding adds a vector $p(m) \in \mathbb{R}^d$ to the embedding of the token at position $m$ : the layer reads $h_m = x_m E + p(m)$ . The code $p$ is either a learned table, one trained vector per position up to the training length and undefined beyond it, or a fixed function of $m$ .

Definition 2 (Sinusoidal code).

The sinusoidal code is

p(m) = \bigl(\cos m\omega_0,\ \sin m\omega_0,\ \cos m\omega_1,\ \sin m\omega_1,\ \dots,\ \cos m\omega_{d/2-1},\ \sin m\omega_{d/2-1}\bigr).

This is Vaswani and colleagues’ (2017) code. They put the sine first in each plane, which is a fixed reflection of every plane and changes no inner product. The attention topic’s circle features $(\cos\theta, \sin\theta)$ are its one-plane case. Picture plane $j$ as a clock whose hand points at angle $m\omega_j$ : the first hand turns one radian per position, the last almost not at all.

Proposition 2 (Shift is a rotation).

(i) For all integers $m$ and $k$ , $p(m + k) = R_\omega(k)\, p(m)$ .

(ii) The map $k \mapsto R_\omega(k)$ is a homomorphism from $\mathbb{Z}$ into the rotation group $SO(d)$ : $R_\omega(k) R_\omega(\ell) = R_\omega(k + \ell)$ and $R_\omega(k)^\top = R_\omega(-k)$ .

(iii) Every position lies on the same sphere, $\lVert p(m) \rVert^2 = d/2$ .

(iv) The code is injective: $p(m) = p(n)$ only if $m = n$ .

Proof.

For (i), the angle-addition formulas give $(\cos(m + k)\omega, \sin(m + k)\omega) = R(k\omega)(\cos m\omega, \sin m\omega)$ in each plane. For (ii), rotations of one plane compose by adding angles, and $R(\varphi)^\top = R(-\varphi)$ ; each block has determinant $\cos^2 + \sin^2 = 1$ . For (iii), each plane contributes $\cos^2 + \sin^2 = 1$ . For (iv), plane $0$ alone has $\omega_0 = 1$ , so $p(m) = p(n)$ forces $m - n$ to be a multiple of $2\pi$ , which for integers means $m = n$ , since $\pi$ is irrational.

∎

Vaswani and colleagues chose the code because, they wrote, a shifted position is a linear function of the original; (i) is that claim, and the linear function is a rotation that does not depend on $m$ . In the language of Linear Systems (its Example 5), $R_\omega(k) = \exp(kA)$ with the block-skew generator $A = \bigoplus_j \omega_j J$ . Since every $R_\omega(k)$ is orthogonal, the inner product of two positions cannot depend on where they are.

Theorem 1 (The stationary kernel, and the logarithmic ruler).

(i) For all $m$ and $n$ , $\langle p(m), p(n) \rangle = \kappa_d(n - m)$ , where

\kappa_d(\Delta) = \sum_{j=0}^{d/2-1} \cos(\Delta\omega_j).

Every Gram matrix $[\kappa_d(m_i - m_k)]$ is positive semidefinite, and Toeplitz over consecutive or evenly spaced positions, and $(2/d)\kappa_d(\Delta) = E\cos(\Delta\Omega_d)$ is the characteristic function of a random frequency $\Omega_d$ uniform on the $d$ points $\pm\omega_j$ .

(ii) Fix $b$ and $\Delta \ne 0$ . As $d \to \infty$ ,

\frac{2}{d}\,\kappa_d(\Delta) \;\to\; I_b(\Delta) = \frac{\operatorname{Ci}(\lvert\Delta\rvert) - \operatorname{Ci}(\lvert\Delta\rvert/b)}{\ln b},

where $\operatorname{Ci}(x) = -\int_x^\infty \cos t\, dt/t$ is the cosine integral, and

\Bigl\lvert I_b(\Delta) - \Bigl(1 - \frac{\gamma + \ln\lvert\Delta\rvert}{\ln b}\Bigr) \Bigr\rvert \le \frac{1}{\ln b}\Bigl(\frac{2}{\lvert\Delta\rvert} + \frac{\Delta^2}{4b^2}\Bigr),

with $\gamma$ Euler’s constant.

Proof.

For (i), angle subtraction gives $\cos m\omega \cos n\omega + \sin m\omega \sin n\omega = \cos((n - m)\omega)$ in each plane. A Gram matrix of vectors is positive semidefinite, since $\sum_{i,k} c_i c_k \kappa_d(m_k - m_i) = \lVert \sum_i c_i\, p(m_i) \rVert^2 \ge 0$ , and over consecutive or evenly spaced positions $m_i - m_k$ depends only on $i - k$ , which is the Toeplitz form. The characteristic-function form is the average of $\cos(\pm\Delta\omega_j)$ over the $d$ points.

For (ii), write $\Omega_d = \pm b^{-U_d}$ with a fair sign and $U_d$ uniform on the grid $\{0, 2/d, \dots, 1 - 2/d\}$ . The grid’s uniform law converges to the uniform law on $[0, 1]$ , so by continuous mapping $\Omega_d$ converges in distribution to $\Omega = \pm b^{-U}$ , $U$ uniform on $[0, 1]$ . Since $\omega \mapsto \cos(\Delta\omega)$ is bounded and continuous, $E\cos(\Delta\Omega_d) \to E\cos(\Delta\Omega)$ . Substituting $\omega = b^{-u}$ , so $du = -d\omega/(\omega \ln b)$ , and then $t = \lvert\Delta\rvert\omega$ ,

E\cos(\Delta\Omega) = \int_0^1 \cos(\Delta b^{-u})\, du = \frac{1}{\ln b}\int_{1/b}^1 \frac{\cos(\lvert\Delta\rvert\omega)}{\omega}\, d\omega = \frac{1}{\ln b}\int_{\lvert\Delta\rvert/b}^{\lvert\Delta\rvert} \frac{\cos t}{t}\, dt = I_b(\Delta).

For the bound, integration by parts gives $\operatorname{Ci}(x) = \sin x/x - \int_x^\infty \sin t\, dt/t^2$ , so $\lvert \operatorname{Ci}(x) \rvert \le 2/x$ . The integral form $\operatorname{Ci}(x) = \gamma + \ln x - \int_0^x (1 - \cos t)\, dt/t$ (DLMF 6.2.12–13), with $0 \le 1 - \cos t \le t^2/2$ , gives $\operatorname{Ci}(x) = \gamma + \ln x - \epsilon$ with $0 \le \epsilon \le x^2/4$ . Put $x = \lvert\Delta\rvert/b$ in the second and $x = \lvert\Delta\rvert$ in the first, and divide by $\ln b$ .

∎

Two readings. First, by Herglotz’s theorem (Brockwell and Davis 1991, §4.3), every positive-definite function of an integer offset is the characteristic function of some measure on the circle, so a stationary position code is a choice of spectral measure. The sinusoidal code chooses $d$ atoms at geometric frequencies, and Rahimi and Recht’s (2007) random features, which the attention topic set aside for positive ones, would sample them instead. Positive definiteness, stationarity, and the spectral picture are the ones formalML builds for Gaussian processes (Gaussian Processes: positive definiteness is its Definition 2, and the other two come later in that article).

Second, (ii) says the similarity of two positions is a logarithmic ruler: past a few positions it is a straight line in $\ln\lvert\Delta\rvert$ , and each doubling of the distance costs the same $\ln 2/\ln b \approx 0.075$ of similarity, because geometric frequencies give every octave of distance its own few planes. The notebook computes:

$\Delta$	$1$	$4$	$16$	$64$	$256$	$1024$
$d = 64$	$0.966$	$0.748$	$0.605$	$0.435$	$0.354$	$0.263$
$d = 512$	$0.973$	$0.768$	$0.631$	$0.485$	$0.313$	$0.186$
$d = 8192$	$0.974$	$0.771$	$0.635$	$0.487$	$0.335$	$0.185$
limit $I_b(\Delta)$	$0.974$	$0.772$	$0.635$	$0.487$	$0.335$	$0.185$
ruler $1 - (\gamma + \ln\Delta)/\ln b$	$0.937$	$0.787$	$0.636$	$0.486$	$0.335$	$0.185$

The convergence in $d$ is not uniform in $\Delta$ : at $\Delta = 256$ , $d = 512$ is farther from the limit than $d = 64$ is. Large offsets need many planes. Note also what the table says against a common claim. Stack the code’s vectors as the rows of a matrix $P$ . The Gram matrix of the positions is $PP^\top = [\kappa_d(m - n)]$ , not a multiple of the identity, which is what the table shows. The columns are a separate matter, and $P^\top P$ is not diagonal either: at $d = 16$ the largest cosine between two columns exceeds $0.999$ over $64$ positions and is still $0.739$ over $4096$ . The orthogonality of sines and cosines (Fourier Series, its Theorem 1) holds for integer harmonics over a full period; these frequencies are geometric, and a window is not a period.

The kernel is stationary, but attention does not compare positions directly. A head with query map $W_Q$ and key map $W_K$ scores position $t$ against position $j$ by $h_t M h_j^\top/\tau$ , with $M = W_Q W_K^\top$ and $\tau = \sqrt{d_k}$ . With $h = xE + p$ , the score is the sum of four terms, content against content, content against position, position against content, and $p(t) M p(j)^\top$ . Is that last term at least a function of the offset?

Theorem 2 (Which score matrices see only offsets).

Let $\omega_0, \dots, \omega_{d/2-1}$ be distinct numbers in $(0, \pi)$ . For $M \in \mathbb{R}^{d \times d}$ , the following are equivalent.

(a) $p(m)^\top M p(n)$ depends only on $n - m$ , for all integers $m$ and $n$ .

(a′) The same holds for all $m$ and $n$ in one window of $d + 1$ consecutive integers.

(b) $M$ commutes with $R_\omega(1)$ .

(c) $M = \bigoplus_j (a_j I + c_j J)$ : block-diagonal, with one rotation-scaling block per plane.

The matrices in (c) form a $d$ -dimensional subspace of the $d^2$ -dimensional space of score matrices. On it, $p(m)^\top M p(n) = \sum_j \bigl(a_j \cos(\Delta\omega_j) - c_j \sin(\Delta\omega_j)\bigr)$ with $\Delta = n - m$ , a trigonometric polynomial in the offset.

Proof.

Write $g(m, n) = p(m)^\top M p(n)$ and $R = R_\omega(1)$ . By Proposition 2, $g(m + 1, n + 1) = p(m)^\top R^\top M R\, p(n)$ . So $g$ depends only on $n - m$ , which is to say $g(m + 1, n + 1) = g(m, n)$ for all $m$ and $n$ , exactly when $p(m)^\top (R^\top M R - M) p(n) = 0$ for all $m$ and $n$ .

The vectors $p(m)$ , for any $d$ consecutive $m$ , span $\mathbb{R}^d$ . In complex coordinates their entries are $e^{\pm i m\omega_j}$ , and the matrix $[e^{i m \lambda}]$ over $d$ consecutive $m$ and the $d$ distinct points $e^{i\lambda} = e^{\pm i\omega_j}$ is a Vandermonde matrix, hence invertible. So the condition holds on a window of $d + 1$ positions, or on all of $\mathbb{Z}$ , exactly when $R^\top M R = M$ , that is, $MR = RM$ . This proves (a) ⇔ (a′) ⇔ (b).

For (b) ⇔ (c), $R$ has the $d$ distinct eigenvalues $e^{\pm i\omega_j}$ , so a matrix commuting with it preserves each of its two-dimensional real eigenplanes, which are the coordinate planes. $M$ is then block-diagonal, and its block $B_j$ commutes with the rotation $R(\omega_j)$ . Writing out $B_j R(\omega_j) = R(\omega_j) B_j$ with $\sin\omega_j \ne 0$ forces $B_j = \bigl(\begin{smallmatrix} a & -c \\ c & a \end{smallmatrix}\bigr) = aI + cJ$ . Conversely, $aI + cJ$ commutes with every rotation of its plane. The formula for the score follows by angle subtraction, since $(\cos\alpha, \sin\alpha)\, J\, (\cos\beta, \sin\beta)^\top = \sin(\alpha - \beta)$ .

∎

The theorem says that adding a stationary code to the content does not make attention relative. The position-against-position term is relative only if the learned $M$ lands in a $d$ -dimensional subspace of a $d^2$ -dimensional space, and the content-against-position terms are not relative at all (Ke, He, and Liu 2021). The relativity defect $\delta(M) = \lVert M - \Pi M \rVert_F / \lVert M \rVert_F$ , with $\Pi$ the orthogonal projection onto the subspace (keep each diagonal block’s $(\operatorname{tr} B/2) I + ((B_{21} - B_{12})/2) J$ , zero the rest), measures the miss. For a random Gaussian $M$ , $\lVert \Pi M \rVert^2 / \lVert M \rVert^2$ has a Beta distribution with mean $1/d$ , so $E\,\delta^2 = 1 - 1/d$ , and at $d = 16$ a random matrix has $\delta \approx 0.968$ . The three sinusoidal heads of the tournament, after training, have $\delta = 0.971$ , $0.955$ , and $0.973$ . Training barely moved them: at initialization they were $0.970$ , $0.956$ , and $0.979$ . Within the training window, nothing rewards relativity.

The attention topic’s trained circle denoiser is the opposite case. Its task was rotation-invariant, and it learned $M = cI$ , which lies in the commutant of a single plane. Exactness is also fragile in another way: over a window short against the slow planes’ wavelengths, the window’s vectors are nearly dependent. The $16$ -dimensional code over $64$ positions has numerical rank $14$ . Its exact condition number falls from $3.69 \times 10^{29}$ over $16$ positions, through $7.14 \times 10^{18}$ over $64$ , to $1.24 \times 10^{11}$ over $256$ and $17.7$ over $4096$ . The first two are past what double precision resolves, so the notebook computes them in $200$ -digit arithmetic. A window constrains a slow plane only once the window has watched it turn.

Relative position: biases and the ALiBi kernel

Since the score should depend on the offset, several schemes put the offset into the score directly.

Definition 3 (Relative-position schemes).

For a causal head scoring position $t$ against $j \le t$ , with offset $r = t - j$ :

Shaw, Uszkoreit, and Vaswani (2018) add a learned vector to the key, $s_{tj} = q_t (k_j + a_{\min(r, K)})^\top/\tau$ , one vector per offset up to a clip $K$ .
T5 (Raffel et al. 2020) adds a learned scalar $\beta_{\text{bucket}(r)}$ , with buckets that are exact for small offsets and logarithmically spaced up to $128$ .
ALiBi (Press, Smith, and Lewis 2022) subtracts a fixed linear penalty, $s_{tj} = q_t k_j^\top/\tau - s\,r$ , with slope $s = 2^{-8h/H}$ for head $h$ of $H$ .

T5’s buckets show the pattern of the section in miniature. With $32$ buckets, offsets below $16$ get their own bucket and the rest share log-spaced ones, the last of which holds every offset from $113$ on (the logarithmic map reaches it at $16 \cdot 8^{15/16} \approx 112.4$ , not at $128$ ). Offsets that far back are indistinguishable. Transformer-XL (Dai et al. 2019) puts relative terms into the score in yet another way. ALiBi is the cleanest, because its penalty multiplies the attention topic’s kernel by a second kernel. Our single head uses $s = 1/2$ , ALiBi’s first slope at $H = 8$ .

Theorem 3 (ALiBi is a product kernel).

Let one causal ALiBi head with slope $s \gt 0$ have content scores $c_{tj} = q_t k_j^\top/\tau$ lying in an interval of width $W$ . Write $\alpha_r$ for the weight on offset $r$ from a fixed query $t$ .

(i) $\alpha_{tj} \propto e^{c_{tj}} \cdot e^{-s(t - j)}$ : a Nadaraya–Watson smoother whose kernel is the product of a content kernel and an exponential kernel in time. In bidirectional form, $e^{-s\lvert t - j \rvert}$ is a positive-definite kernel, and so is its product with any positive-definite content kernel.

(ii) At every length, the weight on offsets $r \ge D$ is at most $e^{W} e^{-sD}/(1 - e^{-s})$ .

(iii) For every offset $r$ , $\alpha_r \le e^{W - sr}\, \alpha_0$ . A key farther back than $W/s$ gets less weight than the query’s own position, whatever its content.

Proof.

Part (i) is the softmax of $c_{tj} - s(t - j)$ . The bidirectional kernel $e^{-s\lvert u \rvert}$ is the Matérn kernel with smoothness $1/2$ , positive definite (Gaussian Processes, its Definition 3), and a product of positive-definite kernels is positive definite (that topic’s Proposition 2). The content kernel has to be one for this to apply. With $W_Q = W_K$ the scores are a Gram matrix and $e^{c_{tj}}$ is positive definite, but with $W_Q \ne W_K$ the matrix $[c_{tj}]$ is not even symmetric, and only the time factor is a kernel in this sense. For (iii), $\alpha_r/\alpha_0 = e^{c_r - c_0 - sr} \le e^{W - sr}$ , since the two content scores differ by at most $W$ . For (ii), sum (iii) over $r \ge D$ and use $\alpha_0 \le 1$ : the sum is at most $e^{W} \sum_{r \ge D} e^{-sr} = e^{W} e^{-sD}/(1 - e^{-s})$ .

∎

Parts (ii) and (iii) are one inequality read two ways. Read as (ii), it is a guarantee: no matter how long the sequence, the head’s weight stays within a band of offsets fixed by $W$ and $s$ , so its behavior at length $10^5$ is its behavior at length $100$ . That is why ALiBi extrapolates, and Chi and colleagues (2023) trace its length extrapolation to exactly such a bounded receptive field. Read as (iii), it is a limit: a key past the horizon $W/s$ is outweighed by the query’s own position, so the head cannot retrieve it on weight alone. The bound in (ii) is vacuous inside the horizon. With $W = 20$ and $s = 1/2$ it is $2.54$ at $D = 40$ , then $1.15 \times 10^{-4}$ at $60$ and $5.24 \times 10^{-9}$ at $80$ . Inside the horizon the content decides; past it the position does. In formalML’s terms, this is Nadaraya–Watson in time with bandwidth $1/s$ (Kernel Regression, its Definition 2).

Rotations: RoPE

Theorem 2 says an added code makes the score relative only if $M$ is special. Rotary position embedding removes the condition by moving the code from the inputs to the queries and keys, and from addition to rotation.

Definition 4 (Rotary position embedding (RoPE)).

With frequencies $\theta_j = b^{-2j/d}$ , rotate the query at position $m$ and the key at position $n$ by their positions:

\tilde q_m = R_\theta(m)\, q_m, \qquad \tilde k_n = R_\theta(n)\, k_n,

and score $\langle \tilde q_m, \tilde k_n \rangle / \tau$ . In complex form, each plane’s coordinates $z_j$ are multiplied by $e^{i m \theta_j}$ .

This is Su and colleagues’ (2024) RoPE. Its frequencies are Definition 2’s, so it is the sinusoidal code, applied by multiplication instead of addition.

Proposition 3 (RoPE is relative by construction, and isometric).

(i) $\langle R_\theta(m) q, R_\theta(n) k \rangle = \langle q, R_\theta(n - m) k \rangle$ .

(ii) $\lVert R_\theta(m) q \rVert = \lVert q \rVert$ .

(iii) For fixed $q$ and $k$ , the offset score $s(\Delta) = \langle q, R_\theta(\Delta) k \rangle$ is the trigonometric polynomial

s(\Delta) = \sum_j \rho_j \cos(\Delta\theta_j + \varphi_j),

with amplitudes $\rho_j = \lVert q_j \rVert\, \lVert k_j \rVert$ and phases $\varphi_j$ the angle from the component of $q$ in plane $j$ to that of $k$ .

Proof.

For (i), $R_\theta(m)^\top R_\theta(n) = R_\theta(-m) R_\theta(n) = R_\theta(n - m)$ by Proposition 2 (ii). Part (ii) holds because rotations are orthogonal. For (iii), in plane $j$ write $q_j = \lVert q_j \rVert (\cos\alpha, \sin\alpha)$ and $k_j = \lVert k_j \rVert (\cos\beta, \sin\beta)$ . Then $R(\Delta\theta_j) k_j$ points at angle $\beta + \Delta\theta_j$ , and its inner product with $q_j$ is $\lVert q_j \rVert \lVert k_j \rVert \cos(\Delta\theta_j + \beta - \alpha)$ .

∎

Part (ii) matters for the kernel reading. The attention topic factored the kernel $e^{\langle q, k \rangle/\tau}$ into a Gaussian in $\lVert q - k \rVert$ times design weights $e^{\lVert k \rVert^2/2\tau}$ (its Proposition 1). RoPE leaves every norm alone, so the design weights carry no position: RoPE moves only angles. A rotation applied to both vectors would preserve their inner product, as formalRAG uses for quantization (Optimized Product Quantization, its Theorem 1). RoPE applies different rotations to the two, and what survives is exactly the rotation by the offset.

Is RoPE one clever choice among many? The next theorem says it is the only linear one.

Theorem 4 (RoPE is the general linear relative encoding).

Let $L_m \in \mathbb{R}^{d \times d}$ , for $m \in \mathbb{Z}$ , with $L_0 = I$ . The following are equivalent.

(a) $\langle L_m q, L_n k \rangle$ depends only on $q$ , $k$ , and $n - m$ , for all $q$ , $k$ , $m$ , and $n$ .

(b) $L_m = U^m$ for some orthogonal matrix $U$ .

(c) There are an orthogonal $Q$ , angles $\theta_1, \dots, \theta_\ell \in (0, \pi)$ , and integers $p, p' \ge 0$ with

L_m = Q \Bigl( \bigoplus_{i=1}^{\ell} R(m\theta_i) \oplus I_p \oplus (-1)^m I_{p'} \Bigr) Q^\top.

Proof.

(a) ⇒ (b): The bilinear form $\langle L_m q, L_n k \rangle = q^\top L_m^\top L_n k$ determines the matrix $L_m^\top L_n$ , so (a) says $L_m^\top L_n = G(n - m)$ for some function $G$ . Taking $m = n$ gives $L_m^\top L_m = G(0) = L_0^\top L_0 = I$ , so every $L_m$ is orthogonal. Taking $m = 0$ gives $G(k) = L_k$ . Then $L_m^\top L_{m+k} = L_k$ , so $L_{m+k} = L_m L_k$ , and by induction $L_m = L_1^m$ for all $m$ , with $U = L_1$ .

(b) ⇒ (c): An orthogonal matrix is normal, and its real normal form (Horn and Johnson 2013, §2.5) is $U = Q (\bigoplus_i R(\theta_i) \oplus I_p \oplus (-I_{p'})) Q^\top$ with $Q$ orthogonal and $\theta_i \in (0, \pi)$ . Raise it to the $m$ -th power.

(c) ⇒ (a): $L_m^\top L_n = Q\,(\bigoplus_i R((n - m)\theta_i) \oplus I_p \oplus (-1)^{n-m} I_{p'})\, Q^\top$ , a function of $n - m$ .

∎

The orthogonal $Q$ is absorbed by the learned maps $W_Q$ and $W_K$ , so up to the projections every linear encoding that makes scores relative is RoPE with some frequencies. It may also leave planes unrotated, as partial rotary embeddings do, or flip a coordinate’s sign at every step, since $I_{p'}$ can have odd size. RoPE’s only free choice is the spectrum, and it takes Definition 2’s. Su and colleagues derive the two-dimensional case under further assumptions on the encoding; (b) needs none.

Su and colleagues also argue that RoPE gives a “long-term decay” of scores with distance. The argument is a bound, and it is worth seeing exactly what it bounds.

Proposition 4 (What long-term decay means).

In complex form, let $h_j = \bar q_j k_j$ for planes $j = 0, \dots, d/2 - 1$ , let $h_{d/2} = 0$ , and let $S_j = \sum_{l \lt j} e^{i\Delta\theta_l}$ . The offset score is $s(\Delta) = \operatorname{Re} \sum_j h_j e^{i\Delta\theta_j}$ .

(i) $\bigl\lvert \sum_j h_j e^{i\Delta\theta_j} \bigr\rvert \le \bigl(\max_j \lvert h_{j+1} - h_j \rvert\bigr) \sum_{j=1}^{d/2} \lvert S_j \rvert$ .

(ii) If $q = k$ and each plane has $E\lVert q_j \rVert^2 = \sigma^2$ , then $E\langle q, R_\theta(\Delta) q \rangle = \sigma^2 \kappa_d(\Delta)$ .

(iii) If $q$ and $k$ are independent, and the planes of $k$ are independent and each rotation-invariant in distribution, then the distribution of $\langle q, R_\theta(\Delta) k \rangle$ does not depend on $\Delta$ .

Proof.

For (i), sum by parts: $\sum_{j} h_j (S_{j+1} - S_j) = -\sum_{j} S_{j+1}(h_{j+1} - h_j)$ , since $S_0 = 0$ and $h_{d/2} = 0$ , and bound each term. For (ii), $u^\top R(\varphi) u = \lVert u \rVert^2 \cos\varphi$ in each plane, so the expectation is $\sum_j \sigma^2 \cos(\Delta\theta_j)$ . For (iii), independence lets each plane turn on its own, so $R_\theta(\Delta) k$ has the same distribution as $k$ , and it is independent of $q$ .

∎

The bound in (i) depends on $\Delta$ through the envelope $\sum_j \lvert S_j \rvert$ , which falls with distance on average. At $d = 128$ its mean over each octave of $\Delta$ falls from $2018$ at $\Delta = 1$ to $312$ over offsets $2048$ to $4095$ , or from $31.5$ to $4.9$ per plane, though not monotonically within an octave. That is a measurement, not a theorem. The independence in (iii) is needed: take $d = 4$ , $k_0 = k_1 = (\cos\beta, \sin\beta)$ with $\beta$ uniform, and $q = (1, 0, 1, 0)$ . Each plane of $k$ is rotation-invariant, but the score has variance $1 + \cos(\Delta(\theta_0 - \theta_1))$ . Parts (ii) and (iii) say what the envelope is about. A key that matches its query decays as Theorem 1’s logarithmic ruler, and a key unrelated to its query does not decay at all. “Long-term decay” is a property of envelopes and matched pairs, and a trained head decays only where it has learned to.

Past the training window

Now train. The task needs both content and order. Each sequence holds $n - 1$ tokens, each carrying one of $8$ symbols and, with probability $0.1$ , a mark; a query token ends the sequence, and the target is the symbol on the most recent marked token. One causal head of width $d = 16$ reads it at the query, with each of five schemes: no position, the added sinusoidal code, an added learned table, RoPE, and ALiBi. Each is trained from scratch in NumPy with hand-written gradients for $1{,}500$ Adam steps at length $32$ , three seeds per scheme, and tested on $2{,}000$ fresh sequences at each length. Accuracy, as the mean of three seeds:

scheme	32	64	128	256
no position	$0.542$	$0.367$	$0.286$	$0.237$
sinusoidal, added	$0.996$	$0.201$	$0.134$	$0.142$
learned table, added	$0.989$	—	—	—
RoPE	$1.000$	$0.947$	$0.872$	$0.466$
ALiBi	$0.999$	$0.979$	$0.978$	$0.978$

Every scheme but the first reads the task within the window. Past it, the learned table has nothing to say, and the added sinusoidal code collapses at twice the length: its trained score depends on absolute position, and the absolute positions are new. Theorem 2 accounts for one part of that dependence, not all of it. Replace only the position-against-position term with its projection onto the commutant, and the head still reads $0.202$ , $0.136$ , and $0.143$ at $64$ , $128$ , and $256$ , because the position-against-content term separates keys as strongly: at length $32$ their spreads across keys are within $6\%$ of each other in every seed. Training always put the query at position $31$ . Shift a long sequence so that the query sits there again, keeping every offset, and the head reads $0.769$ at $64$ , but only $0.313$ at $256$ , where older keys land on positions training never saw. ALiBi holds, as Theorem 3 (ii) predicts. RoPE degrades steadily, from $0.947$ to $0.466$ , and at $256$ its three seeds range from about $0.31$ to $0.68$ .

Why does a relative encoding fail to generalize in length? Because “relative” constrains the score to be a function of the offset, not to be a sensible one. Training sees offsets $0$ to $31$ only. By Proposition 3 (iii), the head’s score for a marked token at offset $r$ is a trigonometric polynomial, and training pins it down on $[0, 31]$ alone. In seed $0$ the score falls from $114.10$ at $r = 1$ to a first trough of $5.91$ at $r = 34$ , as the task rewards. Past the window it dives to $-55.86$ at $r = 90$ , and only then climbs, to $84.14$ at $r = 181$ , higher than the score of any recent marked token sixteen or more back. All three seeds behave alike: first troughs at $32$ to $34$ , deepest points of $-55.86$ , $-38.96$ , and $-19.39$ at $89$ to $90$ , peaks at $181$ to $182$ , and the largest amplitude in plane $3$ . The census in Theorem 5 explains the shape. Only planes $0$ and $1$ turn a whole circle in $32$ positions, so the head carries its sense of recency in planes $2$ and $3$ , whose wavelengths, $62.8$ and $198.7$ , are long enough that half a turn spans the window. A causal head reads the score at $\Delta = -r$ , so where a plane bottoms depends on its trained phase, not on its half wavelength of $31.4$ alone. Plane $2$ bottoms near $r = 27$ in seed $0$ , and plane $3$ , still falling, carries the sum to the first trough. One turn later plane $2$ bottoms again, near $r = 90$ , which is the dive. After three turns of plane $2$ ( $188.5$ positions) and one of plane $3$ ( $198.7$ ), the two come back toward phase, and both crest near $r = 180$ .

The fixes that practitioners use to extend RoPE’s context change the angles the head sees.

Definition 5 (Context-extension maps).

To read a head trained at length $L$ on length $L' = \kappa L$ :

Position interpolation (PI; Chen et al. 2023) maps positions $m \mapsto m/\kappa$ .
NTK-aware scaling (bloc97 2023) changes the base, $b \mapsto b\,\kappa^{d/(d-2)}$ .
YaRN (Peng et al. 2024) gives plane $j$ the frequency $(1 - \gamma_j)\theta_j/\kappa + \gamma_j\theta_j$ , where $\gamma_j$ rises linearly from $0$ to $1$ as $L/\lambda_j$ , the number of turns the plane made in training, goes from $1$ to $32$ . It also multiplies the scores by $(0.1 \ln\kappa + 1)^2$ .

Theorem 5 (Wavelength census and rescalings).

Let $\lambda_j = 2\pi/\theta_j$ be the wavelength of plane $j$ .

(i) Over the training offsets $0, \dots, L - 1$ , plane $j$ turns through a whole circle exactly when $(L - 1)\theta_j \ge 2\pi$ . Any other plane sees only the arc $[0, (L - 1)\theta_j]$ , and past $L$ it produces angles the head never saw.

(ii) PI maps test offsets $0, \dots, \kappa L - 1$ into $[0, L - 1/\kappa]$ , within one position of the trained range, and shrinks the angle between neighbors $\kappa$ -fold in every plane.

(iii) NTK-aware scaling gives plane $j$ the frequency $\theta_j \kappa^{-2j/(d-2)}$ : plane $0$ is unchanged, and the slowest plane is divided by exactly $\kappa$ .

(iv) For $a(\Delta) = \operatorname{Re} \sum_j h_j e^{i\Delta\theta_j}$ , which with Proposition 4’s $h_j$ is the offset score, and integers $\Delta_1 \lt \Delta_2$ , the linear interpolation $\ell$ of $a$ between them satisfies

\lvert a(\Delta) - \ell(\Delta) \rvert \le \tfrac{1}{2} (\Delta - \Delta_1)(\Delta_2 - \Delta) \sum_j \lvert h_j \rvert \theta_j^2 \quad\text{for } \Delta_1 \le \Delta \le \Delta_2,

and $\sum_j \theta_j^2 = (1 - b^{-2})/(1 - b^{-4/d})$ .

Proof.

Part (i): the angles $r\theta_j$ for $r = 0, \dots, L - 1$ sweep the arc $[0, (L - 1)\theta_j]$ , which is a whole circle exactly when its length reaches $2\pi$ . Part (ii): the largest test offset maps to $(\kappa L - 1)/\kappa = L - 1/\kappa$ , and neighbors differ by $\theta_j/\kappa$ in plane $j$ . Part (iii): $(b\kappa^{d/(d-2)})^{-2j/d} = b^{-2j/d} \kappa^{-2j/(d-2)}$ , which is $1$ times $\theta_0$ at $j = 0$ and $\theta_{d/2-1}/\kappa$ at $j = d/2 - 1$ . Part (iv): the interpolation remainder is $a(\Delta) - \ell(\Delta) = \tfrac{1}{2} a''(\xi)(\Delta - \Delta_1)(\Delta - \Delta_2)$ for some $\xi$ between $\Delta_1$ and $\Delta_2$ , and $\lvert a'' \rvert = \lvert \operatorname{Re} \sum_j h_j (i\theta_j)^2 e^{i\xi\theta_j} \rvert \le \sum_j \lvert h_j \rvert \theta_j^2$ . The sum $\sum_j b^{-4j/d}$ is geometric.

∎

At $d = 16$ and $L = 32$ the wavelengths are $6.3$ , $19.9$ , $62.8$ , $198.7$ , $628.3$ , $1987$ , $6283$ , and $19869$ , so two planes of eight complete a turn. PI keeps every angle in range at a cost in resolution. NTK-aware scaling keeps the fast planes’ resolution and interpolates the slow ones. At $\kappa = 4$ its per-plane factors run $1$ , $0.820$ , $0.673$ , $0.552$ , $0.453$ , $0.371$ , $0.305$ , and $0.250$ . YaRN, at this small $d$ , is nearly PI: its factors are $0.349$ , $0.265$ , then $0.25$ in every other plane.

Part (iv) is the reason interpolation is safer than extrapolation. Between trained positions the score is pinned by its curvature, and past them nothing pins it, so $\lvert a \rvert$ can reach $\sum_j \lvert h_j \rvert$ . Chen and colleagues (2023, their Theorem 2.1) state this bound with the constant $d/(4\ln b)$ in place of $\sum_j \theta_j^2$ . Their proof bounds $1/(1 - b^{-4/d})$ by $d/(4\ln b)$ using $c^x \le 1 + x\ln c$ for $x \lt 0$ , which is the reverse of what convexity gives, so the stated constant is too small. It is $2.56$ times too small at $d = 16$ and $1.15$ times at $d = 128$ . The stated bound then fails: for $h_j = 1$ on $[0, 1]$ at $\Delta = 1/2$ , the interpolation error at $d = 128$ is $0.459$ , while the stated bound is $0.434$ and the corrected one is $0.500$ . With the corrected constant, their conclusion, that interpolated scores are far more stable than extrapolated ones, stands.

Re-reading the trained RoPE heads with each map, with no further training:

RoPE head read with	64	128	256
nothing	$0.947$	$0.872$	$0.466$
PI	$0.998$	$0.980$	$0.917$
NTK-aware	$0.970$	$0.966$	$0.977$
YaRN	$0.998$	$0.979$	$0.914$

Each map rescues the head, and on this task NTK-aware scaling rescues it best at $256$ , where its $0.977$ ties ALiBi’s $0.978$ within sampling error on the same test sequences: it stretches the slow planes just enough to push the rebound past the longest sequence while keeping the fast planes sharp. The next section’s task ranks them differently.

Kernels over offsets, side by side

The laboratory has two figures. Figure A is the geometry. Drag two positions and watch the clocks of the sinusoidal code turn, then lock the offset and see the kernel stay put. The ruler view compares the kernel at three dimensions with its limit, and the last view shows a trained sinusoidal head’s score drifting with absolute position where a relative one would be flat. Figure B is the training. It shows the tournament with every scheme’s learned score against offset, the wavelength census with each rescaling’s factor per plane, and one held-out filing from the case study below, with every reader’s answer at every position.

A · Clocks and kernels — the sinusoidal code, the kernel it induces, and what a learned score does with it

position mm = 3

position nn = 10

hand at mhand at n

offset n − m

⟨p(m), p(n)⟩ / (d/2)

0.737

Plane j turns at rate b^(−2j/d): the first hand at one radian per step, the last almost not at all. Lock the offset and move either slider: every hand turns, each by its own rate, and the readout does not move. The inner product of two positions depends only on how far apart they are (Proposition 2 and Theorem 1).

B · Past the training window — heads trained at 32 tokens, read longer

NoPEsinusoidal (absolute)learned table (absolute)RoPEALiBiRoPE + PIRoPE + NTK-awareRoPE + YaRN

RoPE at 32

1.000

RoPE at 64

0.947

RoPE at 128

0.872

RoPE at 256

0.466

score at r = 1

114.10

score at r = 31, the window's edge

7.70

it rises again: highest score past the window

84.14 at r = 181

Training only sees offsets 0 to 31. The RoPE head's score is a trigonometric polynomial in r (Proposition 3): it falls across the window, as the task rewards, dives past it, and then the planes that never completed a turn bring it back up, until a cue 181 positions back outscores a recent one. ALiBi's score is a straight line down. The absolute codes' curves are drawn for a query at the end of a 256-token sequence; the learned table has nothing past 31.

Finance case study: which company was named last?

§ Finance case study: which company was named last?

A filing names a parent company, then its subsidiaries and competitors, and then refers back to “the company”. As a first approximation of which company is meant, take the one named most recently. It is only an approximation, and the question we score is the literal one: which company was named last? That is the cue task of the previous section, on synthetic documents shaped like filings, where mentions come in bursts and the gaps between them run long.

The documents are the shared synthetic corpus’s MD&A sections, earnings calls, and 8-K reports. We train on $300$ of them and test on $100$ held out, with a median length of $198$ tokens and $8$ company mentions. Each entity mention becomes one token. A company mention is marked, with the company’s order of first appearance in its document as its symbol, and every other word carries a hashed symbol. The heads are trained afresh on $32$ -token windows of the training filings, so at test time the only change is length. At every held-out position after the first mention we ask which company was named last: $19{,}285$ queries, of which $7{,}895$ , or $41\%$ , have their answer more than $32$ tokens back. Accuracy, as the mean of three seeds:

reader	all queries	answer more than 32 back
no position	$0.613$	$0.528$
sinusoidal, added	$0.762$	$0.712$
RoPE	$0.878$	$0.721$
ALiBi	$0.828$	$0.580$
RoPE + PI	$0.991$	$0.986$
RoPE + NTK-aware	$0.913$	$0.788$
RoPE + YaRN	$0.990$	$0.985$

ALiBi, which held at every length of the tournament, loses $42\%$ of the filing’s far queries. Theorem 3 (iii) says where. Each trained ALiBi head has a content width $W$ , which the notebook reads off its weights, and a horizon $W/s$ . Here is each head’s accuracy on the queries inside half its horizon and beyond it. We read heads trained on the filings, and also the tournament’s heads, trained on the synthetic stream:

trained on	horizon $W/s$	inside half of it	beyond it
filings, seed 0	$51.1$	$1.000$	$0.723$
filings, seed 1	$53.0$	$1.000$	$0.487$
filings, seed 2	$55.8$	$1.000$	$0.319$
synthetic, seed 0	$38.7$	$1.000$	$0.112$
synthetic, seed 1	$35.8$	$1.000$	$0.087$
synthetic, seed 2	$37.0$	$1.000$	$0.108$

Inside half the horizon every head answers every query, more than $8{,}000$ each. Beyond the horizon every head degrades, but by how much depends on its training. Theorem 3 bounds a weight, not an answer: a mention past the horizon still gets some weight, and whether that is enough depends on how little the nearer words’ values say. The heads trained on filings saw long stretches without a mention and learned to keep those values quiet; the heads trained on the dense synthetic stream did not.

One held-out filing, the one in the laboratory, shows the pattern in miniature. It names Marrow Freight six times, Dunmore Aerospace and Siskin Apparel once each, and at token $64$ Fenwick Logistics, never to be named again. Forty-five tokens later it reports that liquidity decreased, “a result the company called disappointing”. Ask there the literal question, which company was named last. In seed $0$ the heads without position, with the added code, with RoPE, and with ALiBi all answer Marrow Freight, the first company named. That answer is right on $76.6\%$ of all held-out queries and $73.9\%$ of the far ones, so a head that has lost the trail falls back on it. Seeds $1$ and $2$ split, and $5$ of the $12$ answers across seeds are Fenwick. RoPE with interpolated positions answers Fenwick Logistics in all three seeds, at all $145$ positions from token $64$ on, and it is right at every one of the filing’s $209$ positions. A person reading that clause would take “the company” to mean the filer, Marrow Freight. The task scores who was named last, and that is all it scores.

The EDGAR excerpts used elsewhere in this curriculum carry no entity annotations, so they sit this case study out.

Where this goes next

The transformer block. Proposition 1’s counter is one way a causal stack writes position into its residual stream, and a block runs many heads, which can carry many slopes and many frequencies at once. The transformer-block topic assembles them.
Contextual embeddings. BERT adds a learned table (Definition 1), which is undefined past its length. The contextual-embeddings topic trains with it.
Long contexts. formalRAG’s long-context topic models a passage buried in a long context as softly erased (its Proposition 3). Theorem 3’s horizon is one mechanism for such burial, a recency kernel under which far tokens weigh less than near ones, though not the only one, and this page does not model that topic’s U-curve.
Graphs and grids. On a cycle the sinusoids are the eigenvectors of the shift, and a stationary kernel is diagonalized by the discrete Fourier basis, as for the abelian Cayley graphs of formalML’s expander-graphs topic (its Proposition 2).

Honest caveats

⚠ Rigor flag

Long-term decay is about envelopes and matched pairs (Proposition 4). RoPE does not make unrelated keys fade.
The commutant theorem is exact but fragile. Theorem 2 holds in exact arithmetic on any window of $d + 1$ positions, but over a window shorter than the slow planes’ wavelengths its constraint on them is nearly void (exact condition numbers from $3.69 \times 10^{29}$ to $7.14 \times 10^{18}$ for windows of $16$ to $64$ positions at $d = 16$ , past what double precision resolves).
Three results are cited, not proved: Herglotz’s theorem (the converse reading of Theorem 1 (i)), the real normal form of orthogonal matrices (Theorem 4), and the integral form of the cosine integral (Theorem 1 (ii)).
The rescalings are heuristics. NTK-aware scaling and YaRN have no theorem behind their particular choices, and YaRN’s logit scale is a fitted constant. Every rescaling here is applied without fine-tuning, where practice fine-tunes after rescaling, and which one wins depends on the task: NTK-aware scaling on the synthetic stream, PI on the filings.
The tournament is a toy. It is one attention head of width $16$ , eight symbols, and three seeds. It shows mechanisms and measures nothing about language models at scale.
The filings are synthetic. The case study uses the shared synthetic corpus, with hashed symbols for ordinary words and first-appearance labels for companies. The EDGAR excerpts are left out, since they carry no entity annotations. The question scored, which company was named last, is a proxy for which company a clause refers to.
A common claim, corrected. Sinusoidal codes are often said to let a model extrapolate to unseen lengths and to have orthogonal columns. The added code here falls from $0.996$ to $0.201$ at twice its training length. The Gram matrix of its positions is the stationary kernel of Theorem 1, not a multiple of the identity, and its columns are not orthogonal either.
A published bound, corrected. Chen and colleagues’ Theorem 2.1 is false as stated for head dimensions up to $128$ , as Theorem 5 (iv) shows; with the corrected constant their conclusion holds.
The horizon bounds a weight, not an answer. Past Theorem 3’s horizon, how much an ALiBi head still reads depends on what the nearer values carry: $0.32$ to $0.72$ for the heads trained on filings, about $0.1$ for those trained on the synthetic stream.

Implementation

The companion notebook (notebookPath) imports positional_encodings_geometry.py, which owns every number on this page. That module imports the attention topic’s module for its attention function and its harness conventions, the neural-language-models topic’s gradient checker and Adam constants, and the shared finance corpus. It trains $27$ single-head models from scratch in NumPy, $15$ for the tournament and $12$ for the filings, and runs a harness of $24$ assertion-based tests in about half a minute on a laptop CPU. The tests cover:

Proposition 1 exactly, and the heads without positions against the best multiset reader, computed exactly.
Proposition 2 and Theorem 1 on random instances, the cosine-integral limit at $d = 8192$ , and the logarithmic ruler.
Theorem 2: the window’s null space at random frequencies, a random matrix’s defect, the trained heads’ defects before and after training, and the condition numbers in $200$ -digit arithmetic; what projecting one term and moving the query do to the added code.
T5’s buckets, and Theorem 3 on random instances up to length $8192$ .
Proposition 3 against the attention topic’s attention function; Theorem 4 through scipy’s real Schur form; Proposition 4 by Monte Carlo, including the counterexample that makes (iii) need independent planes.
Theorem 5’s census and rescalings, the interpolation bound on random instances, and the counterexample to the stated constant.
Every hand-written gradient against central differences; the tournament, the RoPE rebound through its dive, and the filings, including the one-pass filing reader against the plain forward pass at every position and the case study’s one filing.

The laboratory reads its data from a JSON file that one of those tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.

Overview & Motivation

Order, and how a mask leaks it

Position as a set of clocks

Relative position: biases and the ALiBi kernel

Rotations: RoPE

Past the training window

Kernels over offsets, side by side

Finance case study: which company was named last?

Where this goes next

Honest caveats

Implementation

References & Further Reading