Attention as Kernel Smoothing

Overview & Motivation

The neural-language-models topic gave a recurrent network a long-lag task: read a bit, read $60$ noise symbols, then report the bit. At initialization the gradient that reaches the bit from the final loss is $2.45 \times 10^{-10}$ for an Elman network, because it passes through $60$ step Jacobians and its Theorem 5 says how fast their product shrinks. Give the same task to one layer of attention, which reads every earlier position directly, and the same census gives $3.76 \times 10^{-5}$. Trained by the same protocol at lag $60$, the attention layer reports the bit correctly on every fresh sequence, at lag $60$ and at lag $600$.

That topic ended with a promise: attention replaces the recurrent state’s compression with a weighted lookup over every earlier position, and this topic reads that lookup as kernel smoothing. The reading is exact, and it explains the experiment above and several others.

1. Attention is a kernel smoother. Softmax attention is the Nadaraya–Watson estimator of nonparametric regression, with the kernel $e^{\langle q, k \rangle / \tau}$. That kernel is a Gaussian in the distance between query and key, tilted toward long keys.
2. The temperature is a bandwidth. At low temperature attention is a lookup, at high temperature an average. On keys spaced around a circle the smoother is a Fourier low-pass filter whose gains are ratios of Bessel functions, so its bias–variance trade-off is a closed form, and a trained attention layer finds its optimum.
3. The $1/\sqrt d$ scale keeps random dot products from saturating the softmax, whose Jacobian vanishes as it saturates.
4. A lookup’s gradient takes one step. It is diluted by the number of positions competing for weight, not decayed through a product of Jacobians. Between lags $10$ and $480$ the gradient reaching the bit falls from $2.22 \times 10^{-4}$ to $4.88 \times 10^{-6}$ for attention, and from $1.88 \times 10^{-4}$ to $9.41 \times 10^{-46}$ for the Elman network.

Then the lookup goes to work on text. A kernel over a document’s own earlier words, mixed with the n-gram topic’s Kneser–Ney, reads across sentence boundaries that no earlier model on this site could see. On three real 10-K excerpts it cuts Kneser–Ney’s code from $5.660$ to $4.498$ bits per character, mostly by copying words the training text never contained, though lzma, at $2.908$, still codes them shorter.

---

A weighted lookup

Rows are tokens, as in the neural-language-models topic. The inputs $x_1, \dots, x_n \in \mathbb{R}^p$ are mapped to queries $q_t = x_t W_Q$, keys $k_j = x_j W_K$ in $\mathbb{R}^{d_k}$, and values $v_j = x_j W_V$ in $\mathbb{R}^{d_v}$. Position $t$ may read the positions $\mathcal{J}_t$: all of them in bidirectional attention, and $j \le t$ in causal attention, the kind a language model uses. Write $V_{\max} = \max_j \lVert v_j \rVert$.

Before attention, there was regression. Given design points $k_j$ and responses $v_j$, a kernel smoother predicts at a new point by averaging the responses, each weighted by how close its design point is.

This is Nadaraya’s (1964) and Watson’s (1964) estimator, whose bias, variance, and optimal bandwidth formalML develops (Kernel Regression, its Definition 2 and Theorems 1–2).

Bahdanau, Cho, and Bengio (2015) introduced attention as a soft lookup over every source position. Vaswani and colleagues (2017) gave the scaled dot-product form. Put the two definitions side by side and the first proposition is nearly immediate. Its content is in what the kernel turns out to be.

Tsai and colleagues (2019) read attention as a kernel smoother. The factorization in (ii) says which kernel smoother it is. The temperature sets a squared bandwidth, so the default bandwidth grows like $d_k^{1/4}$. The keys’ lengths act as a design density: of two keys equally far from the query, the longer one gets exactly $e^{(\lVert k_j \rVert^2 - \lVert k_i \rVert^2)/2\tau}$ times the weight. On the sphere the kernel is the von Mises–Fisher one that formalRAG develops (Hypersphere and von Mises–Fisher Geometry, its Definition 1).

A kernel smoother is also a conditional mean. Put a Gaussian bump on every key and a point mass on every value, and attention computes the posterior.

The attention weights are the responsibilities of an isotropic Gaussian mixture whose means are the keys: one soft assignment of the query to a key, then the mean of the assigned values. Gabbur, Bilkhu, and Movellan (2021) give a probabilistic reading of dot-product attention along these lines. The joint $\hat p$ has atoms in $v$, so it is not a density. Its marginal in $q$ is a weighted kernel density estimate, a valid density by the argument formalStatistics gives for kernel density estimates (Kernel Density Estimation, its Theorem 1). The notebook checks the conditional mean the way it is defined: on $10^5$ draws from the mixture, the residual $V - o(Q)$ is uncorrelated with $1$, $Q_1$, and $Q_2$, while the smoother that drops the design weights fails the same test.

A smoother averages over a set, and a set has no order. The next proposition is why the positional-encodings topic exists.

At the last word, one causal layer without positions reads the dog bit the man today exactly as it reads the man bit the dog today. The last word has to match, because that is the $x_t$ of (ii). The notebook reverses a corpus sentence’s prefix, good morning and welcome to the brightwater analytics, keeping its last word, and one random causal layer’s output at the last position is unchanged to rounding. Two layers differ by more than $10^{-3}$, asserted. The causal mask itself leaks order, which is why causal language models trained without positional encodings still learn some (Haviv et al. 2022).

---

Temperature is bandwidth

Take one query and a fixed set of scores, and turn the temperature. At $\tau \to 0$ the weight piles onto the best-scoring key; at $\tau \to \infty$ it spreads evenly. Both limits come with rates.

The first limit is retrieval: when the best key is separated from the rest, attention returns its value with an error exponentially small in the gap. That is the one-step retrieval of Ramsauer and colleagues’ (2021) modern Hopfield networks, and its zero-temperature end is the MaxSim lookup of late-interaction retrieval in formalRAG. The second limit is a bag-of-values average. Between them lies the bandwidth.

How should the bandwidth be set? formalML answers for the Nadaraya–Watson estimator asymptotically: bias of order $h^2$, variance of order $1/(nh)$, and an optimal bandwidth of order $n^{-1/5}$ in one dimension. For attention there is a setting where the answer is exact at every $n$. Place $n$ keys at equally spaced angles around a circle, which is what a sinusoidal positional encoding of a single frequency does.

With one frequency of position, attention is a low-pass filter. The Fourier mode $k$ keeps the fraction $\rho_k$ of its amplitude, and once $n$ is large enough that the aliased terms vanish, $\rho_k = I_k(c)/I_0(c)$; at $n = 64$ and $c = 7.3$ the first six gains are $0.929$, $0.746$, $0.520$, $0.318$, $0.172$, and $0.083$. The first gain $I_1(c)/I_0(c)$ is formalRAG’s mean resultant length of the von Mises–Fisher distribution on the circle (its Proposition 2), and the Bessel normalizer is its Theorem 2. Raising $c$, which lowers the temperature, opens the filter: less bias, more noise. This is the trade-off formalML derives asymptotically for kernel regression, here as a closed form at every $n$, and it rests on the orthogonality and Parseval’s identity of formalCalculus (Fourier Series & Orthogonal Expansions). The notebook checks (i) against numpy’s FFT on a grid of $n$ and $c$, to $10^{-12}$, and (ii) against simulated noise.

The asymptote is approached quickly. Two spectra are used below: a smooth one, with random Fourier coefficients of variance $1$ at frequency $1$ and $\tfrac14$ at frequency $2$, and a rough one, with variance $\tfrac14$ at each of the frequencies $1$ through $6$, both with noise $\sigma = \tfrac12$. Computed from Theorem 1’s exact formula, the ratio $c^*(n)/(Cn^{2/5})$, where $C = (4B/\sigma^2\sqrt\pi)^{2/5}$, is:

spectrum	$C$	$n = 32$	$64$	$128$	$1{,}024$	$10^6$	$10^8$
smooth	$2.636$	$0.949$	$0.962$	$0.972$	$0.988$	$0.9993$	$0.9999$
rough	$17.513$	$0.816$	$0.906$	$0.929$	$0.970$	$0.9981$	$0.9997$

Between $n = 10^6$ and $10^8$ the log–log slope of $c^*$ is $0.4001$ for the smooth spectrum and $0.4003$ for the rough one.

Does a trained attention layer find this optimum? Collins and colleagues (2024) prove that a softmax attention unit trained on regression tasks learns a window that narrows as the tasks’ functions grow less smooth and widens with label noise. Here the optimum is known exactly. One attention layer reads the tokens $(\cos\theta_j, \sin\theta_j, y_j)$; its queries and keys see the position only, so its scores are $u(\theta_i) M u(\theta_j)^\top$ with $M = W_Q W_K^\top/\sqrt2$, and its values are the $y_j$. Adam trains it on fresh random functions from one spectrum, at random rotations $\theta_0$, to minimize the squared error against the clean $f$. The task distribution is invariant under rotations and reflections, which points to a population optimum of $M = c^* I$, with $c^*$ from Theorem 1. Invariance alone doesn’t prove that, since a symmetric objective can have asymmetric minimizers. So the harness searches: it minimizes the exact population risk over all $2 \times 2$ matrices $M$, from one isotropic and five random starts, for each row below, and every search ends within $0.1\%$ of $c^* I$, at Theorem 1’s risk to $10^{-6}$. That is numerical evidence, not a proof that the minimum is global.

spectrum, $n$	learned $c$	Theorem 1’s $c^*$	$\lVert M - c^* I \rVert / \lVert c^* I \rVert$	$R(c^*)$
smooth, $32$	$10.006$	$10.011$	$0.12\%$	$0.0546$
smooth, $64$	$13.320$	$13.391$	$0.53\%$	$0.0316$
rough, $32$	$57.201$	$57.130$	$0.13\%$	$0.1405$

Here $M$ is the average over the second half of training, and the learned $c$ is half its trace. The trained layer lands within $0.53\%$ of the theorem’s optimum at worst, from a start near uniform attention. That worst case is nearly all the miss in $c$ itself, $0.525\%$ of $c^*$. Anisotropy is a separate and smaller effect: $\lVert M - cI \rVert / \lVert M \rVert$ is at most $0.11\%$. The layer learns a kernel more than five times sharper for the rough functions than for the smooth ones, and a sharper one for the longer context.

---

Why $1/\sqrt d$

The temperature of Definition 2 is not a tuned constant but $\sqrt{d_k}$. Vaswani and colleagues gave the reason in a footnote: dot products of random vectors grow with the dimension, and large scores push the softmax into a region with tiny gradients. Both halves can be made exact.

Part (iii) stops at convergence in probability for a reason. The keys change with $d$, and the theorem states no coupling across dimensions. With $n = 2$, $1 - \alpha_{\max} = 1/(1 + e^{\lvert S_d \rvert})$ with $S_d = \langle q, k_1 - k_2 \rangle$. If the draws are nested, one coordinate added at each $d$, then $S_d$ is a recurrent random walk. If they are independent across $d$, then $P(\lvert S_d \rvert \lt 1)$ is of order $1/\sqrt d$, whose sum diverges, so the Borel–Cantelli lemma applies. Either way $\lvert S_d \rvert \lt 1$ infinitely often, almost surely, and then $1 - \alpha_{\max} \gt 1/(1 + e)$: almost sure convergence fails.

Without the scale, the kernel’s bandwidth shrinks like $d^{-1/2}$ in units of the scores, attention collapses into a lookup at initialization, and by (ii) the gradient through the scores vanishes with it. The neural-language-models topic’s covariance identity, there a fact about the output layer, is here a cost. With $16$ keys, mean $1 - \alpha_{\max}$ and mean $\lVert J \rVert_2$ over $600$ random draws at each dimension are:

$d$	$4$	$16$	$64$	$256$	$1{,}024$
$1 - \alpha_{\max}$, unscaled	$0.571$	$0.309$	$0.139$	$0.073$	$0.041$
$1 - \alpha_{\max}$, scaled	$0.766$	$0.755$	$0.748$	$0.751$	$0.759$
$\lVert J \rVert_2$, unscaled	$0.257$	$0.250$	$0.157$	$0.093$	$0.056$
$\lVert J \rVert_2$, scaled	$0.192$	$0.203$	$0.208$	$0.207$	$0.203$

---

A lookup instead of a state

The neural-language-models topic’s Theorem 5 bounds the gradient that reaches state $k$ from the loss at step $t$ by a product of $t - k$ step Jacobians. Attention has no such product.

The lookup’s path from $t$ back to $k$ has length one. Its only loss is the factor $\alpha_{tk}$, a dilution by the positions competing for weight, and the scores can undo it by concentrating the weight. That is what “replaces the compression with a weighted lookup” means as a derivative. The long-lag task shows it.

The bit symbols differ from the noise symbols, so the layer can find the bit by content alone, with no notion of position; the honest caveats return to this. The census below repeats the neural-language-models topic’s protocol: the norm of the gradient of the batch-mean loss reaching the bit’s own representation, at initialization, the median over $12$ initializations on batches of $256$. The recurrent rows at lag $60$ are that topic’s own census, re-run.

lag $L$	$10$	$30$	$60$	$120$	$240$	$480$
attention	$2.22 \times 10^{-4}$	$5.71 \times 10^{-5}$	$3.76 \times 10^{-5}$	$2.77 \times 10^{-5}$	$9.81 \times 10^{-6}$	$4.88 \times 10^{-6}$
Elman	$1.88 \times 10^{-4}$	$2.78 \times 10^{-7}$	$2.45 \times 10^{-10}$	$1.37 \times 10^{-15}$	$1.87 \times 10^{-33}$	$9.41 \times 10^{-46}$
LSTM, forget bias $1$	$1.98 \times 10^{-4}$	$3.28 \times 10^{-5}$	$2.42 \times 10^{-7}$	$3.35 \times 10^{-11}$	$1.73 \times 10^{-16}$	$1.83 \times 10^{-31}$
LSTM, forget bias $3$	$1.12 \times 10^{-3}$	$8.73 \times 10^{-4}$	$1.42 \times 10^{-3}$	$2.36 \times 10^{-4}$	$1.51 \times 10^{-4}$	$1.29 \times 10^{-7}$

Attention’s gradient times $L + 1$ stays between $1.77 \times 10^{-3}$ and $3.35 \times 10^{-3}$ across the whole range: dilution, roughly $1/(L + 1)$, the equal-score case of Proposition 5. The scores at initialization are not exactly equal, so the match is only rough. The Elman network’s falls by $41$ orders of magnitude. The LSTM with forget bias $3$ is ahead of attention at lag $60$, because its cell path damps the bit far less, keeping $\sigma(3)^{60} = 0.054$ of it over that lag (that topic’s Proposition 4). But its gradient still falls, unevenly from lag to lag, and by lag $480$ attention has overtaken it. A forget gate postpones the decay; a lookup has none.

Trained by the same protocol, $600$ Adam steps at lag $60$, the attention layer answers all $2{,}000$ fresh test sequences correctly, at a test loss of $3.0 \times 10^{-6}$. That topic reported $0.486$ for the Elman network, $0.531$ for the LSTM with forget bias $1$, and $1.000$ with forget bias $3$. The attention layer also answers every sequence correctly at lag $600$, ten times the lag it was trained on: a lookup never counted the steps.

---

Attention over a document

The recurrent network of the neural-language-models topic reset its state at every sentence, and Kneser–Ney never looks past two words. Neither can use the fact that a company named in one sentence is likely to be named again three sentences later. A lookup over the whole document can. To see what such a lookup buys on text, before any training, take its keys from fixed word vectors.

This is attention with fixed queries and keys, a value per position, and Nadaraya–Watson’s estimator applied to classification: the cache predicts a word in proportion to the kernel weight of the earlier positions where it occurred. Grave, Joulin, and Usunier (2017) built such a cache over a recurrent network’s hidden states; here the keys are word vectors, so there is nothing to train. The two ends of the bandwidth are older models.

The unigram cache is Kuhn and De Mori’s (1990). The exact-context limit is a bigram cache, with one caveat that the corpus forces: $1{,}225$ pairs of training words, among $358$ of the $3{,}425$, have identical vectors, because they were seen in identical contexts, so the limit pools them. Of the $358$, $348$ were seen only once, many of them pseudo-words such as bondbook. The other ten are names such as aisha and rahman and ordinary words such as field and offshore. Between the limits, a context never seen before in the document borrows from the earlier contexts nearest to it, the neural-language-models topic’s smoothing by geometry, now over a document.

Inverse bandwidth $\beta$ and mixture weight $\lambda$ are chosen on the development documents, never on held-out text. For each $\beta$, the table gives the dev-chosen $\lambda$, the development loss, and the held-out loss, in nats per token over the closed vocabulary.

$\beta$	$0$	$\tfrac12$	$1$	$2$	$4$	$8$	$16$	$\infty$
dev $\lambda$	$0.99$	$0.99$	$0.98$	$0.98$	$0.98$	$0.99$	$0.99$	$0.99$
dev	$2.7898$	$2.7894$	$2.7887$	$2.7870$	$2.7871$	$2.7918$	$2.7925$	$2.7937$
held out	$2.8361$	$2.8353$	$2.8306$	$2.8282$	$2.8284$	$2.8386$	$2.8408$	$2.8424$

The bandwidth trade-off appears on text. Development data chooses $\beta = 2$, between the unigram cache and the exact-context cache, and on held-out text that choice beats both. With standard errors clustered by document, because a cache makes a document’s sentences dependent, Kneser–Ney scores $2.855$ nats per token ($\pm 0.026$) and the mixture $2.828$ ($\pm 0.025$). The paired gain, $0.0269 \pm 0.0025$, is more than ten standard errors.

The gain is where a cache should help. On the $41.9\%$ of positions whose target already occurred in the document, the loss falls from $1.798$ to $1.706$ nats; on first mentions it rises from $3.617$ to $3.637$, the price of the weight the cache takes. By distance back to the previous mention:

previous mention	share of repeats	in an earlier sentence	Kneser–Ney	mixture
$1$–$12$ tokens back	$36.3\%$	$89.1\%$	$1.427$	$1.380$
$13$–$50$	$45.5\%$	$99.2\%$	$1.764$	$1.682$
more than $50$	$18.2\%$	$100\%$	$2.621$	$2.414$

The farther back the previous mention, the more the cache gains, and nearly every repeat’s previous mention is in an earlier sentence, where the per-sentence recurrent network had nothing. The gain on those repeats alone is $1.31$ times the net gain. The net gain is still small, $0.637$ to $0.632$ bits per character over the open vocabulary, because the synthetic generator draws its words from corpus-wide restaurants: a document repeats itself only a little more than the corpus does. Real text repeats itself more (Church and Gale 1995), and the finance case study below is where that shows.

The keys so far were fixed. Learning them is the step from a cache to attention.

The pointer loss asks the kernel to put its weight where the target is, as in Merity and colleagues’ (2017) pointer sentinel, and it needs no Kneser–Ney, whose probabilities on its own training text are overfit. Starting from Definition 3 at $\beta = 2$, twelve epochs of Adam lower the training pointer loss at every epoch, from $3.001$ to $2.073$, and development data chooses the eleventh epoch with $\lambda = 0.97$. On held-out text the learned kernel’s own pointer loss falls from $3.256$ to $2.383$, and the mixture improves to $2.821$ nats per token, a further $0.0071 \pm 0.0011$, more than six standard errors.

---

Bandwidth, reach, and memory

The laboratory puts the last three sections side by side. Panel A is the circle: drag the concentration $c$ and watch the smoothed curve, the Fourier gains, and the exact risk; $c^*$ and the trained layer’s $c$ are marked. Panel B is the lookup: one held-out document and the three filings of the case study below, token by token, with the bits each variant of the cache spends on every token and, for selected tokens, where the kernel looks; and the gradient census of the long-lag task.

smooth f, n = 32smooth f, n = 64rough f, n = 32

the smootherthe exact risk

concentration cc = 11.0

c = 0: the meangrid point nearest c*c = 400.0: nearly a lookup

true fattention output

Dots: the noisy values y. The output is Definition 2 computed from these values; the readouts below are the exact risk over all functions of this spectrum.

risk R(c)

0.0548

bias²

0.0097

variance

0.0451

gain ρ₁

0.953

c* (Theorem 1)

10.011

R(c*)

0.0546

trained layer's c

10.006

Raising c narrows the kernel: the gains rise toward 1, so less of f is smoothed away (bias falls) and more of the noise passes through (variance rises). They balance at c*, and the attention layer trained on random functions of this kind lands there. At c = 0 every key gets the same weight and the output is the mean.

---

a document, token by tokenthe gradient across lags

held-out filingAppleExxonMobilMicrosoft

Kneser–Ney aloneβ = 0: unigram cacheβ = 2: the dev choiceβ = ∞: exact contextlearned kernel

thankyouforjoiningthedunmoreaerospacequarterlyresultscall¶resultsreflectedmultibrokercrosspremiumshelfbaseandtranspricecoprices¶theplanreallocatesresourcesfrommidsuppliersandmultibroker^•partnershipstowardoverindexanddepositbaserebatebooks¶managementhighlightedchannellineinterfleetroutersandoutreceivableasareasoffocus¶dunmoreaerospacereportedrevenueofmillionforfiscal¶prioritiesfortheyearincludecouponlinesshelfagestocksideoverindexsanddepositingcomponents¶costofrevenuewasmillion¶dunmorehasitsrootsingermany¶activityintransindexpipingandfleetworkheadcounttrackedthebroaderseatworkscrosspremium^•cycle¶dunmoreaerospacecompeteswithmarrowfreightforoutrebatesbusiness¶dunmoreaerospaceguidedtopercentgrowthinthefirstquarter¶thomasaveryhasspokenpubliclyaboutdunmoreaerospace¶theplanreallocatesresourcesfromnetworkingsandfundingtowardbrokerbaseandoutreceivable^•¶operatingmarginheldsteadyacrossoutreceivableandpostmerchant¶operatingcostsinpartnershipsandmarketspartnershipstrailedexpectations¶liquidityfellpercentinordersandundercyclerouterswhichmanagementdescribedasweaker¶operatingmarginwasbroadlyunchangedincorebate¶costsinpricageoutperformedexpectations¶demandforthewrenactuatorshapedtheroutersandnonroyaltysoutlook¶

Shading: bits the chosen model saves on each token against Kneser–Ney (green) or spends (red). Italic: not in the training vocabulary. ¶: a sentence end. Click any token; for the tokens marked •, the eight earlier positions the β = 2 kernel weights most are underlined, darker for more weight, and the word’s own earlier occurrences, the positions it is copied from, are outlined with dashes.

document, Kneser–Ney (bits per character)

0.742

document, β = 2: the dev choice

0.713

token

multibroker

its bits, Kneser–Ney

15.40

its bits, β = 2: the dev choice

10.71

cache probability of this word, β = 2

0.029

earlier occurrences it is copied from

1

β = 2 kernel, heaviest earlier positions: transprice (0.219) · midsuppliers (0.036) · from (0.035) · results (0.032) · the (0.032) · dunmore (0.032) · plan (0.032) · reallocates (0.031)

held outdevelopment

held out at β = 2 (dev λ = 0.98)

2.8282

held out at β = 0

2.8361

held out at β = ∞

2.8424

---

Kernels that factor

A head’s scores $QK^\top/\tau$ form a matrix of rank at most $d_k$. That limits what one head can express, and a stronger factorization makes attention cheap.

This is the neural-language-models topic’s softmax bottleneck (its Theorem 6; Yang et al. 2018), with keys in the place of the vocabulary, and it is Bhojanapalli and colleagues’ (2020) case for wider heads. Hard patterns escape it in the zero-temperature limit of Proposition 4: with $d_k = 2$, keys on a circle and queries pointed at the right angles realize any permutation. Several heads give several kernels, which is where the transformer-block topic begins.

The softmax kernel can also be factored approximately, with features that turn $e^{\langle x, y \rangle}$ into an inner product.

These are Choromanski and colleagues’ (2021) features. Their variance is largest exactly where the kernel is largest: the sharp kernel that makes a lookup useful is the one that does not compress. The point of factoring is the next proposition.

Katharopoulos and colleagues (2020) named the result: transformers with factored kernels are recurrent networks. The lookup has been compressed back into a fixed-size state, the neural-language-models topic’s design, and Proposition 8 gives the price. formalRAG’s Retrieval versus Long Context counts what the unfactored lookup costs; here is what factoring it loses. Replace the kernel cache’s exponential by $m$ positive random features and measure, on the held-out documents, the mean total-variation distance between the approximate attention weights and the exact ones:

$\beta$	$m = 16$	$64$	$256$	$1{,}024$
$\tfrac12$	$0.082$	$0.047$	$0.023$	$0.013$
$1$	$0.169$	$0.096$	$0.049$	$0.029$
$2$ (dev)	$0.264$	$0.221$	$0.117$	$0.096$
$4$	$0.422$	$0.394$	$0.319$	$0.266$
$8$	$0.734$	$0.589$	$0.495$	$0.434$

The error falls with the number of features and grows with the kernel’s sharpness, at every entry. At the dev-chosen setting the mixture’s loss hardly registers it, because the cache carries only $2\%$ of the probability: $2.8327$, $2.8282$, $2.8297$, and $2.8304$ nats per token for $m = 16$, $64$, $256$, and $1{,}024$, against $2.8282$ for the exact cache.

---

Finance case study: the filing as memory

---

Where this goes next

• Positions. Proposition 3 says that at the last word, one layer of attention cannot tell the dog bit the man today from the man bit the dog today. The positional-encodings topic adds position, and Theorem 1’s features $(\cos\theta, \sin\theta)$ are already a sinusoidal encoding of one frequency: several frequencies, and rotations of the queries and keys, are its subject.
• The transformer block. A block runs several heads, each a smoother with its own kernel (Proposition 7), and adds their outputs to a residual stream; the transformer-block topic assembles it.
• Contextual embeddings. Stacking blocks gives each occurrence of a word its own vector, the subject of the contextual-embeddings topic.
• Aspect-based sentiment. Attention to the words that carry an aspect’s sentiment is a kernel lookup whose query is the aspect; the aspect-based sentiment topic asks how far its weights explain its predictions.
• Retrieval. formalRAG’s late interaction scores a document by each query token’s best match, the zero-temperature lookup of Proposition 4, and its long-context topic prices the full lookup.

---

Honest caveats

---

Implementation

The companion notebook (notebookPath) imports attention_as_kernel_smoothing.py, which owns every number on this page. It imports the neural-language-models topic’s module for its gradient checkers, its long-lag task and census, its Kneser–Ney scores, and its spelling cache; the distributional-semantics topic’s module for the word vectors; and the perplexity topic’s speller and ratio estimator. It trains the circle denoiser, the lag-task attention layer, and the learned kernel from scratch in NumPy, and runs a harness of $21$ assertion-based tests in under a minute on a laptop CPU. The tests cover:

• Propositions 1–3 on random instances and a corpus sentence’s prefix; Proposition 2’s conditional mean by Monte Carlo.
• Proposition 4’s bounds; Theorem 1 against numpy’s FFT and simulated noise; Corollary 1 on the exact formula to $n = 10^8$; the trained denoiser against Theorem 1’s optimum; a search over general $M$ on the exact population risk.
• Theorem 2 by Monte Carlo; Proposition 5 against complex-step derivatives; Proposition 6, the census, the lag sweep, and training.
• Definition 3’s limits, the development grid, and the held-out benchmark with document-clustered standard errors; Definition 4’s gradient and training.
• Propositions 7–9 on random instances; the random-feature cache; the finance case study.

Every hand-written gradient is checked against complex-step derivatives and central differences. The laboratory reads its data from a JSON file that one of those tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.