Fine-Tuning and Low-Rank Adaptation

Overview & Motivation

The aspect sentiment topic ended on a limit. One attention layer whose query is the aspect is a signed vote, and replacing one word by another moves the vote in a direction no context can reverse, so no such layer reads “sales were not weak this period”. It pointed to the standard escape: a pretrained encoder, fine-tuned, in which every word’s representation is already a function of its neighbors. This page does that. The encoder is the one the masked-LM topic trained from scratch on synthetic filings: an $879$-word vocabulary, $32$ positions, width $d = 64$, two pre-LN blocks, $158{,}447$ parameters. The task is Topic 19’s: read the polarity of a gold aspect, from the mean of the encoder’s final states over the aspect’s words.

There are three ways to adapt a pretrained network to a task:

• Read it. Train a classifier on its frozen features, the linear probe of formalML’s representation-learning.
• Rewrite it. Train every weight from the pretrained start with a new head (Devlin and colleagues 2019).
• Nudge it. Freeze every weight and train a low-rank correction to each matrix (Hu and colleagues 2022).

On $14{,}179$ test aspects, the probe reads $0.761$. The same encoder never pretrained, fine-tuned identically, reads $0.951$. The pretrained encoder, fine-tuned, reads $0.970$ to $0.989$ over three seeds. The gap between the last two is what pretraining buys, and its largest part, construction by construction, is negation: negated adjectives are read at $0.920$ to $0.991$ against $0.717$. Counted in rows, plain adjectives gain more, since there are four times as many of them.

The page proves what can be said exactly about the third way and measures the rest on this encoder, in order:

1. One weight, two factors. What the factorization gives and costs.
2. The first step. Why the scale must fall like $1/\sqrt r$.
3. What two factors conserve. A hyperbola under gradient flow.
4. Is the update low-rank? Full fine-tuning’s answer.
5. What fine-tuning leaves alone. The span lemma, probe-first fine-tuning, and forgetting.
6. Two lenses on why a small update suffices. Random subspaces and kernels.
7. Reading negation in filings. A shortcut, its repair, and document tone.

---

One weight, two factors

Start with one number. A weight $w_0$ is frozen, and the adaptation moves it by $s\,ab$, a product of two trained numbers $a$ and $b$ and a fixed scale $s$. Every update $\Delta w$ is reached along a whole curve of pairs, the hyperbola $ab = \Delta w/s$, so the parameters are not identified by the function they compute. If $b$ starts at $0$, the update starts at $0$. The matrix version is the same picture in higher dimension.

Hu and colleagues write the transpose, for column vectors, with their $B$ zero and their $A$ random. We keep the roles: $A$ is the random factor and $B$ the one that starts at zero. Their text draws $A$ from a Gaussian; their reference code draws it from a uniform distribution.

On this encoder we adapt the query and value maps of both blocks, as Hu and colleagues do in most of their experiments. Each is a $64 \times 64$ matrix once the four heads sit side by side, so rank $16$ trains $8{,}192$ numbers of the encoder’s $158{,}447$. The One weight, two factors tab draws the one-dimensional picture: the loss $\tfrac12(ab - 2)^2$ in the $(a, b)$ plane, and the paths of gradient descent and of Adam from $(a_0, 0)$.

One weight, two factorsRank and scalingWhat full fine-tuning changedLearning vs. forgettingReading negation

start a₀0.250.511.5

step size η0.020.10.3

minimizers ab = 2a² − b² = a₀² (conserved by gradient flow)gradient descentAdam

GD ends at (a, b)

(1.457, 1.373)

Adam ends at (a, b)

(1.737, 1.151)

GD: max |a² − b² − a₀²|

0.0126

Adam: max |a² − b² − a₀²|

1.5642

Theorem 2: gradient flow keeps a² − b² fixed, so the path rides its hyperbola to the one minimizer on it, and b can grow only as fast as a. GD's drift is O(η) over the run, so it grows with the step; Adam rescales each coordinate, and its paths drift much further. Both paths run 400 steps.

---

The first step of a factored update

Write $G = \partial L / \partial W$ for the gradient full fine-tuning would use at the current weights. The chain rule (formalCalculus jacobian, Theorem 2) gives the factors’ gradients in terms of $G$, and at the zero start one of them vanishes.

For a Gaussian $A_0$ with $r \le k$, the singular values of $A_0/\sigma$ lie within $\sqrt k \pm \sqrt r$ in expectation (Vershynin 2012, Theorem 5.32, with the tail bound of Corollary 5.35). So under Adam $\lVert \Delta W_1 \rVert_F$ is about $\eta\, s\sqrt{rm}$: it stays put as $r$ grows only under $s = \alpha/\sqrt r$, for every draw. Kalajdzievski’s Theorem 3.2 proves the general version for SGD, at every fixed step as $r \to \infty$; part (c) is its first step made exact, and part (d), for Adam, is ours.

On the encoder, from the probe’s head (at a zero head every encoder gradient is zero), we averaged $16$ draws of $A_0$ for each rank, with the same draws under both scales. Under Adam with $\alpha/\sqrt r$, the first step’s size moves from $1.285$ at $r = 1$ to $1.626$ at $r = 64$. With $\alpha/r$ it falls to $0.203$, exactly $1/\sqrt r$ times the other. Under SGD with $\alpha/\sqrt r$, the mean step is $0.0057$ at every rank, but a single draw’s size at $r = 1$, averaged over the $16$, is $0.0428$, and it falls toward the mean, reaching $0.0081$ at $r = k = 64$. The moments behind part (c) give the exact law: for a Gaussian column $\mathbb E[\lVert a\rVert^2 aa^\top] = (k+2)\sigma^4 I$, so $\mathbb E[(aa^\top - \sigma^2 I)^2] = (k+1)\sigma^4 I$ and, for every $G$, $\mathbb E\lVert A_0A_0^\top G\rVert_F^2 = \big(1 + \tfrac{k+1}{r}\big)\lVert r\sigma^2 G\rVert_F^2$. A draw’s root-mean-square size is then $\sqrt{66} \approx 8.1$ times the mean at $r = 1$ and $\sqrt{129/64} \approx 1.42$ times it at $r = 64$, so even at $r = k$ it stays about $\sqrt 2$ above the mean; over the $16$ draws the ratios are $7.7$ and $1.43$. The Rank and scaling tab plots these, and what training made of them.

Trained for three epochs at one rate, the scale decides the update’s size more than its accuracy. Under $\alpha/\sqrt r$, the merged update $\lVert s\,AB\rVert_F$ grows from $28.4$ at rank $1$ to $61.8$ at rank $64$; under $\alpha/r$ it shrinks to $21.4$. On this seed the accuracies cross ($0.954$ against $0.947$ at rank $4$, $0.933$ against $0.956$ at rank $16$), but both gaps sit inside the spread of the headline LoRA run over three seeds ($0.911$ to $0.956$), so neither scale wins here. Rank stability promises that the right learning rate does not move with $r$, not that one scale wins at a fixed rate.

---

What two factors conserve

Back in one dimension, gradient descent on $(a, b)$ for a loss of $w = s\,ab$ has $\dot a = -s\,b\,g$ and $\dot b = -s\,a\,g$ for the same scalar $g$, so $\tfrac{d}{dt}(a^2 - b^2) = 0$. The path never leaves the hyperbola it starts on, and from $(a_0, 0)$ that is $a^2 - b^2 = a_0^2$: the factor that started at zero can grow only as fast as the other.

This is Theorem 2.2 of Du, Hu, and Lee (2018), for consecutive linear layers, and part (b) is their Claim B.1; Arora, Cohen, and Hazan (2018) use the same invariant (their Equation 9). Part (b) makes the drift $O(\eta^2)$ per step, so over a fixed time horizon it is $O(\eta)$. On a small least-squares problem, halving $\eta$ halves the drift at a fixed horizon: the slope against $\eta$ on log axes is $1.004$.

Adam is not gradient descent. It divides every coordinate by its own running scale, and the invariant goes. Fine-tuning the encoder with Adam, the headline LoRA run ends with $A^\top A - BB^\top$ moved, in the worst of its four matrices, by $1.42$ times the size of $A_0^\top A_0$. In the kernel experiment below ($120$ rows, from the probe’s head, $300$ full-batch steps), gradient descent at its stable rate keeps it within $2.4 \times 10^{-6}$, while Adam from the same start moves it by $1.43$. In the lab, gradient descent drifts off only by the $\eta^2$ terms of part (b), and the Adam paths drift much further.

---

Is the update low-rank?

Hu and colleagues hypothesized that the update full fine-tuning makes has a low intrinsic rank, which is why a rank-$r$ correction can stand in for it. We can ask full fine-tuning directly: take its update $\Delta W$ to each of the twelve block matrices (query, key, value, output, and the two MLP maps, in both blocks) and look at its singular values.

This is the Eckart–Young–Mirsky theorem, proved in full as Theorem 2 of formalML’s svd. Two yardsticks say whether an update is low-rank. The first is a random matrix of the same size and the same Frobenius norm, whose largest singular value sits near $\tau(\sqrt k + \sqrt m)$ with $\tau = \lVert\Delta W\rVert_F / \sqrt{km}$, the edge of the Marchenko–Pastur law (formalML’s double-descent, Theorem 1). The second is accuracy: merge the truncated update back in and score the result.

For full fine-tuning’s first seed, every one of the twelve updates has its top singular value $1.76$ to $3.72$ times above that edge, and a stable rank $\lVert\Delta W\rVert_F^2/\sigma_1^2$ between $2.06$ and $5.55$, out of a possible $64$. The accuracy confirms it. With every block-matrix update removed and only the fine-tuned biases, embeddings, and head kept, the model reads $0.480$; with every update truncated to rank $1$, $0.808$; to rank $2$, $0.929$; to rank $4$, $0.977$; to rank $8$, $0.987$; and the full update reads $0.989$. On the other two seeds rank $4$ keeps $0.935$ of $0.970$ and $0.958$ of $0.988$, and on all three rank $8$ is within $0.01$ of the full update. The What full fine-tuning changed tab shows the spectra and slides the truncation.

So a rank-$4$ update to every block matrix exists that does nearly everything full fine-tuning does, and LoRA cannot express it: LoRA adapts only the query and value maps, and no embedding. Inside that class, full fine-tuning’s own update does little. Keep only its query and value updates, put every other weight, bias, and embedding back at its pretrained value, and refit the probe: it reads $0.805$ to $0.826$ at rank $4$ over three seeds, and at most $0.851$ at full rank. LoRA trained at rank $4$ in the same class reads $0.954$ under $\alpha/r$ and $0.947$ under $\alpha/\sqrt r$. The update that does the work is spread over all twelve matrices; inside its own class, LoRA finds a far better one than full fine-tuning did.

---

What fine-tuning leaves alone

Gradient descent on a linear layer has a simple structure: the gradient of $W$ is a sum of outer products of the layer’s inputs with the backpropagated signals, so every step lies in the span of the inputs the layer saw.

Part (c) is the easy half of Kumar and colleagues’ Proposition 3.7. The hard half, which we cite, is their Theorem 3.3: from a head that does not already fit, fine-tuning changes the features on $S$ while leaving them fixed off it, so the head it learns no longer matches the features of inputs off $S$, and the out-of-distribution error is bounded below. Their remedy is LP-FT: fit the probe first, then fine-tune everything from there. On a toy of their form, with $d = 10$, $k = 2$, and a training span of dimension $4$, the features never move off the span ($5 \times 10^{-15}$ at most), and the out-of-distribution loss stays at $1.6 \times 10^{-31}$ from the probe’s head but reaches $6.85$ from a random one.

On the encoder, part (b) is exact. The training rows leave $419$ of the $880$ token rows unseen; full fine-tuning moves exactly the other $461$ and none of the $419$, and LoRA moves no embedding at all. Part (a) is a gradient-descent fact. On a linear toy, SGD leaves the component of the update off the span at $2 \times 10^{-15}$, and Adam, with the same data, at $1.008$.

Forgetting is what fine-tuning does to the pretrained skill, and we measure it as the rise of held-out masked-LM loss on test filings, from $1.523$ for the pretrained encoder. Full fine-tuning ends at $2.553$ to $2.759$ over three seeds. LP-FT, from the probe’s head, ends at $1.889$ to $1.965$ while reading $0.975$ to $0.981$, within a point of full fine-tuning on average (means $0.977$ and $0.983$; full fine-tuning spreads over $0.970$ to $0.989$) and with less than half its forgetting. LoRA ends at $5.208$ to $5.705$ while reading only $0.911$ to $0.956$.

That ranking seems to contradict Biderman and colleagues (2024), who found on Llama-2-7B that LoRA learns less and forgets less. Here it learns less and forgets more, but LoRA runs at ten times full fine-tuning’s rate (Biderman and colleagues report the same gap in their best rates). Run the two at a shared rate and the ranking turns around. At full fine-tuning’s $10^{-3}$, LoRA reads $0.864$ to $0.880$ and ends at $2.033$ to $2.338$, under full fine-tuning’s $2.553$ to $2.759$. At $3 \times 10^{-3}$, full fine-tuning reads $0.984$ to $0.997$ and ends at $4.011$ to $4.431$; LoRA reads $0.916$ to $0.948$ and ends at $2.729$ to $3.303$. At LoRA’s $10^{-2}$, full fine-tuning reads $0.968$ to $0.989$ and ends at $5.661$ to $6.373$, over LoRA’s $5.208$ to $5.705$. In all nine seed-matched pairs, LoRA learns less and forgets less, which is the direction Biderman and colleagues found. Update size does not explain it: confined to the query and value maps, LoRA moves them by $\lVert s\,AB \rVert_F = 45.1$ at rank $16$, where full fine-tuning at $10^{-3}$ moves the same four matrices by $2.69$ and at $10^{-2}$ by about $15$, while full fine-tuning, which forgets more at the shared rate, moves every other matrix too. LP-FT, at its own rate, forgets less than every LoRA run at every rate, and every LoRA run in the main grid is beaten by every LP-FT run on both counts.

A third way to keep the pretrained skill is to penalize distance from it.

The first statement is Theorem 5 of formalStatistics’ regularization-and-penalized-estimation, and the shrinkage is its Remark 4 on ridge seen through the SVD, here centered at $\theta_0$ instead of $0$. Li, Grandvalet, and Davoine (2018) introduced the penalty as L2-SP; they also penalize the new head toward $0$, which we do not. Elastic weight consolidation (Kirkpatrick and colleagues 2017) replaces $\lambda I$ by the Fisher information, the metric of formalML’s information-geometry. L2-SP trades accuracy for retention: $\lambda = 0.1$ reads $0.905$ with loss $1.712$, and $\lambda = 1$ reads $0.757$ with loss $1.543$. Full fine-tuning at three-tenths of its rate reads $0.920$ with loss $1.815$. LP-FT gets the retention without that price. The Learning vs. forgetting tab plots every run on these two axes.

---

Two lenses on why a small update suffices

LoRA restricts the update to a small set. Two older ideas explain why a small set can be enough, and both can be checked on this encoder.

A random subspace. Li and colleagues (2018) trained networks inside a random subspace, $\theta = \theta_0 + Sz$ for a fixed $P \times D$ matrix $S$ and trained $z \in \mathbb R^D$, and called the least $D$ that reaches $90$ percent of ordinary training the intrinsic dimension. Aghajanyan, Gupta, and Zettlemoyer (2021) measured it for fine-tuning and found that pretraining lowers it. We use the cheapest such $S$.

This is the structure of Theorem 1(c) again, a random map whose mean is the identity. Li and colleagues normalize the columns of $S$ instead, and Aghajanyan and colleagues use the Fastfood transform with learned per-layer scales, and under Adam a column’s scale acts as its learning rate: scaling a column by $c$ scales its gradient in $z$ by $c$, which Adam’s step in $z$ removes, so $\theta$ moves $c$ times as far. The $\pm 1$ entries here move every parameter at one rate. Inside subspaces of dimension $64$, $512$, and $4{,}096$, out of $157{,}568$ encoder parameters, the pretrained encoder reads $0.754$, $0.820$, and $0.870$, and the never-pretrained one $0.722$, $0.786$, and $0.823$. Neither reaches $90$ percent of its own full fine-tuning ($0.885$ and $0.856$) on this grid, so both intrinsic dimensions exceed $4{,}096$. The pretrained encoder leads at every dimension tried, which is Aghajanyan and colleagues’ direction, measured.

A kernel. Near its starting point a network is close to its linearization, $f_{\text{lin}}(x; \theta) = f(x; \theta_0) + J(x)(\theta - \theta_0)$, where $J(x)$ is the Jacobian of the outputs at $\theta_0$.

The kernel is the neural tangent kernel of Jacot, Gabriel, and Hongler (2018). Malladi and colleagues (2023) asked when fine-tuning a language model behaves like gradient descent with it, found that prompt-based fine-tuning often does, and proved that LoRA’s kernel is close to the full one with high probability (their Theorem 7.2), the concentrated form of Corollary 1. The lazy-training section of formalML’s double-descent is the regime in general.

A zero head has a zero Jacobian, so we linearize LoRA (rank $16$, query and value maps) around LP-FT’s start, the probe’s head held fixed, on $120$ training rows. Kernel gradient descent and the real LoRA model trained by full-batch gradient descent for $300$ steps agree on $0.992$ of $240$ test rows, end at training losses $0.5990$ and $0.5989$, and the kernel moves by $6.5$ percent. But at a stable rate, gradient descent barely trains: the loss falls only from $0.620$. Adam, from the same start, drives the training loss to $5.2 \times 10^{-5}$, its kernel moves by $169$ percent, it agrees with the kernel predictor on $0.658$ of test rows, and its test accuracy falls from $0.767$ to $0.621$. Malladi and colleagues describe Adam’s early steps with a kernel built from signs; on $120$ rows, Adam does not stay in any kernel regime for long.

---

Reading negation in filings

---

Where this sits

LoRA is one of a family of parameter-efficient methods. Adapters (Houlsby and colleagues 2019) insert small trained modules instead of correcting the weights, and the methods differ in where the small set lives, not in the questions this page asked of it. The page’s answers on one toy encoder:

• the factored step is a random sketch of the full one;
• the factors are balanced only under gradient descent;
• the full update is nearly low-rank but spread over every matrix, where LoRA’s class cannot hold it;
• probe-first fine-tuning keeps what pretraining learned better than LoRA does, and at a shared rate LoRA forgets less than full fine-tuning.

Sun, Huang, and Qiu (2019) gave the encoder the aspect as a second sentence. Our reader takes the aspect’s own contextual states instead, which fits Topic 19’s definition and read better than a pair input averaged over every position when we prototyped both. The extraction topics fine-tune the same kind of encoder for span labeling, and formalRAG’s dense-retrieval-dual-encoders bounds the rank of a dual encoder’s scores, where this page bounded the rank of an update.

---

Honest caveats

---

Implementation

The companion notebook (notebookPath) imports fine_tuning_low_rank_adaptation.py, which owns every number on this page. That module imports the masked-LM topic’s module for the encoder, its training run, its corruption channel, and its vocabulary. It imports the aspect sentiment topic’s module for the aspect rows, the constructions, the contrast sets, the tone-bearing sentences, and, through it, the linear classifiers topic’s Newton solver and the document baselines. The encoder’s backward pass is lifted from the masked-LM training loop and pinned equal to it, and its attention is rewritten as flattened matrix products, pinned equal to the transformer topic’s. The harness checks:

• every gradient against central differences;
• Theorem 1’s identities exactly, its means and the fluctuation law by Monte Carlo, and Adam’s bounds on random instances;
• Theorem 2’s one-step identity and its drift’s order;
• Proposition 3’s shrinkage, Proposition 4’s moments, and Proposition 5 and Corollary 1 on random linear models;
• Theorem 3 on the linear toys and, for the embeddings, on every fine-tuning run;
• every comparison this page states, across the $42$ fine-tuning runs, including both methods at three shared rates, the truncations at every seed and inside LoRA’s class, and what the encoder calls the deleted-”not” copies.

The laboratory reads its data from a JSON file that one of the harness’s tests requires to equal the module’s viz_payload() exactly, so the figure cannot drift from the code.