PEFT: Fine-Tuning in a Small Subspace

The idea in one sentence

Fine-tuning asks a trained model to change, and the discovery behind every method in this chapter is that the change fits in a tiny structured container: name the shape of that container and you have named the method.

The cost of changing your mind

Part VII measured what it costs to run a model. This part opens with a different bill: what it costs to change one.

Full fine-tuning means every weight moves, and a weight that moves needs company. The optimizer that trains modern networks, AdamW, keeps two running averages per parameter, and the backward pass needs a gradient for each one, so a model in training carries several full copies of itself. The PEFT library’s README measures this on an 80 GB A100. Fine-tuning the 3-billion-parameter T0 takes 47.14 GB of GPU memory. The same model with a LoRA adapter takes 14.4 GB, and 9.8 GB if the optimizer state parks in CPU RAM. The 12-billion-parameter mt0 does not fit at all: full fine-tuning runs out of memory on a card with eighty billion bytes, while its LoRA run takes 56 GB. All four numbers in the figure are theirs; everything measured later in this chapter was measured here.

The bill keeps arriving after training. A full fine-tune of T0 produces an 11 GB checkpoint, one per task. The LoRA checkpoint for the same job is 19 MB, which is why the inset on the right of Figure 1 looks like a misprint and is not. Ten tasks, ten full fine-tunes: 110 GB. Ten adapters: about the size of one phone photo album.

There is a third bill, and you can compute it on one line. For an $n \times n$ weight matrix, the forward pass costs about $n^2$ multiply-adds. The backward pass costs two more $n^2$ blocks: one to push the gradient through to the input, one to build the gradient for the weight itself. So training a matrix costs three times what running it costs. Freeze the matrix and that third block vanishes; a rank-$r$ adapter adds back only $6nr$, which is loose change when $r$ is small. Training with LoRA costs $2n^2 + 6nr$ per matrix against full fine-tuning’s $3n^2$: slightly more than two thirds, before any memory is counted. The counting is from Schulman and colleagues’ study LoRA Without Regret at Thinking Machines, which this chapter will lean on twice more, because it measured things nobody else had bothered to measure.

Freeze the map, learn the change

Here is the whole trick in one line. The trained model applies its weight matrix $W$. Instead of editing $W$, keep it exactly as it is, and learn a separate update $\Delta W$ beside it:

The two paths add at the output, so the base model never changes. Every gradient, every optimizer average, every byte you save at the end belongs to $\Delta W$ alone. If $\Delta W$ were an arbitrary dense matrix this would save nothing, since it would be as big as $W$. So the entire field lives inside one question.

What shape should $\Delta W$ take, so that it is small enough to be cheap and rich enough to carry the change? Each answer to that question is a named method, and there are dozens of names. The good news is that the answers sort into a handful of families, and every family runs on mathematics this course has already built.

The waist returns

Lecture 6 drew this chapter’s first answer before the question existed. Figure 3 there put a frozen $W$ next to a thin $C$ times a wide $R$ and marked their shared dimension $r$ as the waist. That is LoRA, low-rank adaptation, from the 2021 paper by Hu and colleagues at Microsoft (arXiv:2106.09685): set

with $C$ tall and $r$ columns wide, $R$ flat and $r$ rows tall. The update is a rank-$r$ matrix, and Lecture 8 is why that is a real constraint: a rank-16 matrix, however large its footprint, carries 16 independent directions and no more. The bet is that adapting a task needs few directions, even when the model needs millions of coordinates.

The library makes the bet concrete in four lines. Everything below runs on the machine this course is written on, an Apple M3 Max, in float32, on the half-billion-parameter Qwen2.5-0.5B:

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, TaskType, get_peft_model

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B",
                                             dtype=torch.float32)
config = LoraConfig(r=16, lora_alpha=32, task_type=TaskType.CAUSAL_LM)
model = get_peft_model(model, config)

model.print_trainable_parameters()
# trainable params: 1,081,344 || all params: 495,114,112 || trainable%: 0.2184

for name, module in model.named_modules():
    if "layers.0." in name and name.endswith(("lora_A.default", "lora_B.default")):
        print(name, tuple(next(iter(module.parameters())).shape))
# ...layers.0.self_attn.q_proj.lora_A.default (16, 896)
# ...layers.0.self_attn.q_proj.lora_B.default (896, 16)
# ...layers.0.self_attn.v_proj.lora_A.default (16, 896)
# ...layers.0.self_attn.v_proj.lora_B.default (128, 16)

C = model.state_dict()[
    "base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight"]
print("largest entry of the thin factor on day one:", float(C.abs().max()))
# largest entry of the thin factor on day one: 0.0

One naming wrinkle before the numbers. The code calls the wide factor lora_A and the thin one lora_B, because the paper writes the update as $BA$ and applies $A$ first. This course keeps its own letters from Lecture 6: the wide first factor is $R$, the thin second one is $C$. Same matrices, same waist.

The printed shapes say exactly where the budget went. By default the library adapts two matrices per transformer block, the queries and the values. Qwen’s width is 896, so each $R$ is $16 \times 896$. The query update’s $C$ is $896 \times 16$, and the value update’s $C$ is $128 \times 16$, because this model shares key-value heads and its value output is only 128 tall. The adapter never gets a say in any of this; it inherits whatever geometry the frozen model already has.

Three details in Figure 4 do a lot of quiet work. First, $C$ starts as all zeros, and the printed check above confirms it: the largest entry is exactly 0.0. Since $CR = 0$, on day one the wrapped model is the base model, bit for bit, and training starts from what the model already knows instead of from noise. Second, the update enters scaled by $\alpha/r$, here $32/16 = 2$, so if you change the rank later the update’s size does not silently change with it. Third, when training ends you can merge: replace the frozen matrix with $W + CR$ once, and the adapter machinery vanishes. The merged model has the same shape, the same speed, and the same memory as the original. We will watch the merge happen, and measure how little it changes, at the end of the chapter.

The $\alpha/r$ detail hides the chapter’s most surprising piece of algebra, and it takes four sentences. Write the update as a sum of rank-one pieces, $CR = c_1 r_1 + \cdots + c_r r_r$, one column of $C$ times one row of $R$ at a time, the rank-one slabs of Lecture 8. At initialization each piece is statistically identical to the others, so each one’s expected first movement under training is the same, and dividing the sum by $r$ turns the update into an average of $r$ same-shaped random pieces. An average of five such pieces and an average of five hundred start moving the same way. So the first steps of training are the same at every rank: Schulman’s team watched learning curves at rank 4 and rank 512 sit so exactly on top of each other that they went hunting for a bug before believing it. Rank decides where training can eventually go. It has no vote on how training starts.

The $C = 0$ start also does something nobody designed. While $C$ is still small, moving $R$ barely changes the product $CR$, so early training crawls no matter what learning rate you set; as $C$ grows, the same nominal step moves the update more and more. The initialization is a warmup schedule that nobody wrote down. It is also the best explanation on offer for a rule of thumb the same study measured across models and tasks: LoRA’s best learning rate sits at about ten times full fine-tuning’s, and for very short runs higher still, because a LoRA spends its opening steps partly asleep and has to make up the distance.

First, the receipt. The printed count is not something to take on faith; it is something to check by hand, the way this course checks everything:

That is the shape of the era we train in: a half-billion numbers stand still while a million learn.

One practical correction to the defaults before moving on. The library, following the original paper, attaches adapters to the attention matrices, and that is what the receipt above counted. The measured advice is now different: attach LoRA to every weight matrix, and above all to the MLP layers, because that is where most of the parameters live. Schulman’s team found that attention-only LoRA learns slower even when its rank is raised until the parameter counts match, so the gap is about which matrices are covered rather than how many numbers train. Cheap rank everywhere beats generous rank somewhere.

Why a tiny rank is enough

The honest answer used to be that nobody fully knew. It has become an accounting argument, and the accounting is worth learning, because it predicts things the folklore gets wrong.

Start with the oldest evidence, which predates LoRA. In 2020, Aghajanyan, Zettlemoyer and Gupta measured the intrinsic dimension of fine-tuning (arXiv:2012.13255): pick a random low-dimensional subspace of the full weight space, allow updates only inside it, and see how few dimensions still solve the task. For RoBERTa on a standard sentence-pair benchmark, 200 dimensions reached ninety percent of full fine-tuning. Not 200 million: 200. Pretraining does the hard, high-dimensional work once, and a task update mostly steers between abilities the model already has.

Now count in bits instead of dimensions, because bits let you compare things that look incomparable. Allen-Zhu and Li measured how much knowledge trained weights can hold (arXiv:2404.05405): about 2 bits per parameter, in the long-training limit. And a dataset’s information content can be measured too, with a tool this course already owns: sum the log loss over the first epoch, and you have the number of bits needed to write the dataset down given the model. For language data that runs near one bit per token. So both sides of fine-tuning have a size in the same currency. A million-parameter adapter is a two-megabit container. An instruction-tuning set of a few hundred thousand tokens is a few-hundred-kilobit lesson. The container is bigger than the lesson, and that single comparison is the whole story: Schulman’s study names it the low-regret regime, the region where the adapter’s capacity exceeds the dataset’s bits, and measures that inside it LoRA matches full fine-tuning’s learning curve step for step. Push past capacity and there is no cliff; the loss curve peels away from full fine-tuning’s and learning just gets less efficient, which is the graceful version of running out of room.

The same accounting explains the result in this chapter that sounds most like a misprint: for reinforcement learning, rank one is enough. A supervised pass hands the model a graded answer at every token, roughly one bit each. A policy-gradient episode hands the model thousands of tokens of its own output and then one number about the whole thing: better than average, or worse. All the gradient’s contact with the reward flows through that single scalar, so an episode teaches at most a few bits, no matter how long it is.

Run the numbers on a real case, as the study does. A math RL run: about 10,000 problems, 32 attempts each, one verdict per attempt. That is roughly 320,000 bits for the entire training run. A rank-one adapter on an 8-billion-parameter model already holds about 3,000,000 numbers. The container is ten times the size of everything the run can possibly teach, at rank one, and the measured curves agree: LoRA at rank one matches full fine-tuning on RL. The most extreme parameter efficiency in this chapter is not an approximation that happens to work. It is a container correctly sized to an unusually small lesson.

LoRA turns all of this into one dial. Too small an $r$ and the update cannot span the directions the task needs; the loss stalls early, the way the low-rank curves peel off in Schulman’s sweeps. Large enough, and extra rows buy nothing, because the lesson ran out before the container did. The library defaults to $r$ between 8 and 64 because most post-training lessons fit there, and this chapter’s own run will show $r = 16$ overshooting a tiny task by a comfortable margin.

The map of the zoo

The PEFT library’s tuner folder holds about forty named methods, and the names churn every few months. The shapes do not. Sorted by the shape of $\Delta W$, the whole zoo collapses into six families. This section draws the map; Lecture 30 is the full field guide behind it, every method in the library with its own drawing.

Every card was checked against the library’s source before it was drawn, and every percentage below was printed by wrapping the same base model on this machine:

LoRA, LoHa, LoKr, HRA, OFT and FourierFT adapt the query and value matrices of all 24 layers, like the run above. IA3 uses its own defaults, keys and values plus each block’s down-projection. VeRA adapts the queries only, since it shares one frozen pair across layers and needs those layers to agree in shape. The families, one at a time:

Add a thin product. LoRA’s family. AdaLoRA starts wider and learns which directions matter, carrying an importance score per direction and pruning rank away from layers that waste it, so the budget migrates to the layers that need it. DoRA splits each output channel’s length from its direction and trains the lengths separately. HiRA multiplies instead of adding at the very last step: its update is $W \circ (CR)$, the frozen weight scaled entrywise by a low-rank pattern, which lets a rank-16 pair paint a high-rank change because $W$ itself has full rank to lend.

A structured product. LoKr builds $\Delta W$ as a Kronecker product. On this model’s $896 \times 896$ query matrix it stores a $28 \times 28$ and a $32 \times 32$ factor, 1,808 numbers standing in for 802,816, because each entry of the small matrix stamps a whole tile. LoHa multiplies two low-rank products entrywise, $(C_1 R_1) \circ (C_2 R_2)$, spending exactly twice LoRA’s budget; the table shows the factor of two to the digit, 2,162,688 against 1,081,344.

A few coefficients in a fixed basis. FourierFT stores nothing shaped like a matrix at all: 1,000 chosen positions in the frequency plane, one learned number each, and an inverse Fourier transform builds the dense $\Delta W$ from them on demand. The basis is fixed and free; only the coefficients train. WaveFT plays the same game in a wavelet basis. This is the same bet a JPEG makes about images: the interesting matrices are sparse somewhere, once you pick the right somewhere.

Rescale the channels. IA3 trains one number per channel and multiplies: 896 for the queries’ outputs, 128 for the values’, 4,864 across each block’s down-projection input. VeRA, from Kopiczko, Blankevoort and Asano (arXiv:2310.11454), is this family’s sharpest trick, and Figure 8 shows it.

Random projections mix directions well enough that a fixed random $C$ and $R$ can serve every layer at once, with each layer training only two small vectors that rescale the pair’s rows and columns, here 1,152 numbers a layer. The pair itself can be rebuilt from a stored seed instead of shipped, and the whole run trains 27,648 numbers, thirty-nine times fewer than LoRA. RandLoRA extends the idea with several frozen random bases combined by small trainable diagonals.

Two families are missing from this walk, because each earns its own section.

Rotate, do not add

Every additive method can change how strongly the model responds along some direction. A rotation cannot, and that is the point of the orthogonal family. Lecture 13 built matrices with orthonormal columns exactly because they leave every length alone, and this family turns that guarantee into an adapter: instead of $W + \Delta W$, it applies

with $R$ orthogonal. Then for any input, $\norm{RWx} = \norm{Wx}$: whatever the layer used to produce, the adapted layer produces something exactly as long, only turned. The update cannot blow activations up and cannot crush them to zero, no matter what training does to its parameters, because the constraint is built into the shape rather than begged for with a small learning rate.

OFT makes $R$ block-diagonal to keep it affordable. On this model it stores 14 blocks of size 64 per adapted matrix, and inside each block it trains only the strictly-upper skew entries, 2,016 of them, because a classical construction, the Cayley transform, turns any skew-symmetric matrix into an orthogonal one. Orthogonality is not a penalty the loss must maintain; it is an algebraic certainty. BOFT builds the same promise from butterfly factors, sparse rotations that compose into a dense one. HRA composes $r$ Householder reflections, the mirrors $I - 2uu^{\T}$, training only the $r$ mirror normals per matrix, 344,064 numbers in the run above.

Leave the map alone

The last family refuses the question. Lecture 1 turned words into vectors precisely so that the model could compute with them, and soft prompts adapt the model by editing those vectors instead of any weight. Prepend $k$ learned vectors to the input sequence, freeze everything else, and let attention do the rest:

The measured version: 16 virtual tokens on this model is $16 \times 896 = 14{,}336$ trainable numbers, the smallest row of the table by far. The learned vectors are not words and never need to be; they are free points in embedding space, chosen by gradient descent to push the frozen model’s attention where the task lives. Prefix tuning goes deeper: it learns key and value vectors inside every layer’s attention, where prompt tuning stops at the entrance. And p-tuning trains a small network that writes the prompt vectors during training and is thrown away after. Steering a frozen map by adding vectors to its input is the oldest move in this course; it is what every $x + \Delta x$ has done since Part I.

The four-bit base

One cost survives everything above: the frozen $W$ still has to sit in memory for the forward pass. QLoRA, from Dettmers, Pagnoni, Holtzman and Zettlemoyer (arXiv:2305.14314), attacks the last stronghold. Store the frozen base in four bits per weight, and train a full-precision LoRA on top of it.

Four bits allow sixteen values, so the question is where to put them. Trained weights arrive roughly normally distributed, and the paper’s NF4 data type places its sixteen levels to be, in the authors’ words, information-theoretically optimal for exactly that distribution: dense where weights crowd, sparse in the tails. Each multiply unpacks a block of $W$ back to 16 bits, uses it, and drops it, so the wide copy never exists all at once. A second trick the paper calls double quantization compresses the per-block scale factors themselves. The gradient story is what makes it sound: gradients flow through the dequantized weights as constants and land only in $C$ and $R$, which live in full precision, so the base’s rounding is a fixed lens, never an accumulating error. The arithmetic is the headline: a 7B model that needs 14 GB in bfloat16 stores in about 3.5 GB, and the authors fine-tuned a 65-billion-parameter model on a single 48 GB card. This chapter cannot run that here, and does not pretend to; every QLoRA number above is theirs.

Sixty steps on this laptop

Time to train one. Twelve sentences in this course’s own voice are the whole dataset, and the run fits in a coffee sip: sixty steps of AdamW on the adapter, everything else frozen.

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, TaskType, get_peft_model

torch.manual_seed(0)
device = "mps" if torch.backends.mps.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B",
                                             dtype=torch.float32).to(device)

corpus = [
    "The column space of A holds every vector that A can produce.",
    "The null space of A holds every vector that A sends to zero.",
    "The rank of a matrix is the number of independent columns it has.",
    "A basis is a set of independent vectors that spans the space.",
    "Elimination factors A into a lower triangle L times an upper triangle U.",
    "The projection of b onto a line keeps the part of b along that line.",
    "Orthogonal vectors have inner product zero and never lean on each other.",
    "An eigenvector keeps its direction when the matrix acts on it.",
    "The four subspaces come in two orthogonal pairs.",
    "Least squares picks the x that makes the error perpendicular to the column space.",
    "A rank one matrix is one column times one row.",
    "Every A equals C times R, a thin matrix times a wide one.",
]
batch = tok(corpus, return_tensors="pt", padding=True)
ids, attn = batch.input_ids.to(device), batch.attention_mask.to(device)
labels = ids.clone()
labels[attn == 0] = -100

prompt = tok("The null space of A holds", return_tensors="pt").input_ids.to(device)

def continue_text(m):
    out = m.generate(prompt, max_new_tokens=12, do_sample=False,
                     pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True)

print("base:", repr(continue_text(model)))
# base: ' true for all matrices A. What is the dimension of the'

peft_model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32,
                                              task_type=TaskType.CAUSAL_LM))
opt = torch.optim.AdamW(
    (p for p in peft_model.parameters() if p.requires_grad), lr=2e-4)

t0 = time.time()
for step in range(1, 61):
    loss = peft_model(input_ids=ids, attention_mask=attn, labels=labels).loss
    loss.backward()
    opt.step()
    opt.zero_grad()
    if step % 10 == 0:
        print(f"step {step:2d}   loss {loss.item():.3f}")
print(f"{time.time() - t0:.0f} seconds on {device}")
# step 10   loss 2.700
# step 20   loss 1.398
# step 30   loss 0.576
# step 40   loss 0.282
# step 50   loss 0.150
# step 60   loss 0.095
# 5 seconds on mps

peft_model.eval()
print("tuned:", repr(continue_text(peft_model)))
with peft_model.disable_adapter():
    print("off:  ", repr(continue_text(peft_model)))
# tuned: ' every vector that A sends to zero. It is a free'
# off:   ' true for all matrices A. What is the dimension of the'

peft_model.save_pretrained("qwen-notes")
import os
print(os.path.getsize("qwen-notes/adapter_model.safetensors"), "bytes on disk")
# 4338000 bytes on disk

Read the three continuations together. The base model, asked what the null space of $A$ holds, wanders off into a quiz question. After five seconds of training, the adapter answers in this course’s words. And inside disable_adapter(), the original wandering answer comes back verbatim, because the base weights were never touched; the old model is always intact, one context manager away. That is catastrophic forgetting handled by construction rather than by care: you cannot forget what you never overwrote.

The file on disk is 4,338,000 bytes: 1,081,344 float32 numbers at 4 bytes each, plus a small header. The receipt from Figure 5 again, now as a file size.

The saved adapter loads by name, and a model can hold several and switch:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

device = "mps" if torch.backends.mps.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B",
                                            dtype=torch.float32).to(device)

model = PeftModel.from_pretrained(base, "qwen-notes", adapter_name="notes")
prompt = tok("The null space of A holds", return_tensors="pt").input_ids.to(device)

def continue_text():
    out = model.generate(prompt, max_new_tokens=12, do_sample=False,
                         pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True)

model.set_adapter("notes")
print(model.active_adapter, "->", repr(continue_text()))
# notes -> ' every vector that A sends to zero. It is a free'

with torch.no_grad():
    before = model(prompt).logits
merged = model.merge_and_unload()
with torch.no_grad():
    after = merged(prompt).logits

print("largest logit change from merging:", float((before - after).abs().max()))
print("what is left:", type(merged).__name__)
# largest logit change from merging: 7.2479248046875e-05
# what is left: Qwen2ForCausalLM

The last two lines are Figure 4’s third act, measured. Merging computed $W + CR$ once per adapted matrix, and the largest change in any output logit was seven hundredths of a thousandth, float32 rounding and nothing else. What remains is a plain Qwen2ForCausalLM: no adapter classes, no extra multiply at inference, no trace of the surgery except the new behavior. In one earlier run here, a freshly initialized adapter was compared against the raw base the same way, and the largest logit difference was exactly zero, which is the $C = 0$ initialization keeping its promise to the last bit.

What this cannot do

Honesty about the limits, since the zoo’s marketing rarely supplies it. A rank-16 update per matrix is a bet that the change is small in a precise sense, and some changes are not. A new language, a new modality, deep new knowledge: those move the model far from where pretraining left it, and the intrinsic-dimension argument stops applying, because that argument was measured on tasks near the pretrained model’s abilities. When the destination is far, full fine-tuning, or bigger surgery still, remains the honest tool, and the PEFT README itself frames its methods as reaching performance comparable to full fine-tuning on downstream adaptation, which is the near-destination case.

The two-matrix shape itself has a cost that capacity arguments miss. Training the product $CR$ is a different optimization problem from training a single matrix, and Schulman’s study caught one place where the difference bites: at large batch sizes LoRA pays a loss penalty that full fine-tuning does not, and raising the rank does not buy it back. The penalty belongs to the parametrization, the price of the waist itself. At the batch sizes where both methods do their best work, small ones, the gap closes.

The sixty-step run above also shows the small print if you read it twice. The adapter learned this course’s sentences quickly because the base model already knew English, already knew mathematics, and only needed steering toward one voice. The million numbers did not add knowledge; they selected among abilities half a billion numbers already paid for. That is what fine-tuning in a small subspace means, and it is a strength exactly as long as you know that is what you bought.

Where this is going

Look back at the map once more. The waist came from Lecture 6, the meaning of rank from Lecture 8, the geometry of never-stretching updates from Lecture 13, the steering power of added vectors from Lectures 1 and 11, and the machine all of this runs on from Part VII. A field that looks like forty papers a year is, from where this course stands, six shapes of one matrix, and you knew all six before this chapter started. That is what Part VIII is for: watching the course’s own mathematics get hired. It grows from here; the arithmetic of quantization deserves a chapter of its own, and the shelf of methods will have changed again by the time you read this, which is exactly why the map is drawn in shapes instead of names.