The Adapter Field Guide

The idea in one sentence

Lecture 29 asked one question, what shape is $\Delta W$, and this guide answers it forty-six times: every method in the PEFT library’s tuner folder, one drawing each, sorted into families by shape.

How to read this guide

Every drawing uses the same grammar, so the figures teach by contrast. Ink outlines are frozen structure. Green is what trains. Cyan is frozen random material. Yellow marks chosen cells, purple marks prompt vectors, and blue marks a rotation or an action. When a figure shows a number, it was measured in Lecture 29’s run on this machine; when it quotes a claim, the source is named. Every mechanism below was checked against the library’s source code (peft 0.20.0) before its figure was drawn; the drawing shows what the code does, and where a method’s finer points go beyond what could be verified, the entry says only what could.

Each entry closes with a few lines of pseudo-code: the forward pass, with what trains named in the comments. They are shapes, not runnable programs; Lecture 29 has the runnable one, and any method here drops into its sixty-step run by changing one config line.

This is a reference chapter, built to be skimmed, stepped through figure by figure, and come back to. Read one family when you need it. The families:

1. Add a thin product, LoRA’s line.
2. A structured product.
3. A few coefficients in a fixed basis.
4. Rotations.
5. Surgery on the weight’s own subspaces.
6. Cheap vectors.
7. Leave the weights alone, work on the input.
8. Routers and mixtures.

Add a thin product

The founding family. Everything here writes $\Delta W = CR$ or a close relative, and everything inherits the two facts Lecture 29 measured: born at zero, merges away.

LoRA is the reference specimen, the method the whole of Lecture 29 walks through: a thin $C$ times a wide $R$, the waist first drawn in Lecture 6, 0.22 percent of the model in training on this machine’s run.

h = W @ x                        # frozen
h += (alpha / r) * C @ (R @ x)   # C starts at 0: day one is the base

AdaLoRA notices that a fixed $r$ everywhere is a guess. It scores each direction’s importance while training and moves rank away from layers that waste it, so the same budget ends up wide where the task needs width and starved where it does not.

h = W @ x + C @ (R @ x)          # the stage, unchanged
rank[layer] = budget(importance) # the split is learned while training

DoRA splits each output channel of the adapted weight into a direction and a length, and trains the lengths as their own small dial. The decomposition costs one number per channel and lets training adjust how strongly a channel speaks separately from where it points.

V = W + C @ R                    # the adapted weight
h = m * unit_columns(V) @ x      # direction pinned; the length m trains

HiRA wants a high-rank change at low-rank cost, so it multiplies where LoRA adds: the update is $W \circ (CR)$, the frozen weight rescaled entry by entry through a low-rank pattern. Because $W$ itself has full rank to lend, the product can move far more directions than $r$.

h = (W + W * (C @ R)) @ x        # entrywise: W is rescaled, not added to

DeLoRA puts the update on a leash: the change is kept at a controlled size, with a learned bound $\lambda$ capping how far the adapted weights can drift from where they started. The circle from Lecture 29’s rotation figure returns, now as a budget.

delta = C @ R                    # a thin update
h = (W + cap(delta, lam)) @ x    # drift capped at lam, and lam trains

GLoRA refuses to pick one place to adapt. It gives the layer five slots, the weight becoming $W + W \circ A + B$ with three more slots for the bias, and each slot can be a low-rank pair, a single vector, a scalar, or off, so one method covers scaling, shifting, and bias editing at whatever budget each slot deserves.

h = (W + W * A + B) @ x          # A rescales W, B adds a path
b2 = b * D + E                   # and the bias gets its own sockets

GraLoRA cuts the matrix into a $k \times k$ grid of blocks first and gives every block its own small waist. The parameter count matches plain LoRA at the same $r$, but the update is assembled from local pieces, so its total rank can reach $k$ times further.

for i, j in rooms(W, k):         # k by k rooms
    dW[i, j] = C[i, j] @ R[i, j] # each room, its own waist

Lily attacks depth. Neighboring layers share one thin factor, and a small global pool of wide factors serves the whole model through a learned router, so the bill stops growing linearly with the number of layers.

z = C_shared @ x                 # neighbors share the thin factor
h += blend(router, pool_R) @ z   # a learned mix over few wide factors

PEANuT gives the update a hidden layer. Instead of the linear $CR$, the adapter runs its input through a thin projection, a small nonlinear stack, and a wide projection out, so the weight update it expresses can bend in ways no single low-rank product can.

h = W @ x + B @ phi(A @ x)       # phi bends; a linear update cannot

TinyLoRA is the family’s extremist. It keeps a frozen SVD of the weight, routes a handful of trainable numbers through fixed random projections, and steers the model with a parameter count you can say out loud: its paper is titled “Learning to Reason in 13 Parameters.”

v = params(13)                   # the whole trainable state
dW = frozen_svd @ (frozen_rand @ v)   # structure does the spreading

UniLoRA makes the budget literal: one shared pool of numbers $\theta$ for the whole model, and every entry of every adapter factor is a lookup into it through frozen random wiring. Training moves the pool; the wiring never changes.

C[i, j] = theta[wire_C[i, j]]    # every entry: a lookup into theta
R[i, j] = theta[wire_R[i, j]]    # the wiring is random and frozen

VB-LoRA does the same with structure: a bank of small vectors, shared model-wide, and every row of every factor is a mixture of the bank’s top few entries, chosen and weighted by training. The factors never own their numbers; they borrow them.

winners, w = top2(logits[row])   # scores choose two bank vectors
row = w[0] * bank[winners[0]] + w[1] * bank[winners[1]]

The family also carries dials that change LoRA’s birth rather than its shape. rsLoRA scales the update by $\alpha/\sqrt{r}$ so large ranks stay stable. PiSSA initializes the factors from $W$‘s own principal directions; OLoRA starts them orthonormal via QR, the factorization from Lecture 13; EVA starts from the directions the task’s data actually uses; LoftQ starts where quantization error is smallest, for QLoRA-style training; CorDA builds its start context-first. Same shape, different first step.

C, R = init(method)              # PiSSA, OLoRA, EVA, LoftQ, CorDA...
h = W @ x + scale * C @ (R @ x)  # the forward pass never changes

Three field notes for this family, all measured by Schulman and colleagues in LoRA Without Regret and unpacked in Lecture 29. Attach adapters to every matrix, with the MLPs first, because attention-only adapters learn slower even at matched parameter counts. Port hyperparameters by the ten-times rule: LoRA’s best learning rate sits near ten times full fine-tuning’s. And size the rank by the bit budget: when the adapter’s capacity exceeds what the dataset can teach, these methods match full fine-tuning step for step, which for reinforcement learning happens already at rank one.

A structured product

The second family builds $\Delta W$ from small pieces with algebraic structure, so a few numbers reach a large matrix.

LoKr uses the Kronecker product. Two small trained factors stamp a full-size update, every entry of the first scaling a whole tile shaped like the second; Lecture 29 measured a $28 \times 28$ and a $32 \times 32$ standing in for the full $896 \times 896$, 1,808 numbers for 802,816 slots.

dW = kron(F, G)                  # each entry of F stamps a tile of G

LoHa multiplies two thin products entry by entry, $(C_1 R_1) \circ (C_2 R_2)$. The bill is exactly twice LoRA’s, 2,162,688 against 1,081,344 in the measured table, and the reward is rank: an entrywise product of two rank-$r$ matrices can reach rank $r^2$.

dW = (C1 @ R1) * (C2 @ R2)       # entrywise: the ranks multiply

MiSS reshapes the weight into equal tiles and trains one small block that updates every tile at once, part multiplied in, part added, so a single $r \times r$ block moves the whole matrix.

for tile in tiles(W):            # equal tiles
    dW[tile] = block             # one r by r block, broadcast

C3A trains one row per block and lets rotation write the rest: each block of $\Delta W$ is circulant, every row a shifted copy of the first. A circulant matrix is a convolution in disguise, which is why the method can apply it fast with Fourier transforms, a trick Part VI will build properly when it reaches convolution.

row = params(n)                  # one row per block trains
dW_block = circulant(row)        # shifts write the rest; apply by FFT

A few coefficients in a fixed basis

The third family stores no matrix shape at all. Pick a basis once, train a few coordinates in it, and let the transform build the dense update.

FourierFT trains chosen positions in the frequency plane, 1,000 per matrix in the measured run, 0.0097 percent of the model, and an inverse Fourier transform turns them into a full-size, generally full-rank $\Delta W$.

spectrum[chosen] = coeffs        # 1,000 trained positions
dW = ifft2(spectrum)             # the fixed basis builds the update

WaveFT makes the same bet in a wavelet basis, whose coefficients carry scale and position at once, so a coefficient can say “here, and this fine.”

dW = inverse_wavelet(coeffs)     # each coefficient: a scale and a place

SHiRA picks the simplest basis of all, the entries themselves: a fixed sparse mask of positions, $r(m+n)$ of them per matrix, each trained directly. No factors, no transform, and because the scatter is spread across the whole matrix, the update is high-rank from the start.

dW[mask] = values                # the entries themselves train
                                 # the mask is placed once, by a seed

Rotations

The orthogonal family from Lecture 29, expanded. Everything here multiplies $W$ by an orthogonal $R$ instead of adding to it, and inherits the guarantee Lecture 13 proved: lengths cannot change.

OFT keeps $R$ block-diagonal and stores only skew-symmetric entries, which the Cayley transform turns into an orthogonal block. The rotation is an algebraic certainty; training cannot break it even by accident.

R = cayley(skew)                 # skew in, orthogonal out, always
h = (R @ W) @ x                  # turned, never stretched

BOFT builds the same promise the way an FFT builds a dense transform: from stages of sparse butterfly rotations that compose into a dense one, reaching every coordinate in a few crossing layers.

R = stage2 @ stage1              # sparse butterfly turns compose
h = (R @ W) @ x

HRA composes mirrors. Each factor is a Householder reflection, the $I - 2uu^{\T}$ from QR’s engine room, and only the mirror normals train; a chain of $r$ reflections turns $W$ without ever being able to stretch it.

for u in normals:                # r mirror normals train
    W = W - 2 * u @ (u.T @ W)    # each reflection keeps every length

RoAd is the cheapest turn in the library: take the output coordinates two at a time and give each pair its own 2D rotation angle and scale. Its smallest variant stores about one number per output channel.

for i, j in pairs(h):            # two coordinates at a time
    h[i], h[j] = turn(h[i], h[j], angle, scale)

PSOFT rotates inside the waist. It freezes $C$ and $R$ from the weight’s own SVD and trains only a small $r \times r$ orthogonal transformation between them, so adaptation happens strictly inside the principal subspace.

B, A = svd_top(W, r)             # fixed from W's own SVD
h = W @ x + B @ ((R - I) @ (A @ x))   # only the turn R trains

Surgery on the weight’s own subspaces

Three methods start by asking what $W$ already contains, using the subspace language of Lecture 9, and operate on the answer.

OSF splits $W$ by its own SVD, freezes the top directions where old knowledge lives, and trains only in the leftover subspace, so a new task cannot trample what the model already does. It is built for learning tasks in sequence.

top, tail = split_svd(W)         # strong directions, and the rest
W = top + train(tail)            # old knowledge stays locked

DEFT is the surgeon: it learns a projection that erases a chosen slice of what $W$ says, the projection algebra of Lecture 11, and writes new content in the erased slice, with the whole operation equal to the identity at birth.

h = ((I - P) @ W + Q) @ x        # P erases a slice, Q refills it
                                 # at birth P = 0 and Q = 0: identity

AdaMSS decomposes the weight by SVD, clusters the directions into a few groups, and gives each cluster its own small trainable update, so the adapter’s structure follows the weight’s own geography.

groups = cluster(svd_dirs(W))    # directions, grouped by likeness
dW = sum(update[g] for g in groups)

Cheap vectors

The family that refuses to buy matrices. Everything here trains vectors, and the measured rows in Lecture 29’s table are the smallest for exactly that reason.

IA3 trains one number per channel and multiplies the activations by it: 141,312 numbers on the measured model, 0.0286 percent, and no matrix anywhere.

h = W @ x                        # frozen
h = h * ell                      # one trained dial per channel

BEFT is the humblest entry in the guide: it trains only additive bias vectors and leaves every matrix alone. Move the intercepts, keep the slopes.

h = W @ x + b                    # only b trains: intercepts, not slopes

LN tuning trains only the normalization layers, the small gain-and-shift vectors that sit between the big frozen blocks, retuning the thermostat while leaving the machine.

z = (x - mean(x)) / std(x)       # built into the norm
h = g * z + shift                # only g and the shift train

VeRA is Lecture 29’s shared-pair trick, here in its guide form: one frozen random pair for the whole model, two small trained vectors per layer, 27,648 numbers in the measured run.

A, B = rand(seed)                # one frozen pair, model-wide
h = W @ x + b * (B @ (d * (A @ x)))   # d and b train, per layer

PVeRA keeps VeRA’s silhouette, one shared pair and two small trainable vectors per layer, and refines how the shared pair is built and applied.

A, B = prepare(rand(seed))       # the pair, prepared before use
h = W @ x + b * (B @ (d * (A @ x)))   # the same vectors train

RandLoRA stacks several frozen random bases and trains small diagonal mixes that combine them; summed low-rank pieces can reach full rank, at vector prices.

dW = sum(lam[k] * basis[k])      # frozen pieces, trained amounts

FRoD freezes two random patterns per matrix, one dense low-rank piece and one sparse one, and trains only the small vectors that blend them, covering both smooth and spiky updates from a fixed random menu.

dW = lam1 * dense + lam2 * sparse    # both patterns from a seed

Trainable tokens goes surgical on the embedding table from Lecture 1: train the rows of a few chosen tokens, touch nothing else. Teach the model two new words for the price of two vectors.

E[chosen_rows] = trained         # a few embedding rows
E[everything_else] = frozen

Leave the weights alone

The soft-prompt family from Lecture 29, completed. Nothing here touches $W$ at all; everything edits the vectors the frozen model reads.

Prompt tuning prepends $k$ learned vectors to the input, 14,336 numbers in the measured run, the smallest row of the whole table.

tokens = concat(P, embed(text))  # P: k learned vectors; W never moves

Prefix tuning walks the prompts in deeper: learned key and value vectors join the attention of every layer, so the steering happens at depth instead of only at the entrance.

K = concat(K_learned, K_text)    # every layer's attention
V = concat(V_learned, V_text)    # gets its learned visitors

P-tuning adds a writer: a small MLP or LSTM produces the prompt vectors during training, smoothing their optimization, and is thrown away once the vectors are learned.

P = writer(z)                    # an MLP or LSTM writes the prompt
serve(P)                         # afterward, the writer is discarded

CPT mixes learned prompt tokens into real context tokens and keeps their updates on a leash, projecting each step back toward where the token began, so the prompt stays close to language instead of drifting into arbitrary directions.

P = P + step                     # the update wants to wander
P = pull_back(P, start, radius)  # Lecture 11, used as a guardrail

Multitask prompt tuning learns one shared prompt for all tasks plus a thin per-task tweak of it, multiplied in entrywise, so a new task costs a whisper instead of a full prompt.

P_task = P_shared * tweak[task]  # entrywise; a whisper per task

Adaption prompt, LLaMA-Adapter in the literature, inserts learned prompt tokens into only the top few layers and guards them with a gate initialized at zero, so on day one the model is exactly the base model, the same promise LoRA makes with $C = 0$.

h_top = attn(x) + gate * attn(P) # top layers only
                                 # gate starts at exactly zero

CARTRIDGE stores the prompt after the model has read it: a trained key-value cache, kept as weights and loaded at serving time, so a whole corpus’s worth of context can be distilled once and reused forever.