Hitchhiker's Guide — PEFT: LoRA, QLoRA & Distillation

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember."

The 30-second mental model

Full fine-tuning costs ~16 bytes/param (weights + grads + Adam + master) — a 7B model is 112 GB before activations. PEFT freezes the giant and trains something tiny. LoRA: the fine-tuning update is low-rank, so learn ΔW=(α/r)·A·B (A:d×r, B:r×k) instead of all d·k weights — under 1% trainable. B is zero-init so the model starts as the pretrained one; merge() folds the delta back for zero inference cost. QLoRA: store the frozen base in 4-bit NF4 so a 65B fine-tune fits one GPU. Distillation: train a small student on a big teacher's softened logits to shrink N permanently. LoRA adapts cheaply; distillation shrinks.

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "How many trainable params if I LoRA q,v of a 7B at r=8, 32 layers, d=4096?"
per matrix r·(d+k) = 8·(4096+4096) = 65,536;  ×2 (q,v) ×32 layers ≈ 4.2M  (~0.06% of 7B)

# "Does a 65B fit on one 48GB GPU for fine-tuning?"
fp16 base 65e9·2 = 130 GB → no.  NF4 base 65e9·0.5 = 32.5 GB + tiny adapter → yes (QLoRA)

# "Quantization error if I use a 16-level grid on a block with absmax 0.8?"
step = 2/(16-1) = 0.133;  max err ≈ 0.8 · 0.133/2 ≈ 0.053  (halve it → double the levels)

# "student == teacher, T=2, alpha=0.5 — what's the loss?"
KL = 0 → loss = 0.5 · CE(student, y).  The soft term vanishes; only the hard label remains.

The framework one-liners (where these numbers live in real tools)

from peft import LoraConfig, get_peft_model
cfg = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"],
                 lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(base, cfg)
model.print_trainable_parameters()      # "trainable: 4.2M (0.06%)" — the headline
# ... train ...
merged = model.merge_and_unload()       # fold ΔW into the base → zero inference overhead

# QLoRA: 4-bit frozen base + LoRA adapters
from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype="bfloat16")
# optimizer: paged_adamw_8bit  (pages optimizer state to CPU on a spike)

# Distillation: KL on log-softmax of softened logits, plus the T² and the hard CE
loss = (T*T) * KLDivLoss(reduction="batchmean")(log_softmax(student/T), softmax(teacher/T)) \
       + alpha * cross_entropy(student, labels)

War stories

• The random-B disaster. Someone "improved" the init by making both A and B random. The model's loss spiked on step 0 — they had injected noise straight into the pretrained weights — and it never fully recovered. B is zero for a reason: start at the pretrained model, not near it.
• The unmerged-adapter latency ticket. Inference was mysteriously 15% slower than the base. Cause: the adapter was deployed unmerged, adding a second matmul per layer. One merge_and_unload() and it vanished. Merge before you ship a single task.
• The QLoRA OOM that wasn't the base. A 70B QLoRA fit fine, then OOM'd mid-epoch. It was the optimizer-state spike under gradient checkpointing — fixed by paged_adamw_8bit (page to CPU on the spike), not a smaller model.
• The teacher with the cold temperature. A distillation run barely beat hard-label training. The teacher's softmax was nearly one-hot (T=1), so there was no dark knowledge to transfer. Bumping T to 4 (and keeping the T² factor) unlocked it.
• The "we fine-tuned and it forgot everything" regression. A LoRA over-trained on 200 examples nailed the task and got worse at everything else (catastrophic forgetting). The fix was fewer epochs, a held-out general-capability eval, and more data.

Vocabulary (rapid-fire)

• PEFT — parameter-efficient fine-tuning: freeze the base, train something tiny.
• LoRA — low-rank adaptation; learn ΔW=(α/r)·A·B.
• r / α — rank (capacity) / scaling (magnitude); α/r scales the delta.
• merge — fold ΔW into W for zero inference overhead.
• QLoRA — LoRA on a 4-bit (NF4) frozen base; 65B on one GPU.
• NF4 — NormalFloat4: 16 quantile-spaced 4-bit levels for normal-ish weights.
• double quantization — quantize the per-block scales too (~0.4 bits/param saved).
• paged optimizer — page optimizer state to CPU on a memory spike.
• distillation — train a small student on a big teacher's softened logits.
• dark knowledge — the teacher's wrong-answer probabilities; the signal hard labels lack.
• temperature T — softens logits before softmax; T² rescales the soft loss.
• SFT — supervised fine-tuning on instruction/response pairs (usually LoRA/QLoRA).
• catastrophic forgetting — a fine-tune degrading unrelated capabilities.

Beginner mistakes

• Random-initializing B (or both A and B) instead of zeroing exactly one.
• Shipping an unmerged adapter for a single task and eating needless latency.
• Tuning α and the learning rate at once (they both scale the update — pick one).
• Thinking QLoRA trains in 4-bit (it stores 4-bit, dequantizes per block, trains 16-bit).
• Using one tensor-wide quant scale and letting an outlier crush everyone's precision.
• Dropping the T² factor and wondering why the soft term does nothing at high T.
• Computing softmax/KL without max-subtraction / log-space and getting nan.
• Treating LoRA and distillation as interchangeable — one adapts, one shrinks.
• Blaming r when the real problem is the dataset.

The one thing to take away

The update is low-rank, so freeze the giant and learn (α/r)·A·B with B zero-initialized — you start at the pretrained model, train under 1% of the params, and merge() for zero inference cost. When you need the model smaller (not just adapted), distill it on softened logits. Adapt cheap, or shrink permanently — and know which one the problem is asking for.