Phase 05 — PEFT: LoRA, QLoRA & Knowledge Distillation

The phase where adaptation stops being a luxury. Phase 00 taught you that a full fine-tune holds the weights + gradients + optimizer state + activations in memory — ~16 bytes/param, so a 7B model needs ~112 GB before activations, and a 65B model is hopeless on commodity hardware. This phase is how the industry got around that: don't move the giant weights — freeze them and learn a tiny correction. LoRA, QLoRA, and distillation are the three techniques that turn "fine-tuning needs a cluster" into "fine-tuning needs one GPU and an afternoon."

Why this phase exists

A Senior AI Engineer is asked, constantly: "We need this model to do our task — how?" The naive answer (full fine-tune) is usually wrong on cost, and the lazy answer ("just prompt it harder") is usually wrong on quality. The senior answer is a menu:

• LoRA — freeze W, learn a rank-r update ΔW=(α/r)·A·B. You train r·(d+k) params instead of d·k — often <1% of the model — and at inference you merge() the delta back so there is zero added latency. One base, many cheap swappable adapters.
• QLoRA — store the frozen base in 4-bit NF4, keep the small LoRA adapters in 16-bit, and you can fine-tune a 65B model on a single 48 GB GPU. The base never trains, so its quantization error never compounds.
• Distillation — when you want a permanently smaller model, train a small student on a big teacher's softened logits. The student inherits the teacher's "dark knowledge" (the relative probabilities of the wrong answers), often matching it at a fraction of the inference cost — which, recall Phase 00, is the bill you pay forever.

These are not interchangeable. LoRA/QLoRA adapt a frozen model cheaply; distillation shrinks one permanently. Knowing which lever a problem wants — and being able to defend it with the param-count and memory math — is the skill this phase installs.

What to recall from earlier phases

• Phase 00 (cost math). Training memory ≈ 16 bytes/param (weights + gradients + Adam moments + fp32 master), and inference cost is 2N/token paid forever. Both facts are the reason PEFT and distillation exist; this phase is their direct payoff.
• Phase 00 (quantization). Weights at 0.5 bytes/param (int4) vs 2 (fp16) is the difference between two GPUs and one — QLoRA puts that lever under fine-tuning.
• Phase 02/03 (the transformer & autograd). You know a block is attention projections (q,k,v,o) plus an MLP, and what a gradient/optimizer step costs — PEFT decides which of those weights get gradients at all.

Concept map

                  ┌──────────────────────────────────────────────┐
                  │  Full fine-tune is too expensive (P00 math):  │
                  │  weights + grads + Adam state ≈ 16 B/param     │
                  └──────────────────────────────────────────────┘
                          │                          │
            ADAPT a frozen model              SHRINK to a new model
                          │                          │
            ┌─────────────┴─────────────┐            ▼
            ▼                            ▼      DISTILLATION
          LoRA                        QLoRA      teacher → student
   ΔW = (α/r)·A·B               4-bit NF4 base    softmax(z/T)
   A:(d×r) gauss                + 16-bit LoRA     L = T²·KL(t‖s)
   B:(r×k) ZERO  ─── start = pretrained          + α·CE(s, y)
   train r·(d+k) ≪ d·k          double-quant      "dark knowledge"
            │                   paged optimizer          │
            ▼                          │                 ▼
        merge() ⇒ 0 latency            ▼          smaller N ⇒ less
   swappable adapters         65B on one GPU       inference cost forever
            └──────────────┬───────────┴─────────────────┘
                           ▼
                THE PEFT DECISION (recall: inference cost is forever)
        which layers? what rank? adapt-and-serve, or distill-and-replace?

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Size a LoRA fine-tune: for a 7B model, count the trainable params if you adapt q,v at r=8 across 32 layers; show it is well under 1% and that the optimizer state shrinks by the same factor — the reason it fits one GPU.
• Defend QLoRA on a 65B: compute the 4-bit base footprint (~35 GB) vs the fp16 base (130 GB), add the tiny adapter, and show why one 48 GB GPU now works.
• Adapter zoo: one frozen base + 20 task adapters at a few MB each vs 20 full fine-tuned copies; compute the storage and the swap cost.
• Distill, don't scale (Phase 00 callback): a 70B teacher vs a distilled 7B student at your QPS — compute the perpetual $/1M gap and the breakeven volume.
• Pick the lever: given "we need 5 niche tasks on shared infra" vs "we need one task served at 50k QPS forever," argue LoRA-zoo vs distillation with numbers.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can derive ΔW=(α/r)·A·B and explain the orientation (A:d×r, B:r×k).
• You can explain why B is zero-initialized in one sentence and what breaks if it is not.
• You can compute the LoRA trainable fraction for a given (d,k,r) in your head.
• You can explain QLoRA's three tricks (NF4, double quantization, paged optimizer) and why the base never training is what makes 4-bit safe.
• You can write the distillation loss from memory and explain T and the T² factor.

Key takeaways

• Adapt the update, not the weights. Weight updates during fine-tuning are empirically low-rank, so a rank-r A·B captures most of the gain at r·(d+k) params. This is the whole hypothesis behind LoRA — and why it works at all.
• B zero-init is the design, not a detail. It makes the adapted model identical to the pretrained one at step 0, so training only ever adds skill from a known-good starting point. Lose this and you start from a randomly corrupted model.
• QLoRA decouples storage precision from training precision. The base is frozen, so its 4-bit error is fixed and never compounds; only the high-precision adapters learn. That decoupling is what fits a 65B fine-tune on one GPU.
• Distillation moves capability into a smaller N. Softened logits carry far more signal than hard labels, so the student learns the teacher's function, not just its answers — and a smaller N is cheaper to serve on every request, forever (P00).
• The lever depends on lifetime, not vibes. Many cheap swappable tasks → LoRA zoo. One task served at huge scale → distill and replace. Memory-constrained training → QLoRA. Defend it with the math, not the hype.

Next: Phase 06 — Quantization & Model Compression.