Warmup Guide — Fine-tuning & Instruction Following

Zero-to-expert primer for Phase 06: how a next-token predictor becomes an assistant — SFT and chat templates, LoRA/QLoRA mechanics, and the preference-tuning landscape (RLHF → DPO) — assuming Phase 05's training fluency.

Table of Contents

Chapter 1: The Gap Between a Language Model and an Assistant

A pretrained LM completes text — ask it "How do I bake bread?" and a plausible continuation is more questions ("How long does it take? What flour should I use?"), because the internet contains question lists. Nothing in pretraining says "answer."

Alignment closes the gap in (classically) three stages:

  1. SFT (this phase's core): train on (instruction → good response) pairs — teaches the format and role of being an assistant.
  2. Preference tuning (RLHF/DPO, Ch. 6): teach which of two responses is better — quality, harmlessness, tone — things hard to demonstrate but easy to compare.
  3. The base model's knowledge comes along for free: alignment is widely understood as mostly eliciting and formatting capabilities pretraining built, not adding knowledge — the "superficial alignment hypothesis" (LIMA: 1K excellent examples sufficed for format). This framing predicts both fine-tuning's cheapness and its limits (Ch. 7).

Chapter 2: Supervised Fine-Tuning — Mechanics That Matter

SFT is Phase 05's training loop with three changes that carry all the difficulty:

  • Loss masking: the example is prompt + response, but loss is computed on response tokens only (prompt positions set to ignore_index=-100). Training on prompt tokens teaches the model to generate prompts — a real and common bug whose symptom is the model continuing user-style text. (You met the identical masking in the model-accuracy track's multimodal lab — same mechanism, same off-by-one risk at the boundary.)
  • Hyperparameters shrink: LR ~1e-5–2e-5 (vs 1e-3-ish pretraining), 1–3 epochs, small batches. You are adjusting a finished model, not training one — overcooked SFT shows as repetitive, sycophantic, distribution-collapsed outputs.
  • Data quality dominates data quantity: 1K–10K excellent, diverse examples beat 100K scraped mediocre ones (LIMA's lesson, repeatedly replicated). Curation — dedup, decontamination (your eval sets! — Phase 08), length/format balance — is most of the actual work. Catastrophic forgetting is the tax: narrow SFT data measurably degrades broad capabilities; mitigations are mixing in general data and short training.

Chapter 3: Chat Templates — The Underrated Contract

Multi-turn conversations serialize to one token stream via a template — special tokens marking roles and turns (Phase 01 Ch. 6's special-token machinery, now load-bearing):

<|im_start|>system ... <|im_end|> <|im_start|>user ... <|im_end|> <|im_start|>assistant ...

The facts that cause production incidents:

  • The template is part of the model's weights-contract: it must match training exactly — wrong template degrades quality silently (no error, just a worse model). This is the #1 cause of "this model is bad actually" reports; check tokenizer.apply_chat_template against the model card before any other debugging.
  • EOS/EOT discipline: training must teach the model to emit the end-of-turn token (loss on that token!), else generation never stops — the "model rambles forever" bug is usually a masking bug at the turn boundary.
  • In multi-turn training data, mask everything except assistant turns; whether to train on all assistant turns or only the last is a real design choice (all-turns is standard; it reuses context efficiently).

Chapter 4: LoRA — Fine-Tuning as a Low-Rank Update

Full fine-tuning of a 7B model needs weights + grads + Adam states ≈ 70+ GB (Phase 05 Ch. 2's 2×-FP32 fact) — and produces a full copy per task. LoRA: freeze $W$; learn a low-rank delta:

$$W' = W + \frac{\alpha}{r} BA, \qquad B \in \mathbb{R}^{d \times r} \text{ (init 0)}, \quad A \in \mathbb{R}^{r \times k} \text{ (init gaussian)}, \quad r \in [4, 64]$$

  • Why it works: task adaptation empirically has low intrinsic rank — the change needed is far simpler than the model. B=0 init means training starts exactly at the base model (no cold-start damage); $\alpha/r$ decouples update magnitude from rank.
  • What to adapt: classically attention's $W_q, W_v$; modern practice (QLoRA paper's finding) adapts all linear layers — with rank low, parameters stay <1%.
  • Trainable fraction: r=16 on a 7B → ~0.1–0.5% of parameters; optimizer state shrinks proportionally — this, not gradient compute, is the memory win.
  • Serving: merge ($W + \frac{\alpha}{r}BA$, zero inference overhead, loses swappability) vs keep-separate (tiny extra matmul; hot-swappable per-task adapters over one base — multi-tenant LoRA serving (S-LoRA-style) is a Phase 09-adjacent production pattern worth naming in interviews).

Chapter 5: QLoRA — Training on a Quantized Base

QLoRA composes LoRA with a 4-bit frozen base — backprop flows through the quantized weights into FP16 adapters:

  • NF4: 4-bit grid placed at the quantiles of a standard normal — equal probability mass per level for normal-ish weights — information-theoretically optimal storage when you'll dequantize to float for compute (vs uniform INT4, which is what integer arithmetic hardware wants — the storage-vs-compute distinction).
  • Double quantization: the per-block (64) absmax constants are themselves quantized — ~0.4 bits/param saved (~370 MB at 7B).
  • Paged optimizer states handle memory spikes.
  • Net: 7B fine-tune in ~10 GB, 65B in 48 GB — the democratization moment of 2023, and the lab's actual configuration. Quality: QLoRA ≈ LoRA ≈ full-FT on instruction tasks at matched data (the paper's Guanaco result) — with the caveat that aggressive new domains stress the frozen-base assumption.

(The deep quantization math — and using adapters to recover quantization accuracy — lives in the model-accuracy track's Phase 03/10 warmups; cross-reference rather than re-derive.)

Chapter 6: Preference Tuning — RLHF and DPO

SFT teaches format; preferences teach better-vs-worse — judgments easy to collect pairwise, hard to demonstrate.

  • RLHF (the classic pipeline): (1) train a reward model on human preference pairs (Bradley-Terry loss on chosen-vs-rejected); (2) optimize the policy against it with PPO, with a KL penalty to the SFT model as the leash. Why the leash: unconstrained reward maximization finds the RM's blind spots — reward hacking (length inflation, sycophancy, weird token exploits). RLHF works (it built ChatGPT) but is operationally heavy: four models in memory (policy, reference, RM, value), RL instability, and the RM's quality caps everything.
  • DPO (the 2023 simplification): the KL-constrained RLHF objective has a closed-form optimal policy; inverting it turns preference optimization into a supervised loss on the policy directly:

$$\mathcal{L} = -\log \sigma!\left(\beta\left[\log\tfrac{\pi(y_w|x)}{\pi_{ref}(y_w|x)} - \log\tfrac{\pi(y_l|x)}{\pi_{ref}(y_l|x)}\right]\right)$$

— raise the chosen response's likelihood relative to the reference, lower the rejected's, with $\beta$ playing the KL knob. No reward model, no RL loop, two models in memory. Tradeoffs to know: DPO is bounded by its offline preference data (no exploration), can over-optimize toward verbosity too, and variants (IPO, KTO, ORPO) patch specific failure modes. Industry reality: DPO-family for most open work; RLHF/ online methods at the frontier labs.

Chapter 7: What Fine-Tuning Can and Cannot Do

The judgment chapter — when an inference engineer should say "don't fine-tune":

  • Great for: format/style/persona, task specialization (SQL, extraction schemas), tool-calling patterns, domain vocabulary, latency wins (a tuned 8B replacing a prompted 70B).
  • Poor for: adding knowledge — facts injected by fine-tuning are brittle and hallucination-prone (the model learns to assert in-domain rather than to know); retrieval (RAG, Phase 07) is the right tool for knowledge, fine-tuning for behavior. The slogan that survives scrutiny: RAG for what the model should know, fine-tuning for how it should act.
  • Dangerous for: safety properties (fine-tuning on even benign data measurably erodes refusal training — a real deployment consideration), and anything where eval contamination from your SFT data corrupts your metrics (Phase 08's discipline).
  • The decision ladder to recite: prompt engineering → few-shot → RAG → fine-tune → pretrain-from-scratch, escalating only when the cheaper rung measurably fails.

Lab Walkthrough Guidance

Lab 02 — LoRA & QLoRA:

  1. Implement the LoRA module yourself before importing PEFT: wrap a frozen nn.Linear, add $BA$ with B-zero init; test that step 0 output equals the base model exactly, and that only adapter params receive grads (the same two tests as the model-accuracy multimodal lab — the freezing discipline generalizes).
  2. Prepare the instruction dataset with explicit loss masking; print one fully rendered example (template + mask visualized) and check the EOT token carries loss — five minutes here prevents the two classic failure modes (Ch. 2–3).
  3. SFT a small base (1–3B class) with LoRA; track val loss and generations on fixed prompts; verify the assistant-formatting behavior emerges.
  4. Switch to QLoRA (4-bit NF4 base): match LoRA's quality within noise while logging peak VRAM for both — the memory table is the deliverable.
  5. Ablate: r ∈ {4, 16, 64} and attention-only vs all-linear targets — quality vs adapter size; then merge the best adapter and verify merged == unmerged outputs.

Success Criteria

You are ready for Phase 07 when you can, from memory:

  1. Explain the three alignment stages and the superficial-alignment hypothesis with its evidence and limits.
  2. State the SFT loss-masking rule, the two bugs it prevents, and the EOS-discipline bug's symptom.
  3. Write the LoRA update with both inits and the $\alpha/r$ role; give the trainable-% and the merge-vs-swap tradeoff.
  4. Explain NF4-vs-INT4 as storage-vs-compute optimality and double quantization's saving.
  5. Sketch RLHF's pipeline with the KL leash's purpose, then derive DPO's pitch (closed- form inversion → supervised loss) and its offline limitation.
  6. Recite the RAG-vs-fine-tuning division and the escalation ladder.

Interview Q&A

Q: A team fine-tuned on their product docs and the model now confidently invents product features. What happened and what should they do? Fine-tuning taught behavior (assert fluently about the product domain) not knowledge (the docs' facts aren't reliably stored). In-domain confidence without in-domain grounding = amplified hallucination. Fix: RAG over the docs for facts — fine-tune only for format/tone if needed; keep an eval set of answerable+unanswerable product questions, and measure refusal-when-ungrounded. The conceptual error — fine-tuning as knowledge injection — is the thing to name.

Q: Why does DPO work without a reward model — what's the trick? The RLHF objective (maximize reward minus β·KL-to-reference) has a closed-form optimum: $\pi^(y|x) \propto \pi_{ref}(y|x)\exp(r(y,x)/\beta)$. Solve for $r$ in terms of $\pi^/\pi_{ref}$ and substitute into the Bradley-Terry preference likelihood: the reward cancels into log-likelihood-ratios of the policy itself. The "reward model" is implicit in the policy — you optimize it directly by supervised learning on preference pairs. The cost: you inherit the offline data's coverage; no exploration.

Q: When would you choose full fine-tuning over LoRA today? When the adaptation is large-rank by nature: substantial new domains/languages, changing base behaviors deeply, or continued-pretraining-scale token budgets — measured by LoRA-vs-full ablation gaps, not vibes. Also when serving exactly one task at scale (merge erases LoRA's deployment advantage) and you have the memory anyway. For the standard instruct/persona/task tune, LoRA-family matches quality at ~1% of the optimizer footprint, so it's the default; the burden of proof sits on full FT.

References