Supervised Fine-Tuning (SFT)

Phase 13 · Document 01 · Fine-Tuning and Adaptation Prev: 00 — When to Fine-Tune · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

SFT (Supervised Fine-Tuning) is the standard, foundational fine-tuning method — the one you'll use 90% of the time once you've decided to fine-tune (00). It's how a base or instruct model is taught to follow your task's behavior by training on (input → ideal output) examples. Understanding SFT mechanically — that it's just next-token prediction on your examples, so the model learns to imitate exactly what you show it — explains both its power (consistent format/style at scale) and its limits (it copies your data's flaws, overfits, and learns style not facts, 00). Every other method here builds on SFT: LoRA/QLoRA (02) is how to run SFT efficiently; distillation (04) is SFT on a teacher's outputs; DPO (03) goes a step beyond it.

2. Core Concept

Plain-English primer: teach by example, via next-token loss

Pre-training taught the model to predict the next token over the whole internet (Phase 2.05). SFT continues that same next-token training, but on your curated examples — pairs of an input (prompt/conversation) and the ideal output you want. The model adjusts its weights to make your ideal outputs more probable given those inputs. That's it: SFT = supervised next-token learning on (input → ideal completion) pairs.

The profound consequence: the model learns to imitate your examples. It copies their format, style, structure, and patterns — including any mistakes, biases, or inconsistencies in your data ("garbage in, garbage out", 06). This is why data quality dominates (06) and why SFT teaches behavior, not facts (00) — it learns the shape of good answers, not a reliable knowledge store.

The data format (chat/instruction)

SFT data is typically chat-formatted (ChatML-style) — the same role structure the model serves with (what-happens §0):

{"messages": [
  {"role": "system", "content": "You are a terse SQL assistant. Output only the query."},
  {"role": "user", "content": "users who signed up in the last 7 days"},
  {"role": "assistant", "content": "SELECT * FROM users WHERE created_at >= NOW() - INTERVAL '7 days';"}
]}

Key detail — loss masking: during SFT you typically compute the loss only on the assistant (completion) tokens, not the prompt/system tokens. The model learns to produce the ideal response, not to predict your prompts. (Frameworks handle this; know it exists.)

The training knobs that matter

• Epochs — how many passes over the data. Too few → underfit; too many → overfit (memorizes the training set, generalizes worse). 3–5 epochs is a common SFT range; watch eval loss (it should drop then plateau; if it rises, you're overfitting).
• Learning rate — how big each weight update is. Too high destabilizes; too low barely learns. (LoRA uses higher LRs, e.g. ~1e-4–2e-4; full FT uses lower.)
• Batch size / gradient accumulation — throughput vs memory; accumulate gradients to simulate a larger batch on limited VRAM (Phase 6.02).
• Max sequence length — must fit your examples; longer = more memory.
• Base model choice — fine-tune an instruct model (already follows instructions) for most tasks, or a base model for deep stylistic control; the base sets your ceiling (Phase 1.02).

Overfitting and catastrophic forgetting (the failure modes)

• Overfitting — too many epochs / too little data → the model memorizes training examples and fails on variations (great on eval-that-looks-like-train, bad in prod). Mitigate: more/diverse data (06), fewer epochs, watch eval loss, hold out a real eval set (Phase 12.01).
• Catastrophic forgetting — fine-tuning narrows the model: it gets better at your task but worse at others it used to do. Mitigate: LoRA (freezes the base, less forgetting — 02), lower LR/epochs, and eval on out-of-distribution tasks for regressions (Phase 12).

Full SFT vs parameter-efficient SFT

SFT can update all weights (full fine-tuning — best quality ceiling, expensive, heavy VRAM, more forgetting) or just a small adapter (LoRA/QLoRA — cheap, fast, less forgetting, the production default — 02). The method (next-token loss on your pairs) is identical; LoRA just changes which parameters are trained. Most teams do LoRA SFT.

Two paths to run it

• API SFT (OpenAI/others): upload a JSONL of chat examples, pick a base, set epochs, get a fine-tuned model ID — no infra. Simplest; closed models.
• Open-weight SFT (Hugging Face TRL SFTTrainer, Unsloth, Axolotl): you control everything, run LoRA/QLoRA on your GPU, export GGUF for local serving (Phase 6.03). More control + privacy (Phase 5.01).

Always: hold out an eval set

Split your data into train / eval (e.g., 80/20), never train on the eval split (Phase 12.01 no-leakage), and eval before/after against the base for gains and regressions (Phase 12). Watch eval loss during training to catch overfitting early.

3. Mental Model

   SFT = continue PRE-TRAINING's next-token learning, but on YOUR (input → ideal output) pairs
        → model adjusts weights to make YOUR ideal outputs probable → ★ it IMITATES your examples
        (copies format/style/patterns AND your data's flaws → data quality dominates [06]; behavior-not-facts [00])

   DATA: chat/ChatML pairs [what-happens §0]; LOSS MASKED to assistant (completion) tokens only
   KNOBS: EPOCHS (3–5; too many → OVERFIT) · learning rate · batch/grad-accum [6.02] · max-seq-len · base model
   FAILURE MODES: OVERFITTING (memorize, fail on variations) · CATASTROPHIC FORGETTING (worse at other tasks)
        → watch EVAL LOSS; hold out eval set (no leakage [12.01]); eval before/after + OOD regressions [12]

   FULL SFT (all weights) vs LoRA/QLoRA SFT (adapter, production default [02]) — same method, fewer params trained
   RUN: API SFT (upload JSONL) or open-weight (TRL/Unsloth/Axolotl → GGUF [6.03])

Mnemonic: SFT is next-token learning on your (input→ideal) pairs — the model imitates your examples (flaws included), so data quality is everything. Mask loss to completions, watch eval loss to avoid overfitting, hold out an eval set, and prefer LoRA. It teaches behavior, not facts.

4. Hitchhiker's Guide

What to look for first: your (input → ideal output) dataset in chat format and a held-out eval set (Phase 12.01). With clean data + eval, SFT is straightforward; without them it's guesswork.

What to ignore at first: hyperparameter obsession and full fine-tuning. Start with LoRA SFT, 3 epochs, default LR; tune only if eval shows under/overfitting.

What misleads beginners:

• Thinking SFT teaches knowledge. It teaches imitation of your examples (behavior/style), not facts (00).
• Ignoring data quality. The model copies your data's flaws — consistency/quality dominate (06).
• Too many epochs. Overfits/memorizes — watch eval loss, use 3–5 epochs.
• No held-out eval / leakage. You can't detect overfitting or regressions (Phase 12.01).
• Not testing OOD. Catastrophic forgetting shows up only on tasks outside your training distribution (Phase 12).

How experts reason: they treat SFT as imitation learning — so they invest in data quality (06), format as chat with loss on completions, run LoRA SFT with modest epochs, watch eval loss, hold out an eval set (no leakage), and eval before/after for gains and OOD regressions (Phase 12). They pick the right base model (instruct for most) and keep the recipe simple.

What matters in production: the before/after eval (did SFT beat the prompt/RAG baseline?), no OOD regressions (forgetting), the right epoch count (no overfit), and the maintenance cadence (07).

How to debug/verify: "great on eval, bad in prod" → overfit or unrepresentative data (06); "worse at unrelated tasks" → catastrophic forgetting (lower epochs/LR, use LoRA, 02); "hallucinates facts" → you tried to teach knowledge (use RAG, Phase 9); "no improvement" → data quality or the baseline was already enough.

Questions to ask: is the data clean/consistent chat pairs? loss masked to completions? held-out eval (no leakage)? how many epochs vs eval loss? LoRA or full? did I eval before/after + OOD?

What silently gets expensive/unreliable: poor data (imitated flaws), overfitting (memorization), no held-out eval (undetected overfit), forgetting (OOD regressions), and epoch/LR mis-tuning.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• SFT = supervised next-token learning on (input → ideal output) pairs — continuing pre-training on your examples (Phase 2.05).
• The model imitates your examples — copying format/style/patterns and your data's flaws ("garbage in, garbage out") → data quality dominates (06).
• It teaches behavior, not facts (00).
• Loss is masked to completion (assistant) tokens — the model learns to produce ideal responses.
• Watch eval loss; use ~3–5 epochs — too many overfits (memorizes, fails on variations).
• Catastrophic forgetting narrows the model — test out-of-distribution for regressions (Phase 12); LoRA reduces it (02).
• Hold out an eval set (no leakage); eval before/after vs the base for gains and regressions (Phase 12.01).
• Full SFT vs LoRA/QLoRA SFT — same method, different parameters trained; LoRA is the production default (02). Run via API or open-weight (TRL/Unsloth/Axolotl → GGUF Phase 6.03).

9. Observations from Real Systems

• SFT is the workhorse — most production fine-tunes are LoRA SFT on chat-formatted task data (02).
• It's a stage in the alignment pipeline (pre-train → SFT → preference tuning 03) — instruct models are SFT'd base models.
• The classic overfit story: 10 epochs on 50 examples → memorizes them, fails on anything phrased differently → fix with more data + fewer epochs + eval loss (06).
• TRL/Unsloth/Axolotl are the standard open-weight SFT stacks; OpenAI's fine-tuning API is the no-infra path.
• Catastrophic forgetting is real — full fine-tunes on narrow data can degrade general ability; teams use LoRA + OOD eval to catch it (02, Phase 12).

10. Common Misconceptions

11. Engineering Decision Framework

RUN SFT (after deciding to fine-tune [00]):
 1. DATA: clean, consistent (input → ideal output) CHAT pairs; quality > quantity [06]; hold out an EVAL set (no leakage) [12.01].
 2. BASE MODEL: instruct (most tasks) or base (deep style); the base sets the ceiling [1.02].
 3. METHOD: LoRA/QLoRA SFT (default) [02]; full FT only if you need the ceiling and have the VRAM.
 4. HYPERPARAMS: ~3–5 epochs, sane LR, fit max-seq-len, grad-accum for VRAM [6.02]; loss masked to completions.
 5. MONITOR eval loss (drop→plateau; rising = overfit); stop early if needed.
 6. EVAL before/after vs base: gains AND out-of-distribution REGRESSIONS (forgetting) [12].
 7. RUN via API (JSONL) or open-weight (TRL/Unsloth/Axolotl → GGUF [6.03]); deploy + maintain [07].

12. Hands-On Lab

Goal

Run a small LoRA SFT on a behavioral task, watch eval loss to avoid overfitting, and eval before/after for gains and OOD regressions.

Prerequisites

• A behavioral task + a clean dataset of (input → ideal output) chat pairs (06); a GPU (or OpenAI fine-tuning API); pip install unsloth trl datasets (open-weight) or the OpenAI SDK.

Steps

1. Dataset: format ~100–500 examples as chat pairs; split 80/20 train/eval; confirm no leakage (Phase 12.01).
2. Baseline: eval the base/instruct model (with a good prompt) on the eval set (Phase 12) — the bar SFT must beat.
3. LoRA SFT: fine-tune (Unsloth/TRL SFTTrainer or OpenAI API) for 3 epochs; log eval loss each epoch (02).
4. Overfit demo: also run 10 epochs; show eval loss rises (overfitting) and the model memorizes/degrades on variations.
5. Eval before/after: compare base vs fine-tuned on the eval set (target metric) and on an out-of-distribution set (catch forgetting) (Phase 12).
6. Decide: did SFT beat the baseline without OOD regressions? Document.

Expected output

A LoRA-SFT'd model with an eval-loss curve (3 vs 10 epochs), a before/after comparison vs the baseline, and an OOD-regression check — demonstrating SFT-as-imitation, overfitting, and the need for held-out eval.

Debugging tips

• Eval loss rises → overfitting; reduce epochs / add data.
• Hallucinates facts → it's knowledge, not behavior; use RAG (Phase 9).

Extension task

Try full fine-tuning vs LoRA on the same data and compare quality, VRAM, and forgetting (02).

Production extension

Export to GGUF for local serving (Phase 6.03) or deploy the adapter (07); wire before/after eval into a gate (Phase 12.08).

What to measure

Eval loss (3 vs 10 epochs), before/after target metric, OOD regression, train vs eval gap (overfit indicator).

Deliverables

• A LoRA-SFT'd model + an eval-loss curve.
• A before/after comparison vs the baseline + an OOD-regression check.
• A documented epoch/overfit finding.

13. Verification Questions

Basic

1. What is SFT, mechanically (the loss, the data)?
2. Why does "the model imitates your examples" make data quality dominate?
3. What is loss masking in SFT?

Applied 4. How do you detect and prevent overfitting in SFT? 5. What is catastrophic forgetting, and how do you check for it?

Debugging 6. SFT model is great on eval, poor in prod. Two causes. 7. After SFT the model got worse at unrelated tasks. What happened, and the fix?

System design 8. Design an SFT pipeline: data → split → LoRA SFT → eval-loss monitoring → before/after + OOD eval → deploy.

Startup / product 9. Why does SFT teaching behavior-not-facts shape what you should (and shouldn't) fine-tune for?

14. Takeaways

1. SFT = next-token learning on your (input→ideal) pairs — the model imitates your examples (flaws included), so data quality dominates (06).
2. It teaches behavior, not facts (00); loss is masked to completions.
3. Watch eval loss, use ~3–5 epochs — too many overfits; hold out an eval set (no leakage) (Phase 12.01).
4. Catastrophic forgetting is real — eval out-of-distribution for regressions; LoRA reduces it (02).
5. LoRA SFT is the default; eval before/after vs the base for gains and regressions (Phase 12).

15. Artifact Checklist

• A clean (input→ideal) chat dataset split train/eval (no leakage).
• A LoRA SFT run with an eval-loss curve.
• A before/after comparison vs the base.
• An OOD-regression check (catastrophic forgetting).
• A chosen epoch count justified by eval loss.

Up: Phase 13 Index · Next: 02 — LoRA and QLoRA