Distillation

Phase 13 · Document 04 · Fine-Tuning and Adaptation Prev: 03 — DPO and Preference Tuning · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

You have a big, expensive, slow model that does a task well — and you want it cheap and fast in production without losing much quality. Distillation transfers a capable "teacher" model's behavior into a smaller "student" model, so you get most of the quality at a fraction of the cost and latency. It's how teams turn an expensive frontier-model prototype into an affordable production model, how the "small but surprisingly good" open models are often made, and a core lever for the cost/latency discipline (Phase 7.09, Phase 12.06). For a fine-tuning module it's essential: distillation is, in its most practical form, just SFT where a strong model generates the training data (01).

2. Core Concept

Plain-English primer: a smart, expensive model teaches a small, cheap one

You have a teacher (a large, capable, costly model — frontier API or big open-weight) that does your task well. You want a student (a small, cheap, fast model) that does almost as well for far less. Distillation = train the student to reproduce the teacher's behavior on your task. The student learns to imitate the teacher; if the task is narrow enough, a small student can match the teacher closely on that task (even while being far worse in general).

TEACHER (big, capable, expensive, slow)  ──teaches──►  STUDENT (small, cheap, fast)
   goal: most of the teacher's task quality at a fraction of cost/latency [7.09/12.06]

Why it works: narrow tasks don't need a giant model

A frontier model is huge because it must do everything. Your production task is usually narrow (classify, extract, summarize in a format, a specific assistant behavior). A small model has plenty of capacity for one narrow behavior — it just needs to be taught it well. The teacher provides exactly that teaching signal at scale. So distillation shines for focused tasks, not for "be a general assistant" (00).

The two flavors of distillation

1. Hard-label / data distillation (the practical, common one): The teacher generates outputs for your inputs; you then SFT the student on (input → teacher output) pairs (01). This is just SFT where the teacher is your labeler/data generator (05). It needs no access to the teacher's internals — works with a closed API teacher. This is what most people mean by "distillation" in practice and what OpenAI/together-style "distillation" features do.

hard-label:  teacher generates outputs for your inputs → SFT student on (input → teacher output) [01]
             (no teacher internals needed → works with a closed-API teacher)  ← the common path

2. Soft-label / logit distillation (the classic ML one): Train the student to match the teacher's full probability distribution over next tokens (the "soft labels"/logits), not just the single chosen token — often with a temperature and a KL-divergence loss. The soft distribution carries more information ("dog 0.7, wolf 0.2, cat 0.01" teaches relationships) than a one-hot label, so the student learns more efficiently. This is the original Hinton-style knowledge distillation, but it requires access to the teacher's logits — so it's mostly for open-weight teachers, not closed APIs.

soft-label:  match teacher's probability DISTRIBUTION (logits) over tokens (KL loss, temperature)
             richer signal than one-hot, but needs teacher LOGITS → open-weight teachers

The practical recipe (what teams actually do)

1. Pick a strong teacher that does your task well (often a frontier API or a large open model).
2. Generate data: run the teacher over your real inputs (or synthetic prompts, 05) to produce high-quality outputs — optionally including reasoning/chain-of-thought so the student learns the process, not just the answer.
3. Filter the teacher's outputs (drop wrong/low-quality ones — ideally check against ground truth or a judge, Phase 12.02) — teacher output is not automatically correct.
4. SFT (often LoRA) the student on the filtered (input → output) data (01/02).
5. Eval the student vs the teacher on a held-out set: how much quality did you keep, and how much cost/latency did you save (Phase 12.06)?

Distillation vs the alternatives

• vs quantization (Phase 6.06): quantization shrinks the same model (fewer bits); distillation makes a different, smaller model learn the behavior. They compose — distill then quantize.
• vs plain SFT: hard-label distillation is SFT — the distinction is the data comes from a teacher model rather than humans (01/05).
• vs RAG/prompting: if you just need facts/context, RAG/prompting may suffice (00); distillation is for baking a capable behavior into a cheaper model.

Caveats

• The student inherits the teacher's mistakes and biases — garbage teacher → garbage student; filter outputs.
• A student can't exceed the teacher on the distilled behavior (it's imitating); it trades generality for task focus.
• Legal/ToS: some providers restrict using their model's outputs to train competing models — check terms before distilling from a closed API.
• Model collapse risk if you distill from synthetic-on-synthetic without grounding (05).

3. Mental Model

   TEACHER (big/capable/expensive) ──teaches──► STUDENT (small/cheap/fast): most quality, fraction of cost/latency [7.09/12.06]
   WHY: production tasks are NARROW; a small model has capacity for one behavior if TAUGHT well [00]

   HARD-LABEL (practical★): teacher GENERATES outputs → SFT student on (input → teacher output) [01]; no internals → closed-API teacher OK
   SOFT-LABEL (classic): match teacher's token DISTRIBUTION/logits (KL, temperature); richer signal but needs LOGITS → open-weight teacher

   RECIPE: pick teacher → generate data (+ optional CoT) [05] → FILTER (teacher ≠ always right; judge/ground truth [12.02]) →
           SFT (often LoRA) student [01/02] → EVAL student vs teacher (quality kept vs cost/latency saved [12.06])

   vs QUANTIZATION (shrink same model [6.06]) — composes: distill THEN quantize  |  hard-label distillation IS SFT with teacher-made data
   CAVEATS: student inherits teacher's errors/bias · can't exceed teacher · ToS on training from API outputs · collapse if synthetic-on-synthetic [05]

Mnemonic: a big teacher teaches a small student → most of the quality at a fraction of the cost/latency. The practical recipe is just SFT on the teacher's (filtered) outputs. Works best on narrow tasks; the student can't beat the teacher and inherits its flaws.

4. Hitchhiker's Guide

What to look for first: is the task narrow enough that a small student can match the teacher? And do you have a teacher that already does it well? Those decide whether distillation is worth it.

What to ignore at first: soft-label/logit distillation. For most production cases, hard-label (teacher generates data → SFT the student) is simpler, works with a closed-API teacher, and is what people mean by "distillation."

What misleads beginners:

• Trusting teacher outputs blindly. The teacher is not always right — filter against ground truth / a judge (Phase 12.02).
• Expecting the student to beat the teacher. It imitates — it trades generality for task focus.
• Distilling a too-broad task. Narrow tasks distill well; "be a general assistant" into a tiny model does not (00).
• Ignoring ToS. Some providers forbid training competing models on their outputs — check terms.
• Confusing it with quantization. Different smaller model vs same model fewer bits — they compose (Phase 6.06).

How experts reason: they use distillation to collapse an expensive prototype into a cheap production model for a narrow task: pick a strong teacher, generate + filter data (often with reasoning traces), SFT (LoRA) the student (01/02), then eval student-vs-teacher on quality kept vs cost/latency saved (Phase 12.06). They compose with quantization for max savings and respect provider ToS.

What matters in production: quality retained vs the teacher, the cost/latency win (the whole point), the filtering quality of the distillation data, and ToS compliance.

How to debug/verify: student worse than expected → low-quality/unfiltered teacher data, too-broad task, or too-small student; "still costs too much" → quantize the student (Phase 6.06) or distill to a smaller one; always eval student vs teacher on a held-out set (Phase 12).

Questions to ask: is the task narrow? do I have a good teacher? hard-label (API teacher) or soft-label (open teacher)? did I filter teacher outputs? how much quality did I keep vs cost/latency saved? does the teacher's ToS allow it?

What silently gets expensive/unreliable: unfiltered teacher errors propagating into the student, distilling too-broad a task, ToS violations, and forgetting to eval the cost/latency win that justified the whole exercise.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Distillation transfers a capable teacher's behavior into a small, cheap, fast student — most of the task quality at a fraction of cost/latency (Phase 7.09/Phase 12.06).
• It works because production tasks are narrow — a small student has capacity for one behavior if taught well (00).
• Hard-label (data) distillation = SFT on the teacher's (filtered) outputs (01/05) — no teacher internals needed → works with a closed-API teacher; the common path.
• Soft-label (logit) distillation matches the teacher's token distribution (KL/temperature) — richer signal but needs the teacher's logits → open-weight teachers.
• Recipe: pick teacher → generate (+optional CoT) → FILTER → SFT (LoRA) student → eval student vs teacher.
• The teacher isn't always right — filter outputs; the student can't exceed the teacher and inherits its biases/errors.
• Distillation ≠ quantization (smaller model vs fewer bits) but they compose — distill then quantize (Phase 6.06).
• Check ToS — some providers forbid training competing models on their outputs.

9. Observations from Real Systems

• Many strong small open models were distilled from larger ones (Alpaca/Vicuna-style data distillation; reasoning distillation à la Orca) — small models punch above their weight on narrow behaviors.
• API providers offer distillation features (e.g., store completions from a strong model, then fine-tune a cheaper one) — productized hard-label distillation (01).
• Reasoning distillation (teach the chain-of-thought, not just answers) produces notably better small students — distill the process (05).
• Distill-then-quantize is a common production stack for max cost/latency savings (Phase 6.06).
• ToS matters — training a competitor on a closed model's outputs has caused real disputes; teams check terms.

10. Common Misconceptions

11. Engineering Decision Framework

SHOULD YOU DISTILL?
  Have an expensive model that does a NARROW task well, and want it cheaper/faster? → distill.
  Need facts/context, not a baked-in behavior? → RAG/prompting [00].   Same model, fewer bits? → quantize [6.06].

DISTILL:
 1. TEACHER: a model that already does the task well (frontier API or big open).
 2. METHOD: hard-label (teacher generates → SFT student [01]) — default, works with API teacher.
            soft-label (match logits) only if you have the teacher's logits (open-weight) and want max efficiency.
 3. DATA: run teacher over real/synthetic inputs [05]; include reasoning/CoT for harder tasks; FILTER bad outputs (judge/ground truth [12.02]).
 4. STUDENT: SFT (often LoRA [02]) the small model on filtered data.
 5. EVAL student vs teacher on held-out: quality KEPT vs cost/latency SAVED [12.06]. Compose with quantization [6.06] for more savings.
 6. CHECK provider ToS on training from outputs.

12. Hands-On Lab

Goal

Distill a strong teacher into a small student on a narrow task, then eval student vs teacher to quantify quality kept vs cost/latency saved.

Prerequisites

• Access to a capable teacher (API or large open model) and a small student to fine-tune; the SFT/LoRA stack (01/02); a held-out eval set (Phase 12).

Steps

1. Pick a narrow task (e.g., extract structured fields, classify, or summarize in a format) where the teacher does well.
2. Generate data: run the teacher over your inputs (real or synthetic, 05) to produce outputs — optionally include reasoning then the answer.
3. Filter: drop wrong/low-quality teacher outputs (check against ground truth or a judge, Phase 12.02) — the teacher isn't always right.
4. Train the student: SFT (LoRA) the small model on the filtered (input → teacher output) pairs (01/02).
5. Eval student vs teacher: on held-out data, measure quality (how much you kept) and cost/latency (the win) — and vs the base student (uplift) (Phase 12.06).
6. Compose (stretch): quantize the student (Phase 6.06) for even more savings; re-eval.

Expected output

A small student that retains most of the teacher's quality on the narrow task at far lower cost/latency — with a student-vs-teacher eval quantifying the trade, and (stretch) a quantized version.

Debugging tips

• Student underperforms → unfiltered/low-quality teacher data, task too broad, or student too small.
• Marginal cost win → quantize the student or pick a smaller student.

Extension task

Try reasoning distillation (include CoT in the teacher data) vs answers-only; compare student quality.

Production extension

Replace an expensive API call in a pipeline with the distilled student behind an eval gate (Phase 12.08) and a fallback to the teacher for low-confidence cases (Phase 7.07); track the cost saved (Phase 7.09).

What to measure

Quality kept vs teacher, cost/latency saved, uplift over base student, filtering yield, (stretch) quantized student quality.

Deliverables

• A filtered teacher-generated dataset for a narrow task.
• A distilled student (SFT/LoRA).
• A student-vs-teacher eval (quality kept vs cost/latency saved) + optional quantized student.

13. Verification Questions

Basic

1. What does distillation transfer, and from what to what?
2. Why does it work best on narrow tasks?
3. How does hard-label distillation differ from soft-label?

Applied 4. Why must you filter the teacher's outputs? 5. How do distillation and quantization differ, and how do they compose?

Debugging 6. The student underperforms the teacher badly. Three likely causes. 7. The cost win is marginal. What can you do?

System design 8. Design a pipeline that replaces an expensive API call with a distilled student plus a teacher fallback.

Startup / product 9. Why is distillation a key cost lever for a product built on an expensive frontier model, and what legal check do you need?

14. Takeaways

1. Distillation makes a small, cheap, fast student reproduce a capable teacher's behavior — most quality at a fraction of cost/latency (Phase 12.06).
2. It works on narrow tasks — a small model has capacity for one behavior if taught well (00).
3. Hard-label (data) distillation = SFT on the teacher's filtered outputs (01/05); soft-label matches logits (open teacher).
4. Filter teacher outputs (it errs); the student can't beat the teacher and inherits its flaws; check ToS.
5. Distill ≠ quantize but they compose — distill then quantize (Phase 6.06); always eval student vs teacher.

15. Artifact Checklist

• A filtered, teacher-generated dataset for a narrow task.
• A distilled student (SFT/LoRA).
• A student-vs-teacher eval (quality kept vs cost/latency saved).
• (Stretch) a quantized distilled student.
• A ToS check note for the teacher model.

Up: Phase 13 Index · Next: 05 — Synthetic Data