When to Fine-Tune
Phase 13 · Document 00 · Fine-Tuning and Adaptation Prev: Phase 12 — Evaluation · Up: Phase 13 Index
Table of Contents
- Why This Matters
- Core Concept
- Mental Model
- Hitchhiker's Guide
- Warmup Readings
- Deep Readings and External References
- Key Terms
- Important Facts
- Observations from Real Systems
- Common Misconceptions
- Engineering Decision Framework
- Hands-On Lab
- Verification Questions
- Takeaways
- Artifact Checklist
1. Why This Matters
Knowing when not to fine-tune is the most valuable judgment in this phase. Fine-tuning is seductive — it feels like the "real ML" answer — but it's the most expensive, slowest-to-iterate, and highest-maintenance option, and it's the wrong tool for the most common problem people reach for it (adding knowledge). The senior call is almost always: try prompting → few-shot → RAG first; fine-tune last, and only for the right reasons. Getting this wrong wastes weeks building a training pipeline when a better prompt or a RAG index would have solved it in a day — and saddles you with a model that goes stale and needs re-tuning on every base-model update. This overview is the decision and the map of Phase 13; the dedicated docs cover the how (SFT [01], LoRA/QLoRA [02], DPO [03], distillation [04]) once you've decided fine-tuning is right.
2. Core Concept
Plain-English primer: fine-tuning changes the weights
Fine-tuning = continuing to train a pre-trained model on your examples, updating its weights (Phase 1.02) so it internalizes a behavior. Contrast the alternatives, which don't touch the weights:
- Prompting / few-shot — steer behavior via the context (what-happens §5). Instant to change, zero training.
- RAG — inject knowledge at query time (Phase 9). Fresh, citable, no retraining.
- Fine-tuning — bake a behavior into the weights. Powerful for consistency, but expensive and static.
The one rule: fine-tuning teaches BEHAVIOR, not FACTS
The single most important principle (Phase 9.00): fine-tuning reliably teaches behavior/format/style/tone — it does not reliably teach facts. A model fine-tuned on Q&A pairs learns to imitate the style of those answers, not to know the facts (it'll confidently confabulate). For knowledge (your docs, changing data), use RAG (Phase 9); for behavior (output format, persona, task-specific patterns), consider fine-tuning. Conflating these is the #1 fine-tuning mistake — people fine-tune to "teach the model our product" and get a confident hallucinator.
The ladder (try in this order)
1. PROMPTING — clearer instructions, better system prompt [instant, free]
2. FEW-SHOT — show 2–5 examples in the prompt [instant, cheap]
3. RAG — inject knowledge/context at query time [Phase 9] [fresh, citable]
4. FINE-TUNING — bake behavior into the weights (this phase) [expensive, static]
Climb only when the rung below demonstrably plateaus (proven by eval, Phase 12). Most problems are solved by rungs 1–3. (Note: fine-tuning and RAG compose — a fine-tuned model can also use RAG; they're not exclusive.)
When fine-tuning IS the right call
- Consistent output format/structure at scale (JSON schema, code style, specific shape) where prompting is unreliable (Phase 10.02).
- Specific style/tone/persona the prompt can't reliably hold.
- A narrow, repeated task where a smaller fine-tuned model matches a larger one's quality at lower cost/latency (the distillation case, 04, Phase 5.09).
- Behavioral patterns few-shot can't fit (too many examples to fit the window).
- Latency/cost: a tuned small/local model replaces an expensive API call (Phase 6).
You need enough high-quality examples (typically hundreds–thousands) and a willingness to own the maintenance.
The costs people underestimate
- Data is 80% of the work — you need a clean, consistent, representative dataset (06); the training is the easy part.
- Iteration is slow — change a prompt in seconds; re-run a fine-tune in hours and re-evaluate (Phase 12).
- Maintenance burden — fine-tunes go stale; every base-model upgrade means re-tuning (07). A prompt/RAG system inherits model improvements for free; a fine-tune doesn't.
- Overfitting / catastrophic forgetting — fine-tuning can narrow the model (great on your style, worse elsewhere) — measure for regressions (Phase 12).
The map of this phase
Once you've decided to fine-tune: SFT (01) is the standard method; LoRA/QLoRA (02) is the efficient way to do it; DPO/preference tuning (03) goes beyond imitation to judgment; distillation (04) trains a small model from a big one; synthetic data (05) and dataset quality (06) are the data backbone; deployment (07) serves and maintains it.
3. Mental Model
alternatives DON'T touch weights; FINE-TUNING does (bakes behavior into weights [1.02])
★ fine-tuning teaches BEHAVIOR/format/style — NOT FACTS (for knowledge → RAG [9.00]); conflating = #1 mistake
THE LADDER (climb only when the rung below PLATEAUS, proven by eval [12]):
PROMPT → FEW-SHOT → RAG [9] → FINE-TUNE (this phase) [free/instant → expensive/static]
(fine-tune + RAG COMPOSE — not exclusive)
FINE-TUNE WHEN: consistent FORMAT/STYLE/persona · narrow repeated task · small model matches big (distill [04]) · latency/cost
COSTS underestimated: DATA is 80% [06] · slow iteration · MAINTENANCE (stale → re-tune on base updates [07]) · overfitting/forgetting
MAP: SFT [01] (how) · LoRA/QLoRA [02] (efficient) · DPO [03] (judgment) · distillation [04] · synthetic [05] + quality [06] (data) · deploy [07]
Mnemonic: fine-tuning bakes behavior (not facts) into the weights — try prompt → few-shot → RAG first, fine-tune last. It's expensive, slow to iterate, and goes stale (re-tune on base updates). Data is 80% of the work.
4. Hitchhiker's Guide
What to look for first: is this a behavior problem (format/style/task pattern — fine-tuning candidate) or a knowledge problem (use RAG)? And have prompting/few-shot/RAG demonstrably plateaued (by eval)? Those decide whether to fine-tune at all.
What to ignore at first: the training mechanics. The decision (and the data) come first; SFT/LoRA (01/02) is the easy part once you've decided.
What misleads beginners:
- Fine-tuning to add knowledge. Models don't reliably learn facts from fine-tuning — they confabulate; use RAG (Phase 9).
- Skipping the ladder. Reaching for fine-tuning before exhausting prompting/few-shot/RAG — usually a cheaper rung solves it.
- Underestimating data + maintenance. Data is 80% of the work (06); and the fine-tune goes stale on the next base-model release (07).
- No before/after eval. You can't know fine-tuning helped (or caused regressions) without an eval gate (Phase 12).
- Expecting a missing capability. Fine-tuning sharpens existing capability; it won't add one the base model fundamentally lacks.
How experts reason: they climb the ladder (prompt → few-shot → RAG → fine-tune), only fine-tuning when a lower rung provably plateaus and the need is behavioral (format/style/narrow task/cost-latency); they invest in data quality (06), always eval before/after for gains and regressions (Phase 12), prefer LoRA/QLoRA for efficiency (02), and plan for maintenance (07). They use RAG for knowledge and fine-tuning for behavior — often together.
What matters in production: whether fine-tuning beat the prompt/RAG baseline (eval), no regressions on out-of-distribution inputs, the re-tuning cadence as base models evolve, and serving cost/latency vs the API alternative (07).
How to debug/verify: "fine-tuned model hallucinates facts" → you tried to teach knowledge; move it to RAG. "Great on eval, bad in prod" → overfit to training style / unrepresentative data (06/Phase 12.01). "No improvement" → the prompt/RAG baseline was already enough, or data quality is low.
Questions to ask: behavior or knowledge problem? have prompting/few-shot/RAG plateaued (proven)? do I have hundreds+ of high-quality examples? will I own re-tuning on base updates? did I eval before/after for gains and regressions?
What silently gets expensive/unreliable: fine-tuning for knowledge (hallucination), skipping the ladder (wasted weeks), poor data (no gain/overfit), stale fine-tunes (missed base-model improvements), and no before/after eval (unknown effect).
5. Warmup Readings
| Title | Why to read it | What to extract | Difficulty | Time |
|---|---|---|---|---|
| Phase 9.00 — RAG Overview | RAG vs fine-tune | knowledge → RAG | Beginner | 15 min |
| Phase 1.02 — Parameters & Weights | What fine-tuning changes | weights | Beginner | 15 min |
| Phase 12.00 — Evaluation Overview | Prove the plateau / before-after | regression gate | Beginner | 15 min |
| Phase 5.09 — Cost-Quality-Latency | The distillation case | small matches big | Beginner | 20 min |
6. Deep Readings and External References
| Title | URL | Why it matters | Read first | Lab connection |
|---|---|---|---|---|
| OpenAI fine-tuning guide | https://platform.openai.com/docs/guides/fine-tuning | When/how (API) | when to fine-tune | Decision |
| Anthropic — fine-tuning vs prompting | https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering | Prompt-first ethos | the ladder | Decision |
| "RAG vs fine-tuning" (survey) | https://arxiv.org/abs/2401.08406 | Knowledge vs behavior | the distinction | RAG-vs-FT |
| Unsloth docs | https://docs.unsloth.ai/ | Efficient OSS fine-tuning | LoRA/QLoRA | 02 |
| Phase 12 — Evaluation | (curriculum) | Eval before/after | golden set | Gate |
7. Key Terms
| Term | Simple meaning | Technical meaning | Why it matters | Where it appears | How to use it |
|---|---|---|---|---|---|
| Fine-tuning | Train on your data | Update weights on examples | Bakes in behavior | this phase | Last resort |
| Behavior vs facts | Style vs knowledge | FT teaches behavior, not facts | The cardinal rule | decision | Facts → RAG [9] |
| The ladder | Try-in-order | prompt→few-shot→RAG→FT | Avoid over-fine-tuning | decision | Climb on plateau |
| SFT | Supervised FT | Train on input→ideal pairs | The standard method | [01] | How-to |
| LoRA/QLoRA | Efficient FT | Adapters on (quantized) base | Production default | [02] | Efficient |
| Maintenance burden | Re-tune over time | Stale on base updates | Hidden cost | [07] | Plan for it |
| Catastrophic forgetting | Narrowing | FT degrades other abilities | Regression risk | eval [12] | Test OOD |
| Distillation | Small from big | Student learns teacher outputs | Cost/latency | [04] | Narrow task |
8. Important Facts
- Fine-tuning teaches behavior/format/style — not facts (Phase 9.00); for knowledge use RAG (Phase 9). Conflating them is the #1 mistake.
- Climb the ladder: prompting → few-shot → RAG → fine-tuning — only fine-tune when lower rungs provably plateau (by eval, Phase 12).
- Fine-tuning and RAG compose — a fine-tuned model can still use RAG; they're not exclusive.
- Fine-tune for: consistent format/style/persona, a narrow repeated task, small-matches-big (distillation [04]), or latency/cost — with enough high-quality examples (hundreds–thousands).
- Data is ~80% of the work (06); the training is the easy part.
- It's the highest-maintenance option — fine-tunes go stale; every base-model upgrade means re-tuning (07).
- Always eval before/after for gains and regressions (overfitting/catastrophic forgetting) (Phase 12).
- Fine-tuning sharpens existing capability — it won't reliably add one the base model fundamentally lacks.
9. Observations from Real Systems
- The field's consensus is "prompt/RAG first" — Anthropic/OpenAI guidance pushes prompting + RAG before fine-tuning; most production problems never need it.
- The classic failure is fine-tuning to "teach our product/docs" → confident hallucination → the team switches to RAG (Phase 9).
- Where fine-tuning wins in production: consistent structured output at scale (Phase 10.02), a specific brand voice, and distilling a cheap specialized model for a narrow high-volume task (04, Phase 5.09).
- LoRA/QLoRA made fine-tuning accessible — efficient enough to run on consumer GPUs (Unsloth) (02, Phase 6).
- The maintenance tax is real — teams with many fine-tunes slow down because each base-model upgrade triggers re-tuning + re-eval (07).
10. Common Misconceptions
| Misconception | Reality |
|---|---|
| "Fine-tune to add our knowledge" | FT teaches behavior, not facts — use RAG [9] |
| "Fine-tuning is the serious/real answer" | It's the last rung; prompt/few-shot/RAG usually suffice |
| "Fine-tuning adds new capabilities" | It sharpens existing ability, doesn't add missing ones |
| "Training is the hard part" | Data is ~80% of the work [06] |
| "Fine-tune once and done" | It goes stale; re-tune on base updates [07] |
| "FT and RAG are either/or" | They compose — use both |
11. Engineering Decision Framework
SHOULD I FINE-TUNE?
0. Is the problem KNOWLEDGE (facts/changing data)? → RAG, not fine-tuning [9]. Behavior/format/style? → continue.
1. CLIMB THE LADDER (prove each plateaus by eval [12]):
prompting → few-shot → RAG → (only now) fine-tuning.
2. WORTH IT? need consistent format/style/persona OR a narrow repeated task OR small-matches-big (distill [04]) OR latency/cost?
AND ≥ hundreds of high-quality examples [06]? AND willing to OWN re-tuning [07]?
3. METHOD: SFT [01] via LoRA/QLoRA [02] (efficient); DPO [03] for judgment/preference; distillation [04] for cost.
4. DATA: invest in quality [06]; synthetic to scale carefully [05]; hold out an eval set (no leakage) [12.01].
5. EVAL before/after for GAINS and REGRESSIONS (OOD) [12]; deploy + plan maintenance [07].
| Problem | Solution (not fine-tuning) |
|---|---|
| Needs fresh/private facts | RAG [9] |
| One-off behavior tweak | Better prompt / few-shot |
| Missing capability | Different/bigger model [5] |
| Consistent format at scale | Fine-tune (SFT [01]) — after prompt plateaus |
| Cheap specialized model | Distillation [04] |
12. Hands-On Lab
Goal
Make the fine-tune-or-not decision rigorously: prove (via eval) whether a prompt/RAG baseline already solves your task before considering fine-tuning.
Prerequisites
- A task + a golden set (Phase 12.01);
pip install openai.
Steps
- Classify the problem: is it behavior (format/style/task pattern) or knowledge (facts)? If knowledge → stop, use RAG (Phase 9).
- Baseline 1 — prompting: write the best system prompt you can; eval on the golden set (Phase 12).
- Baseline 2 — few-shot: add 3–5 in-context examples; re-eval. Did it close the gap?
- Baseline 3 — RAG (if knowledge-adjacent): add retrieval (Phase 9); re-eval.
- Decide: if a baseline hits your quality bar, fine-tuning isn't justified — document that. If all plateau below the bar on a behavioral task with enough data, fine-tuning is a candidate → proceed to 01/02.
- (If proceeding) cost/maintenance estimate: estimate data-collection effort, training cost, and the re-tuning cadence vs the prompt/RAG alternative (07).
Expected output
An eval-backed decision: a table of prompt/few-shot/RAG scores vs your bar, and a documented fine-tune / don't-fine-tune call with justification — the disciplined version of the most important judgment in this phase.
Debugging tips
- Tempted to fine-tune for facts → re-read Phase 9.00; it'll hallucinate.
- "Prompting plateaued" but you didn't really try few-shot/RAG → exhaust the cheaper rungs first.
Extension task
If fine-tuning is justified, build a small dataset (06) and run a LoRA/QLoRA fine-tune (02); eval before/after for gains and regressions (Phase 12).
Production extension
Wire the before/after eval into a gate (Phase 12.08); track the re-tuning cadence and serving cost vs the API baseline (07).
What to measure
Prompt vs few-shot vs RAG scores vs the quality bar; whether a baseline suffices; (if fine-tuning) estimated data/training cost + maintenance cadence.
Deliverables
- A behavior-vs-knowledge classification.
- A ladder eval (prompt/few-shot/RAG vs the bar).
- A documented fine-tune / don't decision with cost/maintenance estimate.
13. Verification Questions
Basic
- What does fine-tuning change that prompting/RAG don't?
- What does fine-tuning teach reliably — and what does it not?
- What's the recommended order to try (the ladder)?
Applied 4. Give two problems you'd solve with RAG instead of fine-tuning, and two where fine-tuning fits. 5. Why is data ~80% of the fine-tuning effort?
Debugging 6. A fine-tuned model confidently states wrong facts. What went wrong, and the fix? 7. A fine-tune scores well on eval but fails in production. Two likely causes.
System design 8. Design the decision process (with eval) for whether to fine-tune a customer-support assistant.
Startup / product 9. Why is the maintenance burden of fine-tuning a strategic cost, and when is it worth paying?
14. Takeaways
- Fine-tuning teaches behavior, not facts — for knowledge use RAG (Phase 9); conflating them is the #1 mistake.
- Climb the ladder: prompting → few-shot → RAG → fine-tuning — fine-tune only when lower rungs provably plateau (eval, Phase 12).
- Fine-tune for behavior/format/style, a narrow repeated task, distillation, or cost/latency — with enough high-quality data.
- Data is ~80% of the work (06); always eval before/after for gains and regressions.
- It's the highest-maintenance option — fine-tunes go stale; plan for re-tuning (07); FT + RAG compose.
15. Artifact Checklist
- A behavior-vs-knowledge classification of the problem.
- A ladder eval (prompt/few-shot/RAG vs the bar) proving (or not) a plateau.
- A documented fine-tune / don't decision with justification.
- A cost + maintenance estimate vs the prompt/RAG baseline.
- (If proceeding) a before/after eval gate plan (Phase 12).
Up: Phase 13 Index · Next: 01 — Supervised Fine-Tuning (SFT)