Synthetic Data

Phase 13 · Document 05 · Fine-Tuning and Adaptation Prev: 04 — Distillation · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The hardest part of fine-tuning is almost never the training — it's getting enough high-quality labeled data (01). Hand-labeling thousands of examples is slow and expensive, and you often don't have examples of the rare/edge cases you most need. Synthetic data — examples generated by an LLM rather than collected from humans — lets you bootstrap, scale, and balance a fine-tuning dataset at a fraction of the cost. It's how most modern instruction/SFT datasets are built (04), and a core practical skill. But it's a double-edged sword: done carelessly it bakes in model artifacts, bias, and "model collapse." This doc is about generating synthetic data that actually helps — and the validation discipline that keeps it from hurting.

2. Core Concept

Plain-English primer: let an LLM write your training data

Instead of paying humans to write thousands of (input → ideal output) examples (01), you prompt a capable model to generate them. Want 2,000 customer-support Q&A pairs in your tone? Prompt a strong model to produce them. Want hard edge cases your real data lacks? Ask the model to generate them. The output is synthetic training data — cheap, fast, and scalable, and you can steer coverage (generate exactly the cases you need).

human-labeled: people write (input → output) examples  → slow, costly, limited coverage
SYNTHETIC:     an LLM generates (input → output) examples → cheap, fast, steerable coverage  ← but must be VALIDATED

This is deeply tied to distillation (04): when the generator is a stronger model than the one you're training, generating data is hard-label distillation. "Synthetic data" is the broader skill (generation + validation + mixing); distillation is one use of it.

Common generation patterns

• Self-instruct / instruction generation: prompt a model to invent diverse instructions and then answer them — how Alpaca-style instruction datasets were built.
• Input → output completion: you have inputs (real or generated); a strong model produces the ideal outputs (distillation, 04).
• Augmentation / paraphrasing: take real examples and ask the model to rephrase/vary them to expand coverage and robustness.
• Edge-case / hard-negative generation: explicitly generate the rare, tricky, adversarial cases your real data is missing (06) — often the highest-value synthetic data.
• Reasoning traces: generate chain-of-thought then the answer, to teach the process (reasoning distillation, 04).
• Seeded / grounded generation: generate from real documents/data (e.g., "write Q&A about this doc") so outputs are grounded in truth, not invented — reduces hallucination and collapse.

The danger: collapse, artifacts, and bias

Synthetic data is risky because it's only as good as the generator and your validation:

• Model collapse — train on a model's own (or similar models') outputs repeatedly and quality degrades: diversity shrinks, errors compound, the distribution narrows toward the model's quirks. Avoid synthetic-on-synthetic loops without fresh grounding in real data.
• AI artifacts / homogeneity — LLM output has tells (over-formal tone, repeated phrasings, "Certainly!", predictable structure). A student trained on it inherits those patterns and low diversity.
• Inherited bias & errors — the generator's biases and factual mistakes propagate into your dataset (and then your model). The generator is not ground truth (04).
• Hallucinated facts — ungrounded generation invents plausible-but-wrong content; fatal if the task needs accuracy → seed/ground in real data.

The fix: validation + mixing (the non-negotiable discipline)

Synthetic data must be filtered and verified, not used raw:

1. Filter for correctness — drop wrong outputs: check against ground truth, run code/tests (for code), apply rules/schemas, or use an LLM judge (with its caveats, Phase 12.02).
2. Filter for quality & diversity — deduplicate, remove low-quality/templated samples, ensure coverage and variety (not 2,000 near-identical examples) (06).
3. Ground in real data — seed generation with real documents/inputs to keep it truthful and avoid collapse.
4. Mix with human/real data — blend synthetic with some real, human examples; this anchors the distribution and dramatically reduces collapse/artifacts. Pure-synthetic is the riskiest.
5. Keep eval data real and held-out — never eval on synthetic data generated by the same model; your eval set should be real and clean (Phase 12.01), or you'll measure the model agreeing with itself.

GENERATE (steer coverage, ground in real data) → FILTER correctness (ground truth/tests/rules/judge [12.02]) →
FILTER quality+diversity (dedup, variety [06]) → MIX with real/human data → train [01];  EVAL on REAL held-out data [12.01]

The strategic role

Synthetic data is the engine behind bootstrapping when you have no data, scaling cheaply, balancing rare classes/edge cases, and distillation (04). Modern instruction-tuning leans heavily on it. The senior skill is using it aggressively but with validation — and knowing pure-synthetic, ungrounded loops are where it goes wrong.

3. Mental Model

   SYNTHETIC = LLM-GENERATED training data: cheap, fast, STEERABLE coverage (vs slow/costly/limited human labels [01])
   tied to DISTILLATION [04]: stronger generator → generating data IS hard-label distillation

   PATTERNS: self-instruct · input→output completion [04] · augment/paraphrase · EDGE-CASE/hard-negative [06] · reasoning traces [04] · SEEDED/grounded
   DANGERS: model COLLAPSE (synthetic-on-synthetic → diversity↓, errors compound) · AI ARTIFACTS/homogeneity · inherited BIAS/ERRORS · HALLUCINATION

   ★ DISCIPLINE: GENERATE (ground in real data) → FILTER correctness (ground truth/tests/rules/judge [12.02]) →
                 FILTER quality+diversity (dedup/variety [06]) → MIX with real/human data → train [01]
                 EVAL on REAL held-out data [12.01] — NEVER eval on same-model synthetic (it just agrees with itself)

   ROLE: bootstrap (no data) · scale cheap · BALANCE rare/edge cases · distill [04].  Pure-synthetic ungrounded = riskiest

Mnemonic: synthetic data = an LLM writes your training data — cheap, fast, steerable coverage. But it collapses/biases/hallucinates if used raw. Generate (grounded) → filter correctness → filter for diversity → mix with real data → and always eval on real held-out data.

4. Hitchhiker's Guide

What to look for first: what's the data gap — no data (bootstrap), too little (scale), or missing edge cases (balance)? And how will you validate correctness? Synthetic without a validation plan is a trap.

What to ignore at first: exotic generation pipelines. Start with a strong generator + a clear filter (ground truth / tests / rules / judge) and mix in some real data.

What misleads beginners:

• Using synthetic data raw. It must be filtered for correctness and diversity (06) — the generator errs and homogenizes.
• Pure-synthetic, ungrounded loops. Causes model collapse — ground in real data and mix in human examples.
• Evaluating on synthetic data. You measure the model agreeing with itself — eval must be real and held-out (Phase 12.01).
• Ignoring AI artifacts. The student inherits the generator's tells/low diversity — dedup, vary, mix.
• Trusting generated facts. Ungrounded generation hallucinates — seed with real documents for factual tasks.

How experts reason: they use synthetic data aggressively to bootstrap/scale/balance but treat it as untrusted until validated: ground generation in real data, steer for coverage (esp. edge cases/hard negatives, 06), filter for correctness (ground truth/tests/rules/judge, Phase 12.02) and diversity, mix with real/human data to avoid collapse, and keep the eval set real and held-out (Phase 12.01).

What matters in production: the validation yield (how much survives filtering), the diversity/coverage of the kept data, whether mixing/grounding prevented collapse, and that the eval set stays real.

How to debug/verify: model learned artifacts/low diversity → too much templated synthetic, no real-data mix; model hallucinates → ungrounded generation; "great on eval, bad in prod" → eval contaminated with same-model synthetic (Phase 12.01); quality decayed over iterations → collapse from synthetic-on-synthetic.

Questions to ask: what's the data gap? how do I validate correctness? is generation grounded in real data? am I steering for diversity/edge cases? am I mixing in real data? is my eval set real and held-out?

What silently gets expensive/unreliable: raw unfiltered synthetic, ungrounded hallucination, collapse from synthetic loops, AI artifacts/low diversity, and the silent killer — evaluating on synthetic data (inflated metrics that don't transfer).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Synthetic data = LLM-generated training examples — cheap, fast, steerable coverage; the way most modern instruction/SFT sets are built (01/04).
• Generating data with a stronger model is hard-label distillation (04) — synthetic data is the broader skill (generate + validate + mix).
• Patterns: self-instruct, input→output completion, augmentation, edge-case/hard-negative generation, reasoning traces, grounded generation.
• Dangers: model collapse (synthetic-on-synthetic), AI artifacts/low diversity, inherited bias/errors, hallucination — the generator is not ground truth.
• Discipline: generate (grounded) → filter correctness (ground truth/tests/rules/judge Phase 12.02) → filter quality+diversity (dedup) → MIX with real/human data → train.
• Grounding in real data + mixing with human data dramatically reduce collapse/artifacts — pure-synthetic ungrounded is riskiest.
• Never evaluate on same-model synthetic data — the eval set must be real and held-out (Phase 12.01), or you measure the model agreeing with itself.
• Highest-value synthetic data is often the edge cases / hard negatives your real data lacks (06).

9. Observations from Real Systems

• Alpaca/Vicuna/WizardLM were built on synthetic instruction data (self-instruct, Evol-Instruct) — proof synthetic data works with curation.
• Phi models showed high-quality curated synthetic ("textbook-quality") data can train strong small models — quality and curation beat raw scale.
• Model collapse is empirically demonstrated (Nature 2024) — recursive training on generated data degrades quality; grounding + real-data mixing are the antidotes.
• Frontier labs use synthetic data heavily (reasoning traces, preference data, edge cases) — but always with validation and real-data anchoring.
• The classic production mistake is evaluating on synthetic data and seeing inflated metrics that don't transfer — eval stays real (Phase 12.01).

10. Common Misconceptions

11. Engineering Decision Framework

SHOULD YOU USE SYNTHETIC DATA?
  No data (bootstrap) / too little (scale) / missing edge cases (balance)? → yes, with validation.
  Have plenty of clean real data? → prefer it; use synthetic to fill gaps.

GENERATE SAFELY:
 1. GROUND generation in real data/docs where facts matter (anti-hallucination, anti-collapse).
 2. STEER for coverage + DIVERSITY — esp. edge cases / hard negatives [06]; vary style/inputs.
 3. FILTER correctness — ground truth / tests / rules-schema / LLM judge (with caveats [12.02]); drop wrong outputs.
 4. FILTER quality+diversity — dedup, remove templated/low-quality; ensure variety [06].
 5. MIX with real/human data — anchors the distribution, cuts collapse/artifacts. Avoid pure-synthetic loops.
 6. EVAL on REAL, held-out data [12.01] — never same-model synthetic.

12. Hands-On Lab

Goal

Build a validated synthetic dataset for a fine-tuning task, with grounding, filtering, diversity, and a real held-out eval — and feel how raw vs validated synthetic differ.

Prerequisites

• A capable generator model; some real examples/documents to ground and mix; the SFT stack (01); a real, held-out eval set (Phase 12.01).

Steps

1. Generate (grounded): prompt the generator to produce (input → output) examples for your task, seeded with real documents/inputs where facts matter; explicitly include edge cases / hard negatives (06).
2. Filter correctness: drop wrong outputs — check against ground truth, run tests/rules, or use an LLM judge (Phase 12.02). Record the yield.
3. Filter quality + diversity: deduplicate, remove templated/low-quality samples, and verify coverage/variety (06).
4. Mix: blend the validated synthetic with some real/human examples.
5. Train + compare: SFT on (a) raw synthetic vs (b) validated+mixed synthetic (01); eval both on the REAL held-out set (Phase 12.01).
6. Collapse demo (stretch): generate a second synthetic batch from your fine-tuned model and train again; observe diversity/quality changes vs grounding+mixing.

Expected output

A validated, grounded, diverse synthetic dataset that (when mixed with real data) trains a better model than raw synthetic on a real eval — making the validation discipline and collapse risk concrete.

Debugging tips

• Inflated metrics → you evaluated on synthetic; switch to real held-out (Phase 12.01).
• Low diversity/artifacts → dedup, vary prompts, mix more real data.

Extension task

Compare grounded vs ungrounded generation on factual accuracy; quantify diversity (n-gram/embedding) of synthetic vs real.

Production extension

Use synthetic generation to balance rare classes/edge cases (06) and to feed distillation (04); keep a real golden eval set as the gate (Phase 12.08).

What to measure

Filter yield, diversity/coverage, raw-vs-validated quality on real eval, grounded-vs-ungrounded accuracy, (stretch) collapse signs over iterations.

Deliverables

• A validated, grounded, mixed synthetic dataset.
• A raw-vs-validated training comparison on a real eval set.
• A note on diversity, yield, and collapse risk.

13. Verification Questions

Basic

1. What is synthetic data and why use it?
2. What is model collapse and what causes it?
3. Why must you never evaluate on same-model synthetic data?

Applied 4. Name three ways to filter/validate generated data. 5. Why mix synthetic with real/human data?

Debugging 6. Your model learned AI artifacts and low diversity. Causes and fixes. 7. Great eval scores but poor production results. Likely cause?

System design 8. Design a synthetic-data pipeline (generate → ground → filter → mix → train) for a task with few real examples.

Startup / product 9. How would you use synthetic data to bootstrap a fine-tune with no labeled data, while avoiding collapse?

14. Takeaways

1. Synthetic data = LLM-generated training examples — cheap, fast, steerable; how most modern SFT sets are built (01/04).
2. It's untrusted until validated — the generator errs, homogenizes, and hallucinates; filter correctness + diversity (06/Phase 12.02).
3. Ground in real data and mix in human data to avoid model collapse and AI artifacts — pure-synthetic loops are the danger.
4. Never evaluate on same-model synthetic — the eval set stays real and held-out (Phase 12.01).
5. Highest-value synthetic data is the edge cases / hard negatives your real data lacks (06).

15. Artifact Checklist

• A grounded, validated synthetic dataset (with filter yield recorded).
• Diversity/coverage checks (dedup, edge cases).
• A real/human mix blended in.
• A raw-vs-validated training comparison on a real eval set.
• A collapse-risk note (grounding + mixing strategy).

Up: Phase 13 Index · Next: 06 — Dataset Quality