Golden Datasets

Phase 12 · Document 01 · Evaluation Prev: 00 — Evaluation Overview · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The golden dataset is the single most valuable artifact you build in LLM engineering — and the foundation everything else in this phase stands on (00). It's the encoded definition of "what good looks like for my task," which is exactly what no benchmark gives you and what makes your eval (and your product) defensible — the moat. Every regression gate, model comparison (08), RAG/agent/code eval (03–05), and judge calibration (02) runs against the golden set. A weak golden set (unrepresentative, no edge cases, tiny, or leaked into training) silently invalidates all of it — you'll "pass eval" and fail in production. Building and maintaining a good one is the highest-leverage eval skill.

2. Core Concept

Plain-English primer: a curated, labeled test set for your task

A golden dataset (a.k.a. eval set / golden set) is a curated collection of (input → expected output) or (input → evaluation criteria) examples that represent your actual use case. At eval time you run the system on each input and score the output against the expected output/criteria (00, scored programmatically or by judge 02).

{
  "id": "ex_017",
  "input": {"system": "You are a support agent for Acme.", "user": "How do I reset my password?"},
  "expected": {                                  // for programmatic scoring
    "contains": ["reset", "email"], "must_not_contain": ["I don't know"], "max_tokens": 200
  },
  "criteria": "Gives clear password-reset steps; doesn't invent features; professional tone",  // for judge
  "tags": ["support", "common", "happy-path"]
}

The example carries whatever the scorer needs: a reference answer, an assertion, or a rubric (00).

The four properties of a good golden set

1. Representative — it mirrors real traffic (the actual distribution of inputs your users send), not just easy/obvious cases. Build it from real or realistic examples — production logs are the best source (Phase 7.08).
2. Includes edge cases + known failure modes — the hard, ambiguous, adversarial, and historically-broken inputs. Bugs you've seen should become permanent test cases (regression prevention).
3. Includes negatives / unanswerable cases — inputs where the right answer is "I don't know" / refuse / no-result. Without these you never test refusal and hallucination (Phase 9.08, 07).
4. Right-sized + version-controlled — 50–200 well-chosen examples beat thousands of random ones; version it (git) so eval results are comparable over time.

The cardinal rule: never train on it (no leakage)

A golden set is eval-only. The moment it's used for training/fine-tuning (or leaks into a model's training data), it stops measuring generalization and starts measuring memorization — your scores inflate while real performance doesn't (the contamination problem, Phase 11.08). Keep it separate, private, and out of any training pipeline. This is also why public benchmarks decay — they leak into training corpora.

How to label / define "expected"

The scorer dictates the label format (00):

• Reference answer (exact/fuzzy match) — for deterministic tasks (extraction, classification).
• Assertions (contains/regex/JSON-valid/range) — programmatic, cheap, robust.
• Rubric / criteria (for LLM-judge 02) — for subjective quality (helpfulness, tone, faithfulness).
• Gold relevant chunks (RAG) / success tests (code/agents) — task-specific ground truth (03/05). Prefer the most objective label the task allows (programmatic > rubric). Labels often need human judgment to create — invest there.

Growing it from production (the flywheel)

A golden set is living, not one-and-done. The flywheel (Phase 9.09/10.09):

1. Production surfaces failures (thumbs-down, escalations, incidents — Phase 7.08).
2. You add those failures (with correct labels) to the golden set as new test cases.
3. Future changes are gated against them → the same bug can't recur. This is how the moat compounds: your golden set becomes an ever-better encoding of your task's hard cases.

Stratify and tag

Tag examples by category/difficulty/feature so you can read scores per slice (not just an average that hides a failing subgroup) — e.g., "we're 95% overall but 60% on refusals." Stratification turns one number into actionable diagnosis (08).

Synthetic data (use with care)

You can generate examples (LLM-written queries/variations) to bootstrap coverage — useful for breadth, but validate them (synthetic data can be unrepresentative or wrong) and never let synthetic crowd out real traffic. Real production examples are worth far more than synthetic ones.

3. Mental Model

   GOLDEN SET = curated (input → expected/criteria) for YOUR task → run system → SCORE [00]
        = the encoded definition of "good for my task" = the MOAT (no benchmark gives this)

   FOUR PROPERTIES:
     ① REPRESENTATIVE (mirror real traffic; build from prod logs)
     ② EDGE CASES + past bugs (bugs → permanent regression tests)
     ③ NEGATIVES / UNANSWERABLE (test refusal + hallucination)
     ④ right-sized (50–200 > thousands random) + VERSION-CONTROLLED

   ★ NEVER TRAIN ON IT (eval-only) → else memorization not generalization (contamination) [11.08]
   LABELS by scorer: reference | assertions | rubric [02] | gold chunks/tests [03/05] — prefer OBJECTIVE
   FLYWHEEL: prod failure → add as labeled test → gate forever → moat COMPOUNDS [7.08]
   TAG/STRATIFY → read scores per slice (averages hide failing subgroups)

Mnemonic: a golden set is your task's "what good looks like" — representative + edge + unanswerable, 50–200, versioned, and NEVER trained on. Grow it from production failures (the flywheel); tag it to read per-slice. It's the moat.

4. Hitchhiker's Guide

What to look for first: is it representative of real traffic and does it include edge cases + unanswerable items? A pretty golden set of only happy-path questions gives false confidence.

What to ignore at first: scale. 50 well-chosen, well-labeled examples beat 5,000 random ones — start small and representative, grow from production.

What misleads beginners:

• Happy-path-only sets. You'll "pass eval" and fail on the hard/edge/unanswerable cases that actually matter.
• Training on the eval set (leakage). Inflates scores via memorization — keep it eval-only and private (Phase 11.08).
• Reading only the average. It hides a failing subgroup — tag and read per slice.
• Static set. Quality drifts and new failure modes appear — grow it from production (Phase 7.08).
• Over-relying on synthetic data. Validate it; real traffic is worth more.

How experts reason: they curate a representative set from real logs, deliberately add edge cases + past bugs + unanswerable items, keep it small/high-signal, versioned, and eval-only, tag/stratify for per-slice reading, prefer objective labels, and run a flywheel that turns production failures into permanent test cases. They treat the golden set as a product asset (the moat), not a throwaway.

What matters in production: golden-set representativeness (does it predict prod?), coverage of edge/unanswerable cases, no leakage, per-slice scores, and the feedback loop growing it (00).

How to debug/verify: if eval passes but prod fails → the golden set isn't representative (missing the failing input distribution) — sample prod and add those cases. If scores look inflated/suspicious → check for leakage into training.

Questions to ask: is it built from real traffic? edge + unanswerable cases? size (50–200)? versioned + eval-only (no leakage)? tagged for per-slice reading? growing from production failures?

What silently gets expensive/unreliable: happy-path-only sets (false confidence), leakage (inflated scores), average-only reading (hidden failures), and stale sets (undetected new failure modes).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The golden dataset is the highest-value eval artifact — the encoded definition of "good for your task" and the moat (00).
• Four properties: representative (from real traffic) · edge cases + past bugs · negatives/unanswerable · right-sized (50–200) + versioned.
• Never train on it (eval-only, private) — leakage turns generalization-measurement into memorization-measurement (contamination) (Phase 11.08).
• 50–200 well-chosen examples beat thousands of random ones.
• Label by what the scorer needs (reference / assertions / rubric / gold chunks / success tests); prefer objective labels (02).
• Include unanswerable/negative cases to test refusal + hallucination (Phase 9.08, 07).
• Tag/stratify to read per-slice scores (averages hide failing subgroups).
• Run the flywheel — production failures become permanent golden-set test cases (Phase 7.08); the moat compounds.

9. Observations from Real Systems

• Mature LLM teams treat the golden set as a core asset — versioned, reviewed, and grown from production — because it's what makes iteration safe (00).
• The flywheel (failures → test cases) is universal — RAG/agent/code teams all add production failures back to the eval set (Phase 9.09/10.09/11.08).
• Public benchmark contamination is why "Verified"/private/fresh eval sets exist — the leakage problem in the wild (Phase 11.08).
• The classic false-confidence bug — "passes our eval, fails in prod" — is a non-representative golden set (happy-path only, missing the real input distribution).
• Synthetic test generation (Ragas, LLM-written) bootstraps coverage but teams validate it and prioritize real examples.

10. Common Misconceptions

11. Engineering Decision Framework

BUILD + MAINTAIN A GOLDEN SET:
 1. SOURCE from REAL traffic (prod logs [7.08]); add realistic hand-crafted cases for coverage.
 2. COVER: happy-path + EDGE CASES + past BUGS (→ permanent regression tests) + NEGATIVES/UNANSWERABLE.
 3. SIZE 50–200 high-signal examples; VERSION (git); mark EVAL-ONLY (never train — no leakage [11.08]).
 4. LABEL by scorer: reference/assertions (objective, preferred) | rubric (judge [02]) | gold chunks/tests [03/05].
 5. TAG/STRATIFY by category/difficulty/feature → read per-slice scores [08].
 6. FLYWHEEL: production failure → add labeled test case → gate forever (moat compounds).
 7. (Optional) SYNTHETIC to bootstrap coverage — validate it; don't crowd out real data.

12. Hands-On Lab

Goal

Build a representative, stratified golden set for your task (with edge + unanswerable cases), version it, and prove a non-representative set gives false confidence.

Prerequisites

• A task + (ideally) some real/realistic example inputs; a scorer from 00/02.

Steps

1. Curate from real(istic) inputs: collect 20–30 examples reflecting your actual input distribution (production logs if available, Phase 7.08).
2. Add coverage: deliberately include edge cases, historically-broken inputs, and unanswerable/negative cases (where the right answer is refuse/no-result).
3. Label: give each a reference or assertions (objective) where possible, and a rubric for subjective ones (02); mark gold chunks/success tests if RAG/code (03/05).
4. Tag: add category/difficulty/feature tags; store as versioned JSON in git, clearly eval-only.
5. Stratified scoring: run your system; report scores per tag/slice (not just the average) — find a failing subgroup the average hid.
6. False-confidence demo: build a second set of only happy-path examples; show it scores much higher than the representative set — proving representativeness matters.

Expected output

A versioned, tagged golden set (representative + edge + unanswerable), per-slice scores revealing a weak subgroup, and a happy-path-vs-representative comparison demonstrating false confidence.

Debugging tips

• Eval too easy/high → not representative; add real hard/edge/unanswerable cases.
• Can't compute scores → label format doesn't match the scorer; align them (00).

Extension task

Wire the flywheel: simulate a production failure, add it (labeled) to the golden set, and show a future change is now gated against it.

Production extension

Source the golden set from real production logs (Phase 7.08), enforce eval-only access (no training pipeline reads it), and grow it continuously via the feedback loop; track per-slice trends (08).

What to measure

Per-slice scores, happy-path vs representative gap, coverage of edge/unanswerable, set size, version history.

Deliverables

• A versioned, tagged golden set (representative + edge + unanswerable, eval-only).
• A per-slice score report (a hidden-failing subgroup found).
• A happy-path-vs-representative comparison (false-confidence demo).

13. Verification Questions

Basic

1. What is a golden dataset, and why is it the most valuable eval artifact?
2. What are its four key properties?
3. Why must you never train on it?

Applied 4. Why do 50–200 representative examples beat thousands of random ones? 5. Why include unanswerable/negative cases?

Debugging 6. Eval passes but production fails. What's wrong with the golden set? 7. Scores look suspiciously high after a fine-tune. What might have happened?

System design 8. Design a golden-set program: sourcing, coverage, labeling, versioning, stratification, the flywheel.

Startup / product 9. Why is the golden dataset a moat and a credibility asset (to customers and investors)?

14. Takeaways

1. The golden dataset is your highest-value eval artifact — the encoded "what good looks like" and the moat.
2. Four properties: representative · edge cases + past bugs · unanswerable/negatives · 50–200 + versioned.
3. Never train on it — leakage inflates scores (contamination); keep it eval-only and private.
4. Label by scorer (prefer objective); tag/stratify to read per-slice scores.
5. Grow it from production failures (the flywheel) — the moat compounds; representativeness predicts production.

15. Artifact Checklist

• A versioned, eval-only golden set (representative, from real traffic).
• Edge cases + past bugs + unanswerable/negatives included.
• Labels matched to the scorer (objective where possible).
• Tags for per-slice scoring.
• A flywheel turning production failures into new test cases.

Up: Phase 12 Index · Next: 02 — LLM-as-Judge