Lab 03 — The Error-Analysis Flywheel & Eval-Gated Delivery

Phase 33 · Lab 03 · Phase README · Warmup

The problem

Labs 01 and 02 built the instruments — a versioned suite, graders, pass@k, the regression gate, execution-graded coding tasks. An instrument you never re-point at reality goes stale: the world shifts, users find inputs you never imagined, and your eval score stops predicting production quality. This lab builds the lifecycle that keeps EDD honest — the loop that turns a shipped agent's failures into next week's eval cases, and the release pipeline that refuses to ship a candidate the evals don't bless.

This is the part practitioners (Hamel Husain, Eugene Yan) call the highest-leverage work in applied LLM engineering, and it is almost never in the demo: reading failures, categorizing them into failure modes, and promoting them into permanent regression tests. Plus the two guardrails that stop the flywheel from spinning off course — a regression gate on every release and a held-out set that catches you overfitting your prompts to the dev suite.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example: error analysis -> flywheel -> gated release

Success criteria

• DatasetVersion.content_hash is order-independent and changes when any case's content changes; append_cases dedupes (id-independently) and only bumps the version on genuinely new cases.
• run_eval returns a pass_rate, per-case passed, Failures with signatures, and the ids of failed safety-tagged cases.
• analyze_failures groups by signature and ranks by count (desc), then signature (asc).
• promote_failures_to_evals grows the suite from failed traces only, deduped against the existing set, and is idempotent on the same trace.
• select_smoke returns exactly the tag-matching cases.
• RegressionGate blocks on a metric drop, on a case regression even when the aggregate is unchanged, and always on a safety failure.
• detect_overfitting fires when dev improves but held-out drops beyond the threshold, and not otherwise.
• ReleasePipeline blocks the first release on a safety failure, blocks a regressing or overfitting candidate (leaving the baseline intact), releases an improving one (updating the baseline), and records every attempt in the ledger.
• All 32 tests pass under both lab and solution.

How this maps to the real stack

• The growing versioned dataset is a LangSmith Dataset with versions, a Braintrust dataset, or a Git-tracked JSONL — the Phase 11 "golden set is the moat," now with the explicit rule that it is append-only history, because a score is only meaningful against a named dataset version.
• Error analysis is the practice popularized in Hamel Husain's evals writing and Eugene Yan's applied-LLM essays: sample failures, open-code them into categories, count the categories, and fix the biggest one. Our analyze_failures is the "count the categories" step made mechanical; the signer that assigns a signature stands in for a human (or an LLM) doing the open-coding.
• The flywheel is the production-telemetry loop every serious eval program runs: sample real traces, label the failures, and add them to the eval set so they can never regress silently again. promote_failures_to_evals is that step; the dedupe is what stops your suite filling with near-identical cases.
• The eval-gated release + ledger is the CI/CD integration: cheap smoke evals per commit (promptfoo/assert thresholds in a PR check), the full suite pre-release (a nightly or release-branch job), and a dashboard/ledger tracking scores across versions (LangSmith experiments, Braintrust's score history).
• The held-out set is standard ML hygiene applied to prompt/agent iteration: because you tune against the dev evals, you will eventually overfit them, and only a set you never looked at during iteration tells you the truth. Anthropic's and OpenAI's eval guidance both stress keeping a held-out slice for exactly this reason.

Limits of the miniature. Grading is exact-match to keep the focus on lifecycle — a real pipeline plugs in the Lab 01 grader ladder and the Lab 02 execution grader here. The failure signer is deterministic; in production, open-coding is human/LLM work and the signatures are messy. "Overfitting" is one simple dev-up/held-out-down rule; real detection also watches confidence intervals and multiple held-out slices. The ledger is an in-memory list, not a durable experiment store.

Extensions (your own machine)

• Wire the Lab 01 EvalRunner + grader ladder in as run_eval so promoted cases carry real grader specs, not just exact-match strings.
• Add bootstrap confidence intervals to each release's pass rate and refuse to declare an improvement unless the CI clears zero — the honest answer to "is this delta real?"
• Rotate the held-out set on a schedule (freeze a fresh slice each quarter) so it never leaks into iteration, and detect when the held-out itself has gone stale.
• Persist the ledger to JSON and render a "score across versions" chart — the eval dashboard.

Interview / resume signal

"Built the EDD lifecycle: an append-only versioned golden dataset, an error-analysis step that clusters failures into a ranked failure-mode report, a flywheel that promotes production-trace failures into new deduped eval cases, and an eval-gated release pipeline — per-commit smoke, full-suite-plus-regression-gate pre-release, a held-out set with an overfitting detector that fires when the dev score climbs while held-out drops, and a version ledger recording every release attempt. It's the machinery that turns 'we have evals' into 'evals are how we ship.'"