Lab 01 — LLM Evaluation Harness + Calibration + Guardrails

Phase: 16 — Evaluation, Observability & Guardrails Difficulty: ⭐⭐⭐☆☆ (the metrics are arithmetic; the statistical honesty and bias control are ⭐⭐⭐⭐⭐) Time: 3–4 hours

The hardest part of shipping an LLM is not building it — it is proving it works, knowing when it doesn't, and stopping it when it goes wrong. This lab builds the instrument panel: closed-QA string metrics (exact match, token-F1, SQuAD normalization), the unbiased pass@k estimator for code/agents, an LLM-as-judge with the position-bias control that makes pairwise verdicts trustworthy, calibration (ECE, Brier), the bootstrap confidence interval that ends "we shipped on one number," and a layered, fail-closed guardrail pipeline (PII, jailbreak, schema, toxicity). Every "model call" is an injected deterministic judge stub — because a serious eval is reproducible infrastructure, not a vibe.

What you build

• normalize_answer / exact_match / token_f1 — closed-QA scoring. Normalize (lowercase, strip punctuation/articles, SQuAD style) so "The Eiffel Tower." matches "eiffel tower", then score exactly (EM) or with partial credit over a token-bag intersection (F1).
• pass_at_k — the unbiased estimator 1 - C(n−c, k)/C(n, k), not the biased 1 − (1 − c/n)^k. The metric for code and agents, where you sample many times and ask "did at least one of k pass?"
• llm_as_judge / pairwise_judge — scoring with a model. Pointwise against a rubric, and pairwise with position-bias control: run both orderings, agree → trust it, disagree → the verdict was position, not content, so return a flagged tie.
• expected_calibration_error / brier_score — does the model know what it knows? ECE is the gap between confidence and accuracy; Brier is a proper scoring rule for probabilistic forecasts.
• bootstrap_ci — a deterministic percentile confidence interval for any metric's mean. The cure for "0.71 beats 0.68" when both intervals overlap.
• Guardrail + run_guardrails — pii_filter, jailbreak_detector, schema_validator, toxicity_stub composed into a layered, fail-closed pipeline: collect every violation, and a crashing rail blocks traffic rather than letting it through.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why normalize_answer exists and what it strips, and hand-compute the F1 of ("the brown fox", "a fox") = 2/3.
• You can derive pass@k = 1 − C(n−c,k)/C(n,k) and explain why test_pass_at_k_monotonic_in_k (it never decreases as k grows) and test_pass_at_1_equals_c_over_n are the soul of the metric.
• You can explain why test_pairwise_position_bias_is_detected_and_controlled is the whole point of the pairwise judge — and name two other judge biases.
• You can explain why ECE is 0 for a perfectly-calibrated set and large for an over-confident one, and why Brier punishes confident wrongness.
• You can explain why bootstrap_ci is deterministic for a seed and why an overlapping CI kills a launch claim.
• You can explain why test_run_guardrails_fails_closed_on_crashing_rail is the difference between a safe system and an incident.

How this maps to the real stack

Limits of the miniature (be honest in the interview): the string metrics are surface-form only — they miss paraphrase (that is what BERTScore / a judge are for); the judge is a deterministic stub, so it shows the mechanism and the bias control, not the messy variance of a real model judge; the PII/jailbreak detectors are regex/substring first-layers that real systems back with NER and classifier models; and the bootstrap assumes i.i.d. examples (clustered/contaminated data breaks it). The harness teaches the contracts and the statistics; production swaps the stubs for models.

Extensions (build these on real hardware / data)

• Add ROUGE-L (longest common subsequence) and discuss why it over-credits length, then add a length penalty.
• Swap the deterministic judge for a real model call behind the same judge(pred, rubric) interface and measure the judge–human agreement rate on a labeled set.
• Add temperature scaling: fit a single scalar T that minimizes ECE on a validation split, and plot the reliability diagram before/after.
• Add a two-sample bootstrap (bootstrap_diff_ci) for the difference between two models and show how to read "is the gap significant?"
• Add a paged trace logger that records (prompt, response, tokens, latency, cost, verdict) per call — the seed of the observability stack you build in Phase 17.

Interview / resume

• Talking points: "Why is pass@k computed with the unbiased estimator, not 1−(1−c/n)^k?" "How do you stop an LLM judge from just preferring the first answer?" "Your model is 90% accurate but says 0.99 on everything — what's wrong and how do you measure it?" "Why is reporting a single eval number a red flag?" "Design the guardrail layer for a customer-facing assistant."
• Resume bullet: Built an offline, deterministic LLM evaluation harness — exact-match/token-F1 scoring, the unbiased pass@k estimator, a position-bias-controlled LLM-as-judge, calibration (ECE/Brier), bootstrap confidence intervals, and a layered fail-closed guardrail pipeline (PII, jailbreak, schema, toxicity) — turning model quality from a vibe into a defensible, reproducible measurement.