Hitchhiker's Guide — Evaluation, Observability & Guardrails

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember when you're shipping."

The 30-second mental model

You can't == an LLM. Output is open-ended, there's no single ground truth, and quality is a distribution. So you build a trust stack: metrics (EM/F1 for closed QA, pass@k for code, an LLM-judge for everything else — each lies in a known way), statistics (bootstrap a CI; never ship on one number), calibration (does it know what it knows? — ECE/Brier), observability (trace every call: prompt/response/tokens/cost/latency), and guardrails (layered, fail-closed PII/jailbreak/schema/toxicity). The judge is biased — control position (run both orders), verbosity, and self-preference. Own your eval set; it's worth more than any prompt.

The numbers and formulas to tattoo on your arm

Back-of-envelope one-liners

# "Is B (0.71) better than A (0.68) on our 200-ex eval set?"
3 points on n=200 → almost certainly noise. Bootstrap both CIs; they overlap → can't ship the claim.

# "Model says 0.99 four times, right once."
acc 0.25, conf 0.99 → ECE ≈ |0.25−0.99| = 0.74 → badly over-confident → temperature-scale it.

# "pass@k for n=10 samples, c=3 correct, k=2?"
1 − C(7,2)/C(10,2) = 1 − 21/45 = 0.533

# "Judge keeps picking the first answer."
forward(a,b)='A', reverse(b,a)='A'→remaps to 'B' → disagree → it's position bias → tie+flag.

The framework one-liners (where these live in real tools)

import evaluate                                   # HF: squad EM/F1, rouge, bertscore, bleu
squad = evaluate.load("squad")                    # EM + token-F1 with SQuAD normalization

from scipy.stats import bootstrap                 # CIs without writing the resampler
bootstrap((scores,), np.mean, confidence_level=0.95, method="percentile")

import torchmetrics                               # CalibrationError(n_bins=15) → ECE

# LLM-as-judge: lm-evaluation-harness, OpenAI/Anthropic evals, or a rubric prompt you own.
# Tracing/online eval: langfuse, arize-phoenix, wandb, mlflow  (→ Phase 17)
# Guardrails: nemoguardrails, llama-guard, guardrails-ai; presidio for PII

War stories

• The 3-point launch that wasn't. Team A/B'd a new prompt, saw 0.71 vs 0.68 on a 150-example eval set, and shipped. Next quarter the "win" had vanished. Bootstrap would have shown overlapping CIs from day one — they shipped noise and burned a quarter chasing the regression that never existed.
• The judge that loved going first. An offline eval declared model X the winner 70% of the time. Someone re-ran every comparison in the other order: X won 70% there too — i.e. the judge just preferred slot A. The "win" was position bias. After controlling order, it was a tie. One re-run saved a wrong model decision.
• The fail-open guardrail. The toxicity classifier started throwing 500s under load. The wrapper caught the exception and... returned "safe." For three hours every request bypassed the safety check silently. Nobody noticed because the dashboard was green. Fail-closed would have blocked (and paged); fail-open made the outage invisible and dangerous.
• The contaminated benchmark. A model scored 92 on a public coding benchmark and 58 on the team's freshly-written held-out set. The 92 was memorization. The held-out number was the real one — and the reason the team kept a private eval set in the first place.
• The schema that "always returns JSON." Until 2% of the time it returned JSON wrapped in markdown fences, or with a trailing comment, and the downstream parser crashed in prod. A two-line schema validator in front would have caught and retried every one.

Vocabulary (rapid-fire)

• EM / token-F1 — closed-QA scoring after SQuAD normalization; F1 is partial credit.
• pass@k — unbiased prob that ≥1 of k samples passes; code/agent metric.
• LLM-as-judge — using a model to grade output; pointwise or pairwise.
• Position / verbosity / self-preference bias — the three judge biases you must control.
• ECE / reliability diagram / Brier — calibration: the gap between confidence and accuracy.
• Temperature scaling — one-scalar post-hoc calibration fix.
• Bootstrap CI — resample-with-replacement confidence interval; the "never one number" tool.
• Contamination / Goodhart — test set leaked into training / optimizing the benchmark not quality.
• Faithfulness — every claim in a RAG answer is supported by the retrieved context.
• Fail-closed — a broken/uncertain guardrail blocks, never passes.
• Trace — structured, linked record of a call: prompt/response/tokens/cost/latency/retrieval/verdict.

Beginner mistakes

• Scoring with == on raw strings — forgetting SQuAD normalization and understating accuracy.
• Using the biased 1−(1−c/n)^k for pass@k.
• Trusting an LLM judge without controlling position (and never validating it against humans).
• Reporting a single eval number with no confidence interval.
• Confusing accuracy with calibration — a high-accuracy model can be dangerously over-confident.
• Optimizing prompts/checkpoints against a public benchmark (overfitting / contamination).
• Building guardrails that fail open when a check errors.
• "We have logs" — unstructured text instead of structured, queryable, linked traces.

The eval set is your real asset

The single highest-leverage thing in this whole phase isn't a metric or a tool — it's the held-out eval set built from real failures. Rules of thumb:

• Start small (50–200 hard, real cases) and grow it from production failures, not from your imagination.
• Keep it private and held-out — never tune prompts or pick checkpoints against the set you report.
• Slice it (by user cohort, input type, difficulty) so an average can't hide a regression in the tail.
• Refresh it; a static eval set goes stale and gets memorized.
• Version it like code. "Which eval set, which version?" should always have an answer.

The reporting checklist (paste into your design review)

[ ] metric named + which flaw it has (surface-form? needs reference? gameable?)
[ ] eval set: held-out, versioned, n = ___, slices reported
[ ] number REPORTED WITH A BOOTSTRAP CI, not a point
[ ] if LLM-judge used: order randomized, rubric attached, human-agreement measured
[ ] calibration checked (ECE) if any decision uses the confidence
[ ] guardrails: layered, fail-closed, violations audit-logged
[ ] online plan: A/B or human eval before the claim is final

The one thing to take away

Before you say a model is "better," produce: the metric (and which lie it tells), a bootstrap CI (not a point), a calibration check, a bias-controlled judge if you used one, and a fail-closed guardrail in front. If you can't, you're guessing; if you can, you're the person the team trusts to say "ship it."