Warmup Guide — Evaluation, Observability & Guardrails

Zero-to-senior primer for Phase 16. We start from the uncomfortable first principle — an LLM's output is open-ended, so there is no single right answer to check against — and build up the entire trust stack: the metric zoo and what each one lies about, the LLM-as-judge method and the biases you must control, calibration (does the model know what it knows?), the statistical rigor that stops you shipping on noise, observability (what to trace and why), and the layered, fail-closed guardrails that contain the model when it goes wrong. The lab turns all of it into a deterministic, offline harness — because a serious eval is infrastructure, not a vibe.

Table of Contents

• Chapter 1: Why Evaluation Is the Hardest Part of Applied LLMs
• Chapter 2: The Metric Zoo — Exact Match, F1, BLEU/ROUGE/BERTScore, pass@k
• Chapter 3: LLM-as-a-Judge — The Method and Its Biases
• Chapter 4: Benchmarks, Contamination & Overfitting
• Chapter 5: Calibration — Does the Model Know What It Knows?
• Chapter 6: Statistical Rigor — Never Ship on One Number
• Chapter 7: Online Eval & Observability — Offline Isn't Enough
• Chapter 8: Guardrails — Layered, Fail-Closed Defense
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Evaluation Is the Hardest Part of Applied LLMs

From zero. In classical ML you have a label. The image is a cat or it isn't; the email is spam or it isn't. Accuracy is correct / total, and that's that. An LLM breaks this in three ways at once:

1. The output is open-ended. "Summarize this article" has infinitely many good answers and infinitely many bad ones. There is no single string to compare against with ==.
2. There is no single ground truth. Even where a reference answer exists, the model can be right in different words ("the capital is Paris" vs "Paris"), or right about something the reference author didn't think of. Surface comparison punishes correct answers.
3. Quality is a distribution, not a number. A model isn't "85% good." It's great on some inputs, catastrophic on a few, and mediocre on the long tail. A single average hides exactly the failures that get you paged.

Why this framing matters. Every technique in this phase is a response to one of those three problems. Normalization and F1 fight problem 1 (surface mismatch). LLM-as-judge fights problem 2 (no single reference). Bootstrap CIs, calibration, and slice-based eval fight problem 3 (it's a distribution). If you remember nothing else: you are never measuring "is it correct"; you are estimating a property of a distribution of open-ended outputs, and every method is a different, flawed estimator.

The senior's habit. When someone says "the new model is better, it scored 87 vs 84," the senior does not nod. They ask: On what eval set? Measured how? With what confidence interval? Is the judge biased? Is the model calibrated? Which slices regressed? Then they have an opinion. The point of this phase is to make those questions reflexive.

The eval flywheel. The thing that actually makes products improve is not a model; it's a loop: collect real failures → add them to an eval set → measure → fix → re-measure → ship → collect new failures. The eval set is the most valuable artifact your team owns — more than any prompt, because it's what tells you whether the next change helped or hurt.

Common misconception. "We'll just eyeball a few outputs." Vibes-based eval works at demo scale and collapses at production scale: you can't eyeball a regression across 10,000 requests, you can't compare two models fairly by gut, and you can't prove to a stakeholder that a change helped. Eyeballing is for generating eval cases, not for deciding.

Chapter 2: The Metric Zoo — Exact Match, F1, BLEU/ROUGE/BERTScore, pass@k

There is no one metric. There is a zoo, each animal good for a different task and lying in a known way. Reaching for the right one — and naming its flaw — is the skill.

Closed-form QA: Exact Match and token-F1

Normalize first. Raw "The Eiffel Tower." and "eiffel tower" are the same answer, but == says no. The SQuAD convention normalizes before scoring: lowercase, strip punctuation, drop the articles a/an/the, collapse whitespace. Skipping this understates accuracy by tens of points — it is the single most common rookie eval bug.

Exact match is 1 if the normalized strings are identical, else 0. Simple, harsh, and right for short factoid answers.

Token-F1 gives partial credit. Treat each answer as a bag of tokens; let shared be the multiset intersection size. Then

$$ \text{precision} = \frac{\text{shared}}{|\text{pred tokens}|}, \quad \text{recall} = \frac{\text{shared}}{|\text{gold tokens}|}, \quad F_1 = \frac{2 \cdot P \cdot R}{P + R}. $$

Worked example: pred "the brown fox" → tokens {brown, fox} (the article drops); gold "a fox" → {fox}. shared = 1, P = 1/2, R = 1/1, so F1 = 2·(1/2)·1 / (1/2 + 1) = 2/3. Memorize this hand-computation; interviewers ask for it.

The flaw. Both are surface-form metrics. "a car" and "an automobile" score 0 despite being synonyms. They reward lexical overlap, not meaning — fine for extractive QA, useless for free-form generation.

Generation: BLEU, ROUGE, BERTScore

• BLEU (translation) — n-gram precision (how many of the candidate's n-grams appear in a reference) with a brevity penalty. Flaw: precision-leaning, ignores meaning, hates valid paraphrase, punishes legitimately short outputs less than it should.
• ROUGE (summarization) — n-gram recall (ROUGE-N) or longest-common-subsequence (ROUGE-L). Flaw: over-credits length and lexical copying; a summary that parrots the source scores well even if it's a bad summary.
• BERTScore — cosine similarity between contextual embeddings of candidate and reference tokens, so paraphrase is rewarded. Better, but it inherits the embedding model's blind spots and still needs a reference.

The deeper truth: n-gram metrics measure lexical overlap, not correctness. They were built for translation/summarization where references are dense, and they correlate poorly with human judgment on open-ended modern LLM outputs. That correlation gap is why LLM-as-judge took over (Chapter 3).

Code and agents: pass@k

For code (and agentic tasks where success is checkable), you don't grade the text — you run it. You sample n completions, c of which pass the unit tests, and ask: if a user could draw k samples, what's the probability at least one passes?

The naive estimate 1 − (1 − c/n)^k is biased for small n (it assumes sampling with replacement from an infinite pool). The Codex paper's unbiased estimator draws without replacement:

$$ \boxed{;\text{pass@}k = 1 - \dfrac{\binom{n-c}{k}}{\binom{n}{k}};} $$

i.e. 1 minus the probability that a random size-k subset of your n samples contains zero correct ones. Two properties to internalize:

• pass@1 = c/n exactly (the empirical success rate).
• Monotone in k: more attempts can only help, so pass@k never decreases as k grows, reaching 1 once you have at least c ≥ 1 correct and k = n.

A numerically stable way to compute it (avoiding huge binomials):

$$ \text{pass@}k = 1 - \prod_{i=0}^{k-1} \frac{(n-c) - i}{n - i}. $$

Common misconception. "pass@10 is just running the model 10 times in production." No — pass@k is an offline capability metric (does the model have it in k tries?), not a deployment strategy. Shipping "sample 10, return the first that passes" is a separate engineering choice with its own cost.

Chapter 3: LLM-as-a-Judge — The Method and Its Biases

The problem it solves. For open-ended outputs there's no reference, and human grading doesn't scale. So you ask a strong model to grade — "LLM-as-a-judge." It's cheap, fast, and correlates surprisingly well with humans (MT-Bench reported ~80%+ agreement, around the level of human–human agreement). It's the workhorse of modern eval.

The two modes.

• Pointwise — "score this answer 1–10 against this rubric." Easy, but absolute scores drift and cluster.
• Pairwise — "which is better, A or B?" More reliable, because relative judgments are easier and more stable than absolute ones. This is what Chatbot Arena and MT-Bench use.

The catch: judges are biased. A judge model is still a model, and it has systematic, documented biases that will silently corrupt your eval if you don't control them:

The position-bias control, mechanically. This is the heart of the lab. To compare a and b:

forward  = judge(a, b)              # verdict in A/B space directly
reverse  = judge(b, a)              # now "A" means b; remap A<->B
if forward == reverse:  trust it    # content drove the verdict
else:                   it's a TIE  # position drove it — refuse to crown a winner, flag it

If a judge always says "A" regardless of content, forward = A but the remapped reverse = B — they disagree, you catch it, you don't ship a coin flip. The disagreement rate across your whole eval set is itself a quality signal: a judge with 30% position-driven flips is not trustworthy.

        a vs b              b vs a (remap)
        ───────             ──────────────
 judge → "A"          judge → "A" → means b → "B"
          │                                    │
          └────────── disagree ───────────────┘
                          ▼
                  winner = "tie", consistent = False   ← position bias caught

The other controls. Rubrics (tell the judge exactly what "good" means, with a scale) cut variance. Reference-guided scoring (give the judge a gold answer to compare against) sharpens hard cases. Calibration against humans (measure judge–human agreement on a labeled slice) is what lets you trust the judge at all — a judge you haven't validated against humans is an unmeasured instrument.

Common misconception. "GPT-4-as-judge is basically ground truth." It's a useful, biased estimator of human preference. Validate its agreement with humans on your task before you believe it, never judge with the same model family you're evaluating (self-preference), and always control position.

Chapter 4: Benchmarks, Contamination & Overfitting

Public benchmarks vs your eval set. MMLU, GSM8K, HumanEval, HELM — public benchmarks are great for comparing models in the abstract. They are terrible as the thing you optimize for your product, for two reasons:

• Contamination. Models train on the internet; the internet contains the benchmarks. A high score may mean "memorized the test set," not "can reason." Suspiciously high numbers, or a big drop on a freshly-written variant of the same task, are the tells. The defense is held-out, private, recent eval data the model could not have seen.
• Distribution mismatch. A benchmark measures quality on its distribution. Your users are a different distribution. A model that's #1 on MMLU can be worse on your support tickets.

Goodhart's law for eval. "When a measure becomes a target, it ceases to be a good measure." If you tune prompts and pick checkpoints by a single benchmark, you overfit the benchmark — you climb the metric while real quality stalls or drops. The defenses: a held-out eval set you don't tune on, multiple metrics and slices (so you can't game one), and periodically refreshing the eval data.

The senior lesson. The benchmark tells you a model is plausible; your own eval set — built from real failures, held out, refreshed — tells you it's right for you. Own your eval set like you own your codebase.

Common misconception. "We picked the top model on the leaderboard, so we're done." Leaderboards rank on someone else's distribution at unbounded cost/latency and are vulnerable to contamination. They're a starting filter, not a decision.

Chapter 5: Calibration — Does the Model Know What It Knows?

The distinction. Accuracy asks "is the answer right?" Calibration asks "when the model says it's 90% sure, is it right 90% of the time?" These are independent. A model can be accurate and over-confident (says 0.99, right 0.7), which is dangerous: downstream systems and humans trust the confidence, and a confident lie is worse than a hedged one.

Reliability diagram. Bucket predictions by confidence and plot, per bucket, empirical accuracy vs average confidence. Perfect calibration is the diagonal y = x. Below the line = over-confident; above = under-confident.

 accuracy
   1 ┤            · perfect (y=x)
     │          ·
     │        ·  ○ under-confident (above the line)
     │      ·
     │    · ●  over-confident (below the line) ← the dangerous one
     │  ·
   0 ┼·──────────────────────► confidence
     0                       1

Expected Calibration Error (ECE). Reduce the diagram to one number: the count-weighted average gap between accuracy and confidence across B bins,

$$ \text{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{N},\bigl|,\text{acc}(B_b) - \text{conf}(B_b),\bigr|. $$

A perfectly calibrated set has ECE = 0 (in each bin, average confidence equals empirical accuracy). An over-confident model — says 0.99 four times, right once (accuracy 0.25) — has one heavy bin with a gap of |0.25 − 0.99| = 0.74, so ECE = 0.74. ECE is sensitive to the bin count, which is why you report B alongside it.

Brier score. A proper scoring rule for probabilistic forecasts — the mean squared error between predicted probability and the 0/1 outcome:

$$ \text{Brier} = \frac{1}{N}\sum_i (p_i - y_i)^2. $$

It's minimized only by reporting your true belief, so it punishes both over- and under-confidence. 0 is perfect; always guessing 0.5 gives 0.25; confidently, completely wrong gives 1.

The fix: temperature scaling. The cheapest, most effective calibration fix (Guo et al., 2017): divide the logits by a single learned scalar T > 1 before softmax to soften over-confident probabilities, fit on a validation set to minimize ECE. One parameter, no retraining, large ECE reduction. (This is a post-hoc calibration of a frozen model — distinct from the sampling temperature in Phase 08, though it's the same arithmetic on the logits.)

Common misconception. "High accuracy means we can trust the confidences." No — accuracy and calibration are orthogonal. Modern deep nets are notoriously over-confident even when accurate. If any downstream decision uses the confidence (routing, abstention, human handoff), you must measure and fix calibration explicitly.

Chapter 6: Statistical Rigor — Never Ship on One Number

The trap. "Model B scored 0.71, model A scored 0.68 — ship B." On a 200-example eval set, a 3-point gap is almost certainly noise. Eval sets are small, examples vary wildly in difficulty, and a point estimate hides all of that. Reporting a single number is the most common way smart teams ship nothing (a non-improvement) and celebrate.

The bootstrap. You want a confidence interval for your metric's mean but you don't know its sampling distribution. The bootstrap manufactures one from the data you have:

1. Resample your n scores with replacement to get a new size-n sample.
2. Compute the metric (the mean) on the resample.
3. Repeat R times (e.g. 1000–10000).
4. The α/2 and 1 − α/2 percentiles of those R means are your (1−α) confidence interval.

 scores [0.7 …]  ──resample w/ replacement──► mean₁
                 ──resample w/ replacement──► mean₂      sort all means,
                 ──resample w/ replacement──► mean₃  ──► take 2.5% & 97.5%
                            ⋮                            percentiles → 95% CI
                 ──resample w/ replacement──► mean_R

Now "0.71" becomes "0.71, 95% CI [0.66, 0.76]." If A's interval [0.63, 0.73] overlaps B's, you cannot claim B is better — the difference is within noise. (For a rigorous comparison you bootstrap the difference A−B and check whether its CI excludes 0; overlapping single-model CIs are a strong but informal signal.)

Determinism is non-negotiable. A seeded random.Random makes the whole procedure reproducible — same seed, same interval — so the result is auditable and CI-friendly. An eval that gives a different number every run is not an instrument.

Offline vs online and human eval (preview of Chapter 7). Offline metrics estimate quality on a fixed set; the real arbiters are online A/B tests (does the user behave better?) and human eval (do experts prefer it?). The honest hierarchy: offline metric → human eval on a sample → online A/B. Each is more expensive and more trustworthy than the last.

Common misconception. "More decimal places = more rigor." 0.7142 is not more trustworthy than 0.71; the interval is what carries the rigor. False precision on a noisy estimate is the opposite of honesty.

Chapter 7: Online Eval & Observability — Offline Isn't Enough

Why offline isn't enough. Your eval set is a snapshot; production is a moving distribution. Users ask things you never imagined, inputs drift, and a prompt that aced offline can fail on real traffic. You need to watch the running system, not just grade it once.

What to trace. For every LLM call, log a structured trace:

• the prompt (template + resolved variables) and the response;
• tokens in/out and cost; latency (TTFT and total);
• the model + version + parameters (temperature, etc.);
• for RAG: the retrieved chunks and their scores (so you can debug why it hallucinated — see Phase 11);
• for agents: the tool calls and intermediate steps;
• any guardrail violations and the eval/judge verdict if you score online.

This is the spine of debugging at 2 a.m.: "why did the model say that?" is unanswerable without the trace, and answerable in minutes with it.

Hallucination & faithfulness eval. A special, high-stakes case. In RAG you can measure faithfulness (is every claim in the answer supported by the retrieved context?) and answer relevance — often with an LLM-judge that checks each claim against the sources. This ties directly back to Phase 11: good retrieval is necessary but not sufficient; you must eval the grounding.

The tools (forward-ref to Phase 17). You don't build the trace store by hand in production:

• W&B / MLflow — experiment tracking, eval runs, model registry.
• Langfuse / Arize Phoenix / LangSmith — LLM-native tracing, prompt/response logging, online eval, cost dashboards.

The mechanics (what to log, how to define a metric, how to compute a CI) are this phase; the platform that runs it continuously, with dashboards and alerts and a registry, is Phase 17. Build the instrument here; industrialize it there.

Common misconception. "We have logs." Unstructured text logs aren't observability. A trace is structured (queryable: filter by model version, slice by user cohort, aggregate cost) and linked (prompt ↔ response ↔ retrieval ↔ verdict). The structure is the whole value.

Chapter 8: Guardrails — Layered, Fail-Closed Defense

The problem. A capable model is also capable of leaking PII, being jailbroken into ignoring its instructions, producing malformed output that crashes downstream parsing, or emitting toxic content. Eval tells you how often; guardrails stop it in real time.

The four classes (input and output).

• PII filter — detect/redact emails, phones, SSNs, etc. Regex is the cheap first layer (run on every request); a NER model is the second. Catches both PII in user input (don't log it) and PII the model emits.
• Jailbreak / prompt-injection detector — flag known attack phrases ("ignore previous instructions," "you are now DAN") and injected instructions hidden in retrieved content. Substring/ classifier layers; this is an arms race, so it's defense-in-depth, never a single perfect filter.
• Schema validator — the model "returns JSON" but returns it wrong often enough that you must validate against a {field: type} schema before any downstream code parses it. Cheap, deterministic, high-value.
• Toxicity / safety classifier — score content for toxicity/harm and block over a threshold (Perspective API, Llama Guard). In the lab this is an injected judge stub so it stays deterministic.

Layered and fail-closed — the two rules that matter.

input ─► [PII] ─► [jailbreak] ─► [schema] ─► [toxicity] ─► allow?
            │          │            │            │
            └──────────┴────────────┴────────────┘
                   collect ALL violations
                            │
          any violation OR any rail crashed ⇒ BLOCK (fail-closed)
                            │
                       audit log

• Layered: run every rail, collect all violations (the audit log must show everything that tripped, not just the first), and allow only if nothing fired.
• Fail-closed: if a rail itself raises — the toxicity classifier is down, the regex throws — that is treated as a violation that blocks, never a pass. "Fail open" (let traffic through when the safety check is broken) is exactly how safety incidents happen. A safety system that silently degrades to "allow everything" when it breaks is worse than no system, because everyone thinks they're protected.

Where guardrails live. Input guardrails run before the model (reject the request, sanitize the prompt); output guardrails run after (redact/block the response). Production frameworks — NeMo Guardrails, Llama Guard, Guardrails AI — give you a declarative way to compose these; the lab builds the mechanism by hand so you understand what they're doing.

Common misconception. "A guardrail is a list of banned words." Real guardrails are a pipeline of heterogeneous checks (regex, classifiers, schema, model-judges) with a clear allow/block policy and an audit trail — and the policy's most important property is which way it fails.

Lab Walkthrough Guidance

The lab (lab-01-eval-harness) turns this material into code. Suggested order (matches the file):

1. String metrics (normalize_answer, exact_match, token_f1) — Chapter 2. The subtleties are the SQuAD normalization (drop articles!) and the bag intersection in F1 (multiplicity matters). Hand-compute ("the brown fox", "a fox") → 2/3 first; the test asserts exactly that.
2. pass_at_k — Chapter 2. Implement the stable product form, handle the n_wrong < k → 1.0 boundary, and validate k ≤ n. test_pass_at_k_matches_closed_form and test_pass_at_k_monotonic_in_k are the soul tests.
3. llm_as_judge / pairwise_judge — Chapter 3. The judge is an injected callable. The position-bias control (run both orders, remap, agree-or-tie) is the bias soul test.
4. Calibration (expected_calibration_error, brier_score) — Chapter 5. Bin by confidence, clamp conf == 1.0 into the last bin, weight by bin count. Brier is a one-line mean of squared errors.
5. bootstrap_ci — Chapter 6. Resample with replacement through the seeded rng, take percentiles of the means. The determinism test (same seed → same interval) is the one that proves it's an instrument.
6. Guardrails (pii_filter, jailbreak_detector, schema_validator, toxicity_stub, run_guardrails) — Chapter 8. The fail-closed behavior — a crashing rail must block — is the test that separates a safe pipeline from an incident.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked output.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can hand-compute token-F1 on a partial overlap and explain why normalization comes first.
• You can write pass@k = 1 − C(n−c,k)/C(n,k) from memory, explain why the naive form is biased, and state the pass@1 = c/n and monotone-in-k properties.
• You can describe the position-bias control in one breath ("run both orders, trust only agreement") and name verbosity and self-preference bias plus their controls.
• You can compute an ECE by hand for a tiny set, say whether the model is over- or under-confident, and name temperature scaling as the fix.
• You can explain why a single eval number is a red flag, what a bootstrap CI adds, and why overlapping intervals kill a launch claim.
• You can draw a layered, fail-closed guardrail pipeline and explain why a crashing rail must block.

Interview Q&A

• "How do you evaluate an open-ended generation task with no reference?" — LLM-as-judge (pairwise preferred), with position-bias control, a rubric, and validated agreement against humans on a labeled slice; report a bootstrap CI, not a point.
• "Why is pass@k computed with the unbiased estimator?" — the naive 1−(1−c/n)^k assumes sampling with replacement and is biased for small n; the unbiased form 1 − C(n−c,k)/C(n,k) draws without replacement, gives pass@1 = c/n, and is monotone in k.
• "Your LLM judge prefers the first answer — how do you fix it?" — run both orderings, remap, and only trust the verdict when they agree; track the disagreement (position-bias) rate as a judge-quality metric. Also name verbosity and self-preference bias.
• "A model is 90% accurate but reports 0.99 on everything. What's wrong and how do you measure it?" — it's miscalibrated (over-confident); measure with ECE / a reliability diagram / Brier; fix cheaply with temperature scaling.
• "Model B beats A by 3 points. Do you ship?" — not yet: bootstrap both CIs; if they overlap, the gap is noise. Bootstrap the difference and check it excludes 0; or run a larger eval / online A/B.
• "What's benchmark contamination and how do you defend against it?" — the test set leaked into training; defend with held-out, private, recently-written eval data and multiple slices; suspect contamination when a public score is high but a fresh variant drops.
• "Design the guardrail layer for a customer-facing assistant." — layered input + output checks (PII, jailbreak/injection, schema, toxicity), collect all violations, fail closed (a broken or uncertain rail blocks), and audit-log violations without logging the PII itself.
• "What do you log for LLM observability and why?" — structured, linked traces: prompt+response, tokens/cost, latency, model+version+params, retrieval (RAG), tool calls (agents), guardrail verdicts — so "why did it say that?" is answerable and cost/quality are sliceable.

References

• Rajpurkar et al., SQuAD: 100,000+ Questions for Machine Comprehension (2016) — the EM/F1 + normalization convention.
• Chen et al., Evaluating Large Language Models Trained on Code ("Codex", 2021) — the unbiased pass@k estimator and HumanEval.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — the method, its biases (position/verbosity/self), and the controls.
• Guo et al., On Calibration of Modern Neural Networks (2017) — ECE, reliability diagrams, and temperature scaling.
• Liang et al., Holistic Evaluation of Language Models ("HELM", 2022) — multi-metric, multi-scenario eval and reported confidence intervals.
• Papineni et al., BLEU (2002); Lin, ROUGE (2004); Zhang et al., BERTScore (2020) — the generation-metric zoo and its flaws.
• Greshake et al., Not What You've Signed Up For: Prompt Injection (2023) — the indirect prompt-injection threat model behind jailbreak guardrails.
• NeMo Guardrails, Llama Guard, and Guardrails AI docs — production guardrail frameworks.
• Langfuse / Arize Phoenix / Weights & Biases / MLflow docs — LLM tracing, eval, and the observability stack (industrialized in Phase 17).