Phase 16 — Evaluation, Observability & Guardrails

The phase that turns "it seems to work" into a measurement you can defend, a trace you can debug, and a fuse that blows before the model does damage. Building an LLM feature is the easy 20%. The hard 80% — the part that decides whether you ship, whether you can tell when you've regressed, and whether you sleep at night — is evaluation, observability, and guardrails. This is the round where senior candidates separate themselves: anyone can demo a prompt; very few can prove it's correct, instrument it in production, and contain it when it fails.

Why this phase exists

Every other phase in this track makes the model do something. This phase makes you able to trust it. The difference between an engineer who "ships an LLM feature" and a Senior AI Engineer is that the senior never reports a number without a confidence interval, never trusts an LLM judge without controlling its biases, never ships without a calibration check, and never deploys without a fail-closed guardrail in front. Five things you have to be able to do on a whiteboard:

1. Score open-ended output. There is no single ground truth. You need a zoo of metrics — EM/F1 for closed QA, pass@k for code, BLEU/ROUGE/BERTScore (and their flaws) for generation, and an LLM-as-judge for everything else — and you need to know which to reach for and why each lies.
2. Control the judge. LLM judges are cheap and scalable and biased — toward position, verbosity, and themselves. The senior move is the control: randomized order, rubrics, reference-guided scoring, calibration against humans.
3. Know what the model doesn't know. A model that's 90% accurate but says "0.99" on everything is miscalibrated, and that's dangerous. ECE, reliability diagrams, and temperature scaling measure and fix the gap between confidence and accuracy.
4. Be statistically honest. "Model A scored 0.71" is not a result; "0.71, 95% CI [0.66, 0.76]" is. The bootstrap is how you stop shipping launches on noise.
5. Fail closed. Guardrails — input/output filters, PII redaction, jailbreak/prompt-injection detection, schema validation — composed as a layered, fail-closed defense, because a broken safety check must block traffic, never silently pass it.

Get fluent here and you become the person the team asks "is it actually good?" — and can answer.

Concept map

                    ┌──────────────────────────────────────────┐
                    │  Can I TRUST this model in production?     │
                    └──────────────────────────────────────────┘
                       │              │                 │
            ┌──────────┘       ┌──────┘          ┌──────┘
            ▼                  ▼                 ▼
       EVALUATION         OBSERVABILITY       GUARDRAILS
       "is it good?"      "what happened?"    "stop it going wrong"
            │                  │                 │
   ┌────────┼────────┐    trace prompts/    ┌────┼─────────┐
   ▼        ▼        ▼    responses/tokens/  ▼    ▼         ▼
 metrics  judge   stats   cost/latency     PII  jailbreak schema
 EM/F1   position  CI/    (W&B/MLflow/      filter detect  validate
 pass@k  -bias    bootstrap Langfuse/Phoenix  └────┬────────┘
 BLEU/   control  "never  → Phase 17)            layered,
 ROUGE/    │      one         │                  FAIL-CLOSED
 BERTSc  calibration number"  │                     │
   │      ECE/Brier  │        │                     │
   └────────┴────────┴────────┴─────────────────────┘
                              ▼
            offline eval → online A/B + human eval → ship/rollback

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Defend a launch with statistics: model B beats model A by 3 points on your eval set. Bootstrap both CIs; show the intervals overlap; conclude you cannot ship the claim — then design the larger eval set or A/B test that would settle it.
• Audit your own LLM judge: take a pairwise eval, run every comparison in both orders, report the disagreement (position-bias) rate, and decide whether the judge is usable.
• Catch the over-confident model: two models, same accuracy, very different ECE; explain which you ship for a high-stakes use case and why calibration, not just accuracy, decided it.
• Design the guardrail stack for a customer-facing assistant: which checks run on input vs output, what fails open vs closed, and how you log a violation for audit without logging the PII itself.
• Spot benchmark contamination: a model scores suspiciously high on a public benchmark; explain contamination, why your own held-out eval set matters, and how overfitting to a benchmark misleads.

Where this sits in the track

This phase consumes the whole curriculum and judges it. The pass@k metric grades the code and agents of Phases 12–13; faithfulness eval grades the RAG of Phase 11; calibration ties back to the sampling and logits of Phase 08; and the cost/latency you trace here are the same numbers you sized in Phase 00. The guardrails defend against the prompt-injection threat model that gets sharper the more capable the agent. And the observability you prototype here — what to log, how to define a metric, how to compute a CI — is exactly what Phase 17 industrializes into a continuous platform with dashboards, alerts, and a model registry. In short: every prior phase built a capability; this phase decides whether you can trust it, and the next phase operationalizes that trust.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can hand-compute token-F1 on a partial overlap and derive pass@k from the closed form.
• You can name three LLM-judge biases and the control for each.
• You can compute an ECE by hand for a tiny set and say whether a model is over- or under-confident.
• You can explain why a single eval number is a red flag and what a bootstrap CI adds.
• You can sketch a layered, fail-closed guardrail pipeline and explain why "fail open" causes incidents.
• You can list what to put in an LLM trace and why structured beats unstructured logs.
• You can explain the offline → human → online A/B trust hierarchy and when each is worth its cost.

Key takeaways

• Evaluation is the hardest part of applied LLMs. Outputs are open-ended, there is no single ground truth, and "quality" is a distribution, not a number. The metric zoo each measures a different shadow of quality — and each lies in a known way. Knowing which lie is the skill.
• LLM-as-judge is powerful and biased; the control is the craft. Position, verbosity, and self-preference bias are real and documented. Randomized order, rubrics, reference-guided scoring, and calibration against humans are what make a judge trustworthy — the method without the control is just a confident error.
• Calibration is a first-class quality, separate from accuracy. A model that doesn't know what it doesn't know is dangerous even when it's often right. ECE/Brier measure it; temperature scaling fixes much of it cheaply.
• Never ship on one number. Bootstrap a confidence interval; if it overlaps your baseline's, your "win" is noise. Statistical honesty is the difference between engineering and theater.
• Guardrails fail closed. Layered input/output defense — PII, jailbreak/prompt-injection, schema, toxicity — where a broken or uncertain check blocks. The whole observability + eval + guardrail stack is what makes an LLM safe to operate, which is exactly the platform you industrialize next.

Next: Phase 17 — MLOps & the Model Platform.