Lab 01 — Guardrail & Eval Harness

Phase: 10 — GenAI in Production | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours

Production GenAI is failure management: hallucinations measured, injection treated as an attack surface, outputs validated before anything downstream trusts them. This lab builds that layer — deterministic and model-free (mock LLM), so the system is what's tested.

What you build

  • Groundedness scoring — sentence-level claim decomposition + token-overlap support; the unsupported-claim rate as the hallucination metric
  • InjectionDetector — pattern detection plus the canary suite: known attack strings whose detection rate is tracked in CI like accuracy
  • PolicyEngine — refusal policy as data (scope, matcher, action, rationale), most-severe-match-wins
  • StructuredOutputGuard — JSON schema validation with a bounded validate-then-repair loop; the aggregated path-specific errors are the repair prompt (Phase 01's collect-don't-raise, weaponized)
  • GuardrailHarness — the release gate: groundedness ≥ threshold AND canary detection ≥ threshold AND zero policy violations

Reference behavior

healthy        groundedness=1.00 unsupported=0.00 attacks_caught=1.00 passed=True
hallucinating  groundedness=0.75 unsupported=0.25 attacks_caught=1.00 passed=False

Run

pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (17 tests)
python solution.py                          # the demo above

Suggested TODO order

  1. sentences / content_tokens / token_overlap_support — pin against the hand-built supported/fabricated cases
  2. groundedness_report — the mixed-answer test computes 2/3 by hand
  3. InjectionDetector.scan + canary_detection_rate — then read the false-positive test: clean questions mentioning "previous quarter" must NOT flag
  4. PolicyEngine.evaluate — scope separation and severity ordering are the two traps
  5. StructuredOutputGuard — the repair test asserts your repair prompt contains the specific errors; that's the design point
  6. GuardrailHarness.evaluate — compose; three gate tests (healthy / hallucinating / policy-violating-but-grounded)

Success criteria

  • All 17 tests pass
  • You can compute an unsupported-claim rate by hand and state the overlap scorer's two failure modes (paraphrase → false alarm; coincidental overlap → false pass)
  • You can explain why the canary suite is data, not code, and what its detection-rate trend tells you
  • You can argue why the policy-violating-but-perfectly-grounded case must still fail the gate

Extensions

  • Swap MockLLM for a real provider; add an NLI-based or LLM-as-judge support scorer; measure agreement with the rule-based scorer on 50 hand-labeled cases
  • Indirect-injection red team: poison a document corpus, measure attack success through full RAG with each defense layer on/off
  • Over-refusal tracking: a benign-prompt suite whose false-refusal rate is gated too
  • Streaming output guards: sentence-buffered groundedness on a token stream

Interview Q&A

Q: How do you measure hallucination in a RAG system? Groundedness: decompose the answer into claims (sentence level), score each for support against the retrieved context (overlap → NLI → LLM-as-judge, in increasing cost and quality), report unsupported-claim rate over a versioned golden set. Trend it per release and gate on it. And split the diagnosis: low context relevance means retrieval is the problem, not generation — most "hallucination" bugs are retrieval bugs.

Q: Your injection detector has 100% canary detection. Are you safe? No — the canary suite only proves you catch known attacks; it's a regression floor, not a security ceiling. Novel attacks bypass pattern detectors by construction. The defenses that hold against unknowns are architectural: least privilege on outputs, allow-listed tools with validated parameters, human confirmation on side effects. Detection is one layer of depth, never the story.

References