Knowledge 08 — Evaluation, Safety & Observability

Goal of this module. JD4 lists "LLM evaluation metrics and frameworks (RAG evaluation, custom benchmarks)" as good-to-have and "high availability, security" as a responsibility — but in a bank, evaluation and safety are the thing that lets you ship at all. Go from "it looks good in the demo" to a measurable, gated, guarded, observable system: RAG/agent evaluation, the metrics, RAGAS/Foundry evaluators, Content Safety, PII handling, red-teaming, and tracing.

Table of Contents

1. Why "it works in the demo" is a trap

LLM systems are non-deterministic and unbounded in input — the same prompt can vary, and users say anything. A demo with 10 cherry-picked questions tells you nothing about the long tail where it confidently lies, leaks PII, or gets jailbroken. In a bank, the failure mode isn't "bad UX" — it's a regulatory incident or financial loss. So you must measure quality and safety on a representative + adversarial set, gate releases on it, and monitor it in production. Evaluation is the difference between a prototype and a system you can put in front of customers.

The mental model: treat the AI system like any safety-critical software — define correctness, test it, gate on it, monitor it — except correctness is statistical, so you measure distributions, not pass/fail on one input.

2. What to evaluate: the metric taxonomy

Split by what can go wrong:

Retrieval quality (RAG):

  • Context recall — did retrieval fetch the chunks needed to answer? (If not, generation can't succeed.)
  • Context precision — are retrieved chunks relevant, not noise?
  • Recall@k / MRR / nDCG — is the right chunk present and ranked high?

Generation quality:

  • Groundedness / Faithfulness — is every claim supported by the retrieved context? The headline banking metric — an ungrounded claim is a hallucination risk.
  • Answer relevance — does it address the question?
  • Correctness — does it match ground truth (where you have it)?
  • Completeness / coherence / fluency — is it whole and well-formed (incl. in Arabic)?
  • Citation accuracy — do cited sources actually support the claims?

Agent/task quality:

  • Task success rate — did the workflow achieve the goal?
  • Tool-call correctness — right tool, right args?
  • Trajectory / step efficiency — sensible path, no runaway loops?

Safety & compliance:

  • Harmful-content rate, jailbreak/injection resistance, PII leakage rate, out-of-scope refusal correctness, advice-avoidance (no investment/legal/tax advice).

Operational:

  • Latency (p50/p95/p99), cost ($/request, tokens), availability, 429 rate.

A senior candidate names groundedness, context recall/precision, answer relevance, task success, safety rate, and cost/latency as the core dashboard.

3. How to evaluate: LLM-as-judge and its pitfalls

For open-ended outputs there's no exact-match answer, so the dominant method is LLM-as-judge: a strong model scores an output against criteria (and often against the context and/or a reference). E.g. groundedness judge: "Given CONTEXT and ANSWER, is every claim in ANSWER supported by CONTEXT? Score 1–5 and list unsupported claims."

Why it works: scales to thousands of cases, correlates reasonably with human judgment when prompted well, and gives explanations you can audit.

Pitfalls you must know (interviewers probe these):

  • Judge bias — position bias (prefers first answer), verbosity bias (prefers longer), self-preference (prefers its own family's style). Mitigate: randomize order, use rubrics, require evidence/citations in the judgment.
  • Not ground truth — the judge is itself an LLM; calibrate it against human labels on a sample and report agreement. Don't treat judge scores as oracle truth.
  • Cost/latency — judging is extra model calls; sample for online eval.
  • Reference-based vs reference-free — with a gold answer you can score correctness; without, you score groundedness/relevance against the context.

Combine: deterministic checks (format valid, citation present, PII absent, refused-when-should) + LLM-judge (groundedness, relevance) + human review on a sample and on low-confidence cases.

4. RAGAS and Azure AI Foundry evaluators

RAGAS — an open-source framework purpose-built for RAG evaluation. Its signature metrics (mostly LLM-judged, reference-free):

  • Faithfulness — claims in the answer supported by the retrieved context (≈ groundedness).
  • Answer Relevancy — answer addresses the question.
  • Context Precision / Context Recall — retrieval quality. RAGAS lets you score a pipeline on a dataset and track these over changes. Great for CI gating (K07 §8).

Azure AI Foundry evaluators — the platform-native option, integrated with Foundry and Azure Monitor:

  • Built-in evaluators — groundedness (incl. a Groundedness Pro that flags ungrounded claims), relevance, coherence, fluency, similarity, retrieval, and risk/safety evaluators (hateful/violent/sexual/self-harm content, protected material, indirect-attack/jailbreak), plus agent evaluators (intent resolution, tool-call accuracy, task adherence).
  • Custom evaluators — define your own (code or prompt-based) for bank-specific criteria ("does it correctly refuse to give investment advice?").
  • Run modes — offline (on a dataset, in CI / Foundry) and continuous/online (sampled production traffic, surfacing scores to Azure Monitor dashboards/alerts — §6).

The interview line: "RAGAS for portable RAG metrics in CI; Foundry evaluators when I want native integration, safety evaluators, and continuous online evaluation tied to Azure Monitor."

5. Building an eval set and a benchmark

A pipeline is only as trustworthy as its eval set:

  • Golden set — representative (question, ideal answer, expected source) tuples covering real intents, edge cases, out-of-scope (must refuse), and both Arabic and English with native review.
  • Adversarial set — jailbreaks, injection attempts (direct + indirect-in-documents), PII-extraction attempts, "make up a rate" traps.
  • Sourcing — from real (anonymized) logs, SME-authored cases, and synthetic generation (then human-vetted). Keep it versioned in Git; grow it from production failures (every incident becomes a test).
  • Custom benchmarks — beyond generic metrics, define task-specific success (e.g. "statement extraction field accuracy ≥ 99% on IBAN/amount", "dispute-workflow success ≥ X%").
  • Stratify — report metrics per language, per intent, per document type, so a regression hidden in Arabic or in one product doesn't average away.

6. Eval in CI and continuous (online) evaluation

Offline (pre-release): run the eval set in CI on every prompt/model/retrieval/code change; fail the build if a core metric regresses (groundedness, safety, task success) past a threshold. This is the gate that makes shipping safe (K07).

Online (post-release): production inputs drift from your eval set, models update, documents change. So continuously evaluate sampled live traffic (Foundry continuous evaluation) for groundedness/safety/quality, surface to Azure Monitor dashboards, and alert on degradation. Combine with user feedback signals (thumbs, escalation rate, "was this helpful"). Close the loop: production failures → new eval cases → fixes → re-gate.

This offline-gate + online-monitor pairing is exactly the "model risk management" posture a bank's validators expect (K09).

7. The guardrail stack (runtime safety)

Evaluation tells you how good it is; guardrails stop bad things at runtime. Defense-in-depth, in order:

USER INPUT
  → input guardrails:  Content Safety (harm categories) + Prompt Shields (jailbreak/injection)
                       + PII detection/redaction + scope/intent check
  → RETRIEVAL (security-trimmed) — K02 §9
  → LLM / AGENT (grounding prompt, least-privilege tools, human-in-loop) — K03, K06
  → output guardrails: groundedness check + Content Safety on output + PII/secret leak filter
                       + format/schema validation + citation check
  → AUDIT LOG everything
RESPONSE

Each layer catches what the others miss. A bank wants both input and output filtering, and the architectural guardrails (least-privilege tools, approvals) that make a breach non-catastrophic even if a filter is bypassed.

8. PII, content safety, and red-teaming

  • Azure AI Content Safety — 4 harm categories (hate, sexual, violence, self-harm) at severity 0–7, Prompt Shields (jailbreak + indirect injection), Groundedness detection, protected material. Run on input and output; set thresholds by risk (K00 §8).
  • PII handling — detect and redact/tokenize PII (names, account numbers, IBANs, IDs) before it goes to the model and before logging. Tools: Azure AI Language PII detection, Microsoft Presidio (open-source, customizable for banking entities). Decide per use case whether the model even needs the raw PII (often it can work on tokens). Minimize PII exposure — data minimization is a compliance principle.
  • Red-teaming — proactively attack your own system: jailbreaks, injections, PII extraction, harmful-content elicitation, hallucination traps. Microsoft's PyRIT automates LLM red-teaming. Maintain the attack set in CI so fixes don't regress. A bank may require documented adversarial testing before launch.
  • Logging discipline — never log raw PII/secrets; redact first; restrict and audit log access; immutable audit store for decisions (regulators).

9. Observability and tracing

You must be able to reconstruct and explain any interaction (a regulator may ask "why did the assistant tell this customer X?"):

  • Distributed tracing — one trace per request spanning API → agent → each tool call → retrieval → model, with inputs/outputs, token counts, latency, and cost per span. OpenTelemetry → Application Insights / Azure Monitor; Foundry/Agent Service emit traces natively.
  • What to capture — prompt + version, retrieved chunks + scores, tool calls + args + results, model + deployment + params, tokens, latency, cost, guardrail triggers, final answer + citations, user feedback. (Redact PII in stored traces.)
  • Dashboards & alerts — quality (groundedness), safety triggers, latency p95/p99, error/429 rate, cost/token trends, RU usage, escalation/deflection rate.
  • Audit vs telemetry — telemetry is for operating; audit is the immutable, complete record for compliance. Keep both; the audit trail of AI decisions is non-negotiable in banking.

10. Common misconceptions

  • "It works in the demo, ship it." Demos hide the long tail; measure on a representative + adversarial set and gate on it.
  • "Groundedness = accuracy." Groundedness = supported-by-context; the context could be wrong. You also need context recall/correctness.
  • "LLM-as-judge is ground truth." It's a biased estimator — calibrate against human labels, randomize order, require evidence, sample.
  • "Content Safety alone makes it safe." Filters are one layer; you also need grounding, least-privilege tools, approvals, PII redaction, and audit.
  • "Evaluate once before launch." Models/docs/inputs drift — gate offline and monitor online continuously.
  • "Log everything verbatim." Never log raw PII/secrets; redact first; audit access.
  • "English evals cover Arabic." They don't — stratify and review Arabic natively.

11. Interview Q&A

Q: How do you evaluate a banking RAG assistant? Build a versioned golden set (representative + edge + out-of-scope + Arabic/English with native review) and an adversarial set. Score retrieval (context recall/precision, recall@k) and generation (groundedness/faithfulness, answer relevance, correctness, citation accuracy) with RAGAS/Foundry evaluators (LLM-as-judge, calibrated to human labels), plus safety metrics (jailbreak/injection resistance, PII leakage, refusal correctness). Stratify by language/intent. Gate releases in CI on core metrics; monitor sampled production traffic continuously via Foundry → Azure Monitor with alerts; feed failures back into the eval set.

Q: Groundedness is high but users complain answers are wrong — how? Groundedness only checks "supported by retrieved context," not "context is correct/complete." Likely low context recall (retrieval missed the right doc, so the model grounded on the wrong-but-present chunk) or stale/incorrect source docs. Fix retrieval (chunking/hybrid/rerank/query rewrite) and source freshness; add a correctness metric against ground truth and a context-recall metric.

Q: How do you handle PII in the pipeline? Detect and redact/tokenize PII (Azure AI Language PII / Presidio with banking entities) before the model and before logging; pass the model only the PII it actually needs (often tokens suffice); minimize and retention-limit; restrict and audit access; never log raw PII; immutable audit for decisions. Combine with Content Safety and security-trimmed retrieval.

Q: What's LLM-as-judge and what are its risks? Using a strong LLM to score outputs against criteria (e.g. groundedness) at scale. Risks: position/verbosity/self-preference bias, and it's not ground truth. Mitigate with rubrics, evidence-required judgments, order randomization, calibration against human labels with reported agreement, and sampling for cost. Pair with deterministic checks and human review.

Q: What does an AI eval gate in CI actually do? On every prompt/model/retrieval/code change, it runs the eval + adversarial sets and computes groundedness, task success, safety, and format metrics; if any core metric regresses past threshold, the build fails — so you can't ship a quality/safety regression. It's the AI analog of unit tests, plus image scanning and IaC checks.

Q: How do you make an AI decision auditable for a regulator? Trace every request end-to-end (prompt+version, retrieved chunks+scores, tool calls+args+results, model+params, guardrail triggers, answer+citations) to an immutable audit store with PII redacted; retain per policy; restrict access. You can then reconstruct exactly why the assistant said what it said — the core regulatory ask.

12. References

  • RAGAS — Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation" (2023); RAGAS docs.
  • LLM-as-judge — Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023).
  • Azure AI Foundry evaluation — Microsoft Learn: Evaluators (built-in + custom), Groundedness detection, Continuous/online evaluation, Agent evaluators.
  • Azure AI Content Safety — Microsoft Learn (harm categories, Prompt Shields, protected material).
  • PII — Azure AI Language PII detection; Microsoft Presidio docs.
  • Red-teaming — Microsoft PyRIT; OWASP Top 10 for LLM Applications; NIST AI RMF and Generative AI Profile.
  • Observability — OpenTelemetry GenAI semantic conventions; Azure Monitor / Application Insights docs.
  • Related: K02 RAG, K03 Agents, K09 Banking/Responsible AI.