« RAG at Scale · System Design Index · Agentic AI Engineer · Cheat Sheet

System Design — Agent Evaluation & Safety Platform

The archetype: Docker ("LLM-as-judge, behavioral regression testing, golden datasets"), Juniper Square ("evaluation frameworks, guardrails, feedback systems"), Wolters Kluwer ("AI-agent evaluation"), and Anthropic ("agent prompts & evals"). The question: "Design a platform that lets product teams evaluate and safely ship agent changes." This is the layer that turns "it worked in the demo" into "we can change this system without breaking it."

1. Requirements & scale

Functional. Teams register agents + golden datasets; every proposed change (prompt, model, tool, retrieval) is scored offline against those datasets; a regression gate blocks a merge that regresses quality or trips a safety check; production traffic is sampled for online evaluation and human feedback; red-team suites probe for injection/exfiltration.

Non-functional. Deterministic and reproducible scoring (same input → same score); fast enough to gate CI (minutes, not hours); trustworthy judges (validated against humans); auditable (every score traceable to a dataset version and a run).

Scale signals to state. N teams × M agents × golden sets of 100s–1000s of cases; evals run on every PR and nightly; production sampling at, say, 1–5% of traffic; red-team suites run on a schedule and before major releases.

Staff vs junior: a junior evaluates on public benchmarks; a Staff engineer says "evaluate YOUR task, not benchmarks" — a curated, versioned golden set of your real inputs is the moat, and the platform's job is to make that cheap to build and trustworthy to run.

2. Architecture

                         ┌──────────────────────────────────────────────┐
   PR / change  ──────►  │  EVAL ORCHESTRATOR                            │
                         │   pick datasets → run agent → score → gate    │
                         └───┬───────────────┬──────────────┬───────────┘
                             │               │              │
                    ┌────────▼──────┐ ┌──────▼──────┐ ┌─────▼─────────┐
                    │ GOLDEN DATA   │ │  SCORERS    │ │  SAFETY /     │
                    │ (versioned)   │ │ programmatic│ │  RED-TEAM     │
                    │ inputs+labels │ │ + LLM-judge │ │  suites       │
                    │ + adversarial │ │ + trajectory│ │  (injection)  │
                    └───────────────┘ └─────┬───────┘ └─────┬─────────┘
                                            │               │
                                     ┌──────▼───────────────▼──────┐
                                     │  REGRESSION + SAFETY GATE    │  ──► pass/block PR
                                     └──────────────────────────────┘
   prod traffic ──(sample 1-5%)──►  ONLINE EVAL + human feedback  ──► new golden cases (flywheel)

Every score ties to a dataset version and a run id (reproducibility), and everything is traced (Phase 14) so a failing case is debuggable.

3. The evaluation stack (the scoring ladder)

Score with the cheapest reliable method first (Phase 11):

  1. Programmatic checks (exact/contains/numeric/schema/regex, "did it call the right tool") — deterministic, free, no judge bias. Use wherever the correct answer is checkable.
  2. Trajectory evaluation — grade the agent's path (tools chosen, order, efficiency), not just the final answer. An agent that got the right answer by a lucky wrong route is a latent bug.
  3. LLM-as-judge — for open-ended quality (helpfulness, groundedness) where no programmatic check exists. But a judge is a model that can be biased (position, verbosity, self-preference), so the platform requires a human-agreement check (Cohen's κ) before a judge is trusted in a gate — κ below ~0.6 means the judge isn't reliable and you don't gate on it.
  4. Human review — the ground truth for the hardest cases and for calibrating the judge; the most expensive, so used sparingly and to seed the flywheel.

Staff vs junior: a junior wires an LLM judge and trusts its number; a Staff engineer treats the judge as an unvalidated instrument and calibrates it against humans first, then monitors the agreement over time.

4. Safety as a gate, not an axis

Quality and safety are combined wrong if you average them: a 9/10-helpful answer that leaks a secret is not a 7/10. Safety is a hard gate — any safety-labeled failure (injection succeeded, PII leaked, a disallowed action, an exfiltration) blocks the change regardless of the quality score. The safety suite runs the red-team attacks from Phase 10 (direct/indirect/memory injection, exfil beacons) against the candidate and asserts the guardrails hold. This is the Docker/OpenAI- Security expectation: evals prove capability; safety gates prove containment, and they're scored differently.

5. Behavioral regression & the golden-set flywheel

The platform's core value is behavioral regression testing: re-run the golden set on every change and block regressions. The golden set is a living asset:

  • Seeded from real product inputs, hand-labeled edge cases, and known-hard examples.
  • Grown by a flywheel: production failures and human-flagged cases (from online eval and feedback — Juniper's "feedback systems") become new golden cases, so the platform gets better at catching the next regression.
  • Versioned: every case and label is versioned, so a score is reproducible and a change to the eval itself is reviewable. Never leak golden cases into training/few-shot (contamination).

Staff vs junior: the junior's eval is a one-off script; the Staff engineer's is a versioned golden set wired into CI as a gate, with a flywheel that grows it from production — that's the defensible moat, and it's what lets the team ship fast safely.

6. Reliability, cost, latency

  • Reliability of the eval itself: deterministic scoring (inject the model/clock — the whole track's discipline), reproducible runs, versioned datasets. A flaky eval is worse than none.
  • Cost: judge calls cost money; use programmatic scoring wherever possible, sample the LLM-judge, and cache scores for unchanged (case, candidate) pairs. Report the eval's own $/run.
  • Latency: CI gates must finish in minutes — parallelize case scoring, cache, and run the expensive judge/red-team suites nightly or pre-release rather than on every commit.

7. Observability

Every eval run is traced (Phase 14): per-case scores, the judge's rationale, the trajectory, the dataset version, the candidate version. A dashboard shows quality/safety over time per agent, the judge↔human agreement (is the judge drifting?), and the flywheel's growth. When a gate blocks a PR, the author sees exactly which cases regressed and why — an eval that only says "blocked" is useless.

8. Tradeoffs to name out loud

DecisionTradeoff
Programmatic vs LLM-judgecheap+reliable+narrow vs flexible+biased+costly; ladder them
Gate on every commit vs nightlyfast feedback vs cost; cheap checks per-commit, heavy suites nightly
Judge threshold / κ barstrict = fewer false passes but more friction; tune to the risk
Golden-set sizemore cases = better coverage but slower/costlier runs; grow via the flywheel
Online sample ratemore = better signal + more cost/latency on prod
Block vs warn on regressionblock = safety, but a flaky eval blocks good changes; reliability first

9. Failure modes (design against these)

  • A trusted-but-wrong judge — never validated against humans, silently mis-scoring. Mitigate: require and monitor Cohen's κ.
  • Golden-set contamination — cases leaked into training/few-shot, so the eval is gamed. Mitigate: strict separation, versioning, no reuse.
  • Metric myopia — one aggregate number hides a regression in a subgroup (a language, a tenant, a hard category). Mitigate: stratified reporting, safety as a separate gate.
  • A flaky eval — non-determinism blocks good PRs and erodes trust. Mitigate: inject the model/clock, reproducible runs.
  • Safety-as-average — a dangerous change passes because its quality score is high. Mitigate: safety is a hard gate.
  • Eval rot — the golden set stops reflecting production. Mitigate: the flywheel; retire stale cases.

10. The 60-second summary (say this)

"A team registers an agent and a versioned golden dataset of their real task. Every change runs through an orchestrator that scores it with a ladder — programmatic checks first, then trajectory eval, then an LLM-judge that I've validated against humans with Cohen's κ — and a regression gate blocks quality drops. Safety is a separate hard gate: a red-team suite runs the injection/ exfiltration attacks and any safety failure blocks the change regardless of quality. Production traffic is sampled for online eval and human feedback, which flywheels new cases back into the golden set. Everything is deterministic, versioned, and traced, so a blocked PR shows exactly which cases regressed and why. The golden set is the moat — you evaluate your task, not benchmarks — and this is what lets teams ship agent changes fast without silently regressing quality or safety."

Related labs: Phase 11 — Evaluation, Phase 10 — Security, Phase 14 — Observability.