Agent Evals

Phase 12 · Document 04 · Evaluation Prev: 03 — RAG Evals · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Agents are multi-step, non-deterministic, branching, and high-stakes (Phase 10), which makes them the hardest thing to evaluate — and the most dangerous to ship un-evaluated. The defining fact is that per-step reliability compounds: 95%/step over 10 steps ≈ 60% end-to-end (Phase 10.09, Phase 5.06), so a single weak tool schema silently tanks task success. Agent eval is also unusual in three ways that this doc frames within the Phase 12 discipline: there are many valid paths (you can't string-match), the trajectory matters (not just the answer), and safety is part of "success." (Mechanics: Phase 10.09; here we situate agent eval as an instance of the unified discipline — golden sets [01], judges [02], harness [08].)

2. Core Concept

Plain-English primer: outcome AND trajectory AND safety

Agent eval has three lenses (vs RAG's two halves, 03):

• Outcome — did it achieve the goal? Task success / resolution rate on a held-out task set (the headline) (Phase 10.06).
• Trajectory — how (the path): per-step tool-call correctness, efficiency (steps/tokens), recovery, loops — what you tune (Phase 10.08).
• Safety — was it safe? Unsafe completion = failure (unapproved destructive action, injection success) (Phase 10.05).

Outcome tells you whether; trajectory tells you why and at what cost/risk; safety can override both.

Compounding: the governing math

P(task success) ≈ ∏ P(step_i correct)   →   0.95^10 ≈ 0.60 ; 0.99^10 ≈ 0.90

End-to-end success is dominated by per-step reliability and step count (Phase 5.06). So agent eval focuses on the loop: raise per-step tool-call reliability (Phase 10.01), add recovery (recovered failures don't count against you), and reduce steps (simpler architecture, Phase 10.03) — not just pick a smarter model.

Scoring: the dataset is tasks-in-an-environment

Unlike (input→output) pairs, an agent eval item is a task: a goal + an environment + a success criterion, often run in a sandbox the agent acts in (Phase 10.05). Because there are many valid paths, you can't string-match — score by:

• Outcome checkers (best, programmatic): did the environment reach the goal state? (tests pass, DB updated, email drafted with the right fields) — objective, the Phase 12 preference (00, Phase 11.08).
• LLM-as-judge (02): for outcomes/trajectory without a clean check ("did this achieve the goal?", "were the tool calls appropriate?") — calibrate.
• Reference-trajectory: compare tool-call sequence to a known-good path — but don't penalize valid alternatives.

The golden set (01) is a set of tasks (incl. unsafe-temptation tasks: injected content, risky requests → success = "refused/contained").

Safety is a first-class success criterion

A task completed unsafely is a failure — an agent that resolves the goal but takes an unapproved destructive action, or gets prompt-injected into exfiltration, failed (Phase 10.05, Phase 10.07). Track safety violations (must be 0) as a hard gate alongside resolution; see 07 for the safety-eval mechanics.

Traces are the eval substrate

Agent eval consumes the trajectories observability produces (Phase 10.08): you score outcome from the final state, trajectory from the step trace, and localize failures (mis-called tool Phase 10.01? wandering Phase 10.03? lost goal/memory Phase 10.04? unrecovered error?). Observability and eval are built together.

Offline + online

Per the discipline (00): offline run the task set in CI, gate model/prompt/tool changes on no-regression in task success + safety; online measure completion, escalation/intervention rate, cost/task, safety events, and feed failures back into the task set (Phase 10.08).

3. Mental Model

   THREE LENSES (vs RAG's two): OUTCOME (task success/resolution — headline) + TRAJECTORY (path: tool-correctness,
        efficiency, recovery, loops) + SAFETY (unsafe completion = FAIL [10.05/10.07])

   ★ per-step reliability COMPOUNDS: P(success) ≈ ∏ P(step) → 0.95^10≈0.60 → tune the LOOP (reliability/recovery/fewer steps), not model IQ [5.06]

   DATASET = TASKS (goal + ENVIRONMENT + success criterion), often a SANDBOX [10.05]; many valid paths → NO string-match
   SCORE: OUTCOME CHECKERS (programmatic, best) > LLM-JUDGE (calibrate [02]) > ref-trajectory (don't penalize valid paths)
   TRACES [10.08] = eval substrate → localize: tool [10.01] / planning [10.03] / memory [10.04] / safety [10.05]
   OFFLINE gate (success + safety) + ONLINE (escalation/cost/safety) → feed failures back

Mnemonic: eval outcome + trajectory + safety. Per-step reliability compounds, so tune the loop. The dataset is tasks-in-an-environment with many valid paths — prefer programmatic outcome checks, calibrate judges, score from traces, and treat unsafe completion as failure.

4. Hitchhiker's Guide

What to look for first: a task set with success criteria (programmatic where possible) and a task-success + safety number you gate on. Then per-step tool reliability + cost/task from traces (Phase 10.08).

What to ignore at first: trajectory-similarity metrics. Start with programmatic outcome checks + a calibrated judge on a small task set; add trajectory detail later. (Mechanics: Phase 10.09.)

What misleads beginners:

• Eval'ing the final answer only. Misses why and the cost/safety of the path — eval the trajectory too.
• Ignoring compounding. Great single-step demos hide low end-to-end success over many steps (Phase 5.06).
• String-matching trajectories. Many valid paths — check outcomes or judge appropriateness.
• Leaving safety out of success. Unsafe completion is a failure (Phase 10.05).
• Trusting the judge. Calibrate (02).

How experts reason: they build a task set (goal + env + success criteria) with programmatic outcome checks + unsafe-temptation tasks, score outcome + trajectory + safety, calibrate judges, gate in CI on success + safety + cost, attribute failures to a stage via traces, and remember success is the loop (per-step reliability), not model IQ.

What matters in production: task-success/resolution, zero safety violations, cost/steps per task, recovery rate, escalation/intervention rate, and a regression gate + online feedback (Phase 10.08).

How to debug low task-success: from traces (Phase 10.08) find where steps fail — invalid tool calls (Phase 10.01)? wandering/looping (Phase 10.03)? lost goal (Phase 10.04)? unrecovered errors? Fix the dominant per-step failure; re-eval.

Questions to ask: task set with success criteria (programmatic?)? outcome + trajectory + safety scored? judge calibrated? CI gate + online feedback? per-step reliability + cost/task?

What silently gets expensive/unreliable: answer-only eval (can't localize), ignored compounding (overestimated reliability), safety excluded (unsafe ships), uncalibrated judges, no gate (silent regressions).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Agent eval has three lenses: outcome (task success), trajectory (path/efficiency/recovery), and safety (Phase 10.09).
• Per-step reliability compounds (∏ P(step); 0.95¹⁰≈0.60) — success is dominated by the loop, not model IQ (Phase 5.06).
• The dataset is tasks-in-an-environment (goal + env + success criterion), often a sandbox (Phase 10.05).
• Many valid paths → don't string-match — prefer programmatic outcome checkers, else a calibrated judge (02).
• Safety is a first-class success criterion — unsafe completion is a failure (Phase 10.05/10.07); include unsafe-temptation tasks; see 07.
• Traces are the eval substrate (Phase 10.08) — score outcomes from state, trajectory from steps, and localize failures to a stage.
• Gate offline (success + safety) + measure online (completion/escalation/cost/safety), feeding failures back.
• Improve success by raising per-step reliability + recovery + reducing steps, not just swapping the model.

9. Observations from Real Systems

• τ-bench, WebArena, AgentBench, SWE-bench evaluate task success in environments, not output strings — the field's shift to outcome-in-an-env eval (Phase 10.09, 05).
• Langfuse/LangSmith/Braintrust score trajectory + outcome from captured traces (Phase 10.08).
• The compounding lesson is empirical — high single-step accuracy still fails long tasks; teams optimize per-step reliability + step count (Phase 5.06).
• Safety evals (injection resistance, unsafe-action refusal) are increasingly part of agent test suites (Phase 10.07, 07).
• Programmatic outcome checks are preferred wherever the task has a checkable end state — objective and cheap vs judge noise.

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE AN AGENT:
 1. TASK SET [01]: goals + ENVIRONMENT + SUCCESS CRITERION (programmatic where possible); SANDBOX [10.05];
    include UNSAFE-TEMPTATION tasks (injection/risky → success = refused/contained) [10.07/07].
 2. SCORE: OUTCOME (programmatic checker preferred; else calibrated judge [02]) + TRAJECTORY (tool-correctness/
    efficiency/recovery/loops, from traces [10.08]) + SAFETY (violations = 0, HARD gate).
 3. ATTRIBUTE failures to a stage: tool [10.01] / planning [10.03] / memory [10.04] / safety [10.05].
 4. RAISE success: per-step reliability + recovery + FEWER steps (simpler architecture [10.03]) — not just model IQ [5.06].
 5. GATE offline (success + safety + cost); MEASURE online (completion/escalation/cost/safety); feed failures back.

12. Hands-On Lab

Goal

Build an agent eval that scores outcome + trajectory + safety, demonstrates compounding, and gates a change — the Phase 12 discipline applied to agents.

Prerequisites

• An agent with tracing (Phase 10.06/10.08); 8–15 tasks with programmatic success criteria, incl. 2–3 unsafe-temptation tasks.

Steps

1. Task set: define goals + env + a programmatic success check per task; add unsafe-temptation tasks with success = "refused/contained" (Phase 10.05/10.07).
2. Outcome eval: run each task N times; compute task success rate via the programmatic checks (or a calibrated judge where none exists, 02).
3. Trajectory eval: from traces (Phase 10.08) compute per-step tool-call correctness, steps/tokens/cost per task, recovery/loop rate.
4. Compounding demo: plot success vs steps required; show longer tasks have lower success, and improving per-step tool reliability (Phase 10.01) lifts end-to-end (Phase 5.06).
5. Safety gate: verify unsafe-temptation tasks score success only if refused/contained; any unapproved destructive action = fail (Phase 10.05).
6. Regression gate: make a change (weaken a tool schema / remove a verification loop); re-run; show task success/safety drops and the gate catches it.

Expected output

An eval report (task success, per-step correctness, cost/task, recovery/loop, safety), a compounding plot, and a regression-gate before/after.

Debugging tips

• High demo, low eval success → compounding; check per-step reliability (Phase 10.01).
• Judge disagrees with humans → calibrate / prefer programmatic checks.

Extension task

Attribute failures to a stage (tool/planning/memory/safety) from traces and fix the dominant one; re-measure (Phase 10.08).

Production extension

Wire success + safety into a CI gate; add online signals (completion/escalation/cost/safety) feeding the task set (Phase 10.08, 08).

What to measure

Task success, per-step tool correctness, steps/tokens/cost per task, recovery/loop rate, safety violations (0), compounding curve, regression-gate delta.

Deliverables

• A task set (goals + env + programmatic success + unsafe-temptation).
• An outcome + trajectory + safety eval report.
• A compounding demo + a regression gate before/after.

13. Verification Questions

Basic

1. What are the three lenses of agent eval, and why all three?
2. Why does per-step reliability dominate task success?
3. Why is the dataset "tasks-in-an-environment" rather than input→output pairs?

Applied 4. Why can't you string-match agent trajectories, and what do you do instead? 5. Why is safety a success criterion, not a separate concern?

Debugging 6. Task success is low despite a strong model. How do you localize the failing stage? 7. The agent "succeeds" but occasionally takes an unapproved destructive action. How does eval score that?

System design 8. Design an agent eval (task set + env, outcome+trajectory+safety, CI gate, online feedback).

Startup / product 9. Why is an agent eval harness (with safety) a prerequisite for safely shipping and iterating an agent product?

14. Takeaways

1. Eval outcome + trajectory + safety — whether, why/cost, and is-it-safe.
2. Per-step reliability compounds — tune the loop (reliability/recovery/fewer steps), not model IQ (Phase 5.06).
3. Dataset = tasks-in-an-environment; many valid paths → prefer programmatic outcome checks, calibrate judges (02).
4. Safety is first-class — unsafe completion is failure; include unsafe-temptation tasks (Phase 10.05/07).
5. Score from traces, gate offline (success + safety), feed online failures back (Phase 10.08).

15. Artifact Checklist

• A task set (goals + environment + programmatic success + unsafe-temptation).
• An outcome + trajectory + safety eval report.
• A compounding demonstration (success vs steps).
• A safety gate (unsafe completion = fail).
• A CI regression gate + online-feedback plan.

Up: Phase 12 Index · Next: 05 — Code Evals