Agent Evaluation

Phase 10 · Document 09 · Agents and Tools Prev: 08 — Agent Observability · Up: Phase 10 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 10: agents are non-deterministic, multi-step, and high-stakes, so you cannot ship or improve one on vibes — you need evaluation. The defining fact is that per-step reliability compounds (00, Phase 5.06): 95%-per-step over 10 steps ≈ 60% end-to-end, so a single bad tool schema or missing recovery silently tanks task success. Agent eval is harder than single-output eval because there are many valid paths to a goal, the trajectory matters (not just the answer), and safety is part of "success." Without it you can't choose a model (Phase 5.06), gate a change, or trust the agent in production. This doc ties together everything in the phase — tool calling, planning, memory, safety, observability — into the discipline that says did it actually work, reliably and safely?

2. Core Concept

Plain-English primer: evaluate outcomes AND trajectories

Agent eval has two complementary lenses:

• Outcome eval (did it succeed?): the end-to-end result — task success / resolution rate on a held-out task set. The metric that ultimately matters (e.g., SWE-bench resolution for code, 06; answer faithfulness for research, 07).
• Trajectory eval (how did it get there?): the path — tool-call correctness, efficiency (steps/tokens), recovery, and whether it stayed safe. Two agents can both "succeed," but one took 4 clean steps and the other 25 with a near-miss destructive action.

You need both: outcome tells you whether, trajectory tells you why and at what cost/risk — and is what you tune.

Per-step reliability compounds (the governing math)

P(task success) ≈ ∏ P(step_i correct)     (roughly, for sequential dependence)
   0.95 per step, 10 steps  → 0.95^10 ≈ 0.60     0.99^10 ≈ 0.90

So end-to-end success is dominated by per-step reliability and step count (Phase 5.06). Implications you eval for: raise per-step tool-call reliability (01), add error recovery (failed steps you recover from don't count against you), and reduce step count (simpler architecture, 03). This is why agent eval focuses on the loop, not just the model's raw IQ.

The agent metric set

Safety is a first-class success criterion — an agent that completes the task but took an unapproved destructive action failed (05).

How to score (the hard part: many valid paths)

Unlike a fixed answer, agents have many valid trajectories, so you can't string-match. Scoring approaches:

• Outcome checkers (best when available): programmatic success tests — tests pass (06), DB in the right state, the email drafted with the right fields. Objective and cheap — use whenever the task has a checkable end state.
• LLM-as-judge for trajectory/outcome: a judge model scores "did this achieve the goal?", "were the tool calls appropriate?", "is the answer grounded?" — for tasks without a clean programmatic check (research quality, 07). Noisy/biased — calibrate against human labels (Phase 9.09, Phase 12.02).
• Reference-trajectory comparison: compare against a known-good path (did it call the expected tools?) — useful for tool-call correctness, but beware penalizing valid alternative paths.
• Human eval: for nuanced/high-stakes quality, the gold standard on a sample; expensive, so reserve it and use it to calibrate the judge.

The eval dataset is a set of tasks (goals + an environment + a success criterion), not (input→output) pairs — often run in a sandboxed env the agent acts in (05). Build it from real usage + hard cases, include unsafe-temptation tasks (does it resist injection / refuse risky actions?).

Offline vs online (and the trace connection)

• Offline: run the task set in CI; gate changes on no-regression in task success + safety. This is your guard when you swap models/prompts/tools (Phase 5.06).
• Online: measure on live traffic — task completion, user thumbs, escalation/intervention rate, cost/task, safety events — and feed failures back into the task set (08).
• Traces are the eval substrate: observability (08) produces the trajectories you score; eval consumes them. Build them together.

3. Mental Model

   TWO LENSES: OUTCOME (task success/resolution — the headline) + TRAJECTORY (path: tool-correctness,
        efficiency steps/tokens/cost, recovery, SAFETY, loops) — need both (whether vs why/cost/risk)

   ★ per-step reliability COMPOUNDS: P(success) ≈ ∏ P(step) → 0.95^10≈0.60 → raise step reliability [01],
        add recovery, reduce steps [03]; success is the LOOP, not just model IQ [5.06]

   SAFETY is a success criterion: task done + unapproved destructive action = FAIL [05/07]

   SCORE (many valid paths → no string match):
     OUTCOME CHECKERS (programmatic, best) > LLM-JUDGE (calibrate vs humans [9.09/12.02]) > ref-trajectory > human
   DATASET = TASKS (goal + env + success criterion), often in a SANDBOX [05]; include unsafe-temptation tasks
   OFFLINE gate (CI) + ONLINE (completion/escalation/cost/safety) ; traces [08] are the substrate

Mnemonic: eval outcome (did it succeed?) AND trajectory (how, at what cost/risk?). Per-step reliability compounds, so tune the loop. Safety is part of success. Prefer programmatic outcome checks; calibrate judges; the dataset is tasks-in-an-environment, gated in CI.

4. Hitchhiker's Guide

What to look for first: a task set with success criteria (ideally programmatic) and a task-success + safety number you gate on. Then per-step tool-call reliability and cost/task from traces (08).

What to ignore at first: elaborate trajectory-similarity metrics. Start with outcome checkers where possible + a calibrated LLM-judge where not, on a small task set; expand later.

What misleads beginners:

• Eval'ing the final answer only. Misses why it failed and the cost/safety of the path — eval the trajectory too (08).
• Ignoring compounding. Good single-step demos hide low end-to-end success over many steps (Phase 5.06).
• String-matching trajectories. Many valid paths exist — penalizing alternatives is wrong; check outcomes or judge appropriateness.
• Leaving safety out of "success." A task done unsafely is a failure (05).
• Trusting the judge as truth. Calibrate against humans; use for direction/regressions (Phase 12.02).
• No offline gate. Swapping a model/prompt silently regresses task success/safety.

How experts reason: they build a task set (goals + env + success criteria) with programmatic outcome checks where possible, score outcome + trajectory + safety, calibrate any LLM-judge, gate changes in CI on task-success + safety + cost, and feed online failures back. They attribute failures to a stage (tool schema [01], planning [03], memory [04], safety [05]) using traces (08) — and remember success is dominated by the loop's per-step reliability, not model IQ alone.

What matters in production: task-success/resolution rate, zero safety violations, cost/task and steps/task trends, recovery rate, and a regression gate + online feedback loop.

How to debug a low task-success: from traces (08), find where steps fail — invalid tool calls (01/02)? wandering/looping (planning [03])? lost goal (memory [04])? unrecovered errors? Fix the dominant per-step failure; re-eval.

Questions to ask: what's the task set + success criterion (programmatic?)? do we eval trajectory + safety, not just outcome? is the judge calibrated? is there a CI gate + online feedback? what's per-step reliability and cost/task?

What silently gets expensive/unreliable: answer-only eval (can't localize), ignored compounding (overestimated reliability), safety excluded from success (unsafe ships), uncalibrated judges (false confidence), and no gate (silent regressions).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Evaluate outcome (task success/resolution) AND trajectory (path, efficiency, recovery, safety) — both are needed.
• Per-step reliability compounds (P(success) ≈ ∏ P(step); 0.95¹⁰≈0.60) — success is dominated by the loop, not model IQ (Phase 5.06).
• Safety is a first-class success criterion — a task completed unsafely is a failure (05/07).
• Agents have many valid paths — don't string-match; prefer programmatic outcome checkers, else a calibrated LLM-judge (Phase 9.09).
• The eval dataset is tasks (goal + environment + success criterion), often run in a sandbox (05) — include unsafe-temptation tasks.
• Traces (08) are the eval substrate — observability and eval are built together.
• Gate changes offline in CI (task success + safety + cost) and measure online (completion/escalation/cost/safety), feeding failures back.
• Attribute failures to a stage (tool schema [01], planning [03], memory [04], safety [05]) to fix the dominant per-step failure.

9. Observations from Real Systems

• SWE-bench (code) and τ-bench / WebArena / AgentBench (tool/web agents) evaluate task success in environments, not output strings — the field's shift to outcome-in-an-env eval (06/07).
• Langfuse/LangSmith/Braintrust score both trajectory and outcome from captured traces (08).
• The compounding lesson is empirical: agents with high single-step accuracy still fail long tasks — teams optimize per-step reliability + step count, not just model choice (Phase 5.06).
• Safety evals (injection resistance, unsafe-action refusal) are increasingly part of agent test suites (05, 07, Phase 14).
• Programmatic outcome checks (tests pass, env state) are preferred wherever the task has a checkable end state — objective and cheap vs judge noise.

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE AN AGENT:
 1. TASK SET: goals + ENVIRONMENT + SUCCESS CRITERION (programmatic where possible); run in a SANDBOX [05];
    include unsafe-temptation/injection tasks [07].
 2. SCORE:
      OUTCOME — task success/resolution (programmatic checker preferred; else calibrated LLM-judge [9.09]).
      TRAJECTORY — per-step tool correctness [01], efficiency (steps/tokens/cost [7.09]), recovery, loops [08].
      SAFETY — zero unapproved/destructive/injection successes [05/07] (a hard gate).
 3. ATTRIBUTE failures to a stage via traces [08]: tool schema [01/02] / planning [03] / memory [04] / safety [05].
 4. RAISE success: per-step reliability + recovery + FEWER steps (simpler architecture [03]) — not just model IQ [5.06].
 5. GATE offline in CI (success + safety + cost); MEASURE online (completion/escalation/cost/safety); feed failures back.

12. Hands-On Lab

Goal

Build an agent eval harness that scores outcome + trajectory + safety on a task set, demonstrates compounding, and gates a change.

Prerequisites

• An agent from 01/06/07 with tracing (08); 8–15 tasks with programmatic success criteria (e.g., failing tests to fix, an env to reach a state), incl. 2–3 unsafe-temptation tasks.

Steps

1. Task set: define goals + environment + a programmatic success check per task (tests pass / state assertion); add unsafe-temptation tasks (injected content, a destructive request) with the success criterion "refused/contained" (05/07).
2. Outcome eval: run each task N times; compute task success rate via the programmatic checks. (Where no programmatic check exists, add a calibrated LLM-judge and compare to a few human labels, Phase 9.09.)
3. Trajectory eval: from traces (08), compute per-step tool-call correctness, steps/tokens/cost per task, recovery rate, loop rate.
4. Compounding demo: plot task success vs number of steps required; show longer tasks have lower success — and that improving per-step tool reliability (01, better schema) lifts end-to-end success (Phase 5.06).
5. Safety gate: verify the unsafe-temptation tasks score as success only if the agent refused/was contained; any unapproved destructive action = fail (05).
6. Regression gate: make a change (e.g., weaken a tool schema, or remove the verification loop in a code agent); re-run; show task success/safety drops and the gate catches it.

Expected output

An eval report: task success rate, per-step tool correctness, cost/task, recovery/loop rate, and safety pass — plus a compounding plot and a regression-gate before/after demonstrating the change's impact.

Debugging tips

• Success high in demo, low in eval → compounding over more steps; check per-step reliability (01).
• Judge disagrees with humans → calibrate / prefer programmatic checks where possible.

Extension task

Attribute failures to a stage (tool schema/planning/memory/safety) from traces and fix the dominant one; re-measure the lift.

Production extension

Wire the task-success + safety eval into CI as a gate, add online signals (completion/escalation/cost/safety), and feed failures back into the task set (08, Phase 12).

What to measure

Task success rate, per-step tool correctness, steps/tokens/cost per task, recovery/loop rate, safety violations (target 0), compounding curve, regression-gate delta.

Deliverables

• A task set (goals + env + programmatic success + unsafe-temptation tasks).
• An outcome + trajectory + safety eval report.
• A compounding demonstration + a regression gate before/after.

13. Verification Questions

Basic

1. What's the difference between outcome and trajectory evaluation, and why need both?
2. Why does per-step reliability dominate task success?
3. Why is safety a success criterion, not a separate concern?

Applied 4. Why can't you string-match agent trajectories, and what do you do instead? 5. When do you prefer a programmatic outcome checker over an LLM-judge?

Debugging 6. Task success is low despite a strong model. How do you localize the failing stage? 7. The agent "succeeds" but occasionally takes an unapproved destructive action. How does eval treat that?

System design 8. Design an agent eval system: task set + environment, outcome+trajectory+safety scoring, CI gate, online feedback.

Startup / product 9. Why is an agent eval harness (with safety) a prerequisite for shipping and iterating an agent product safely?

14. Takeaways

1. Evaluate outcome (task success/resolution) AND trajectory (path, efficiency, recovery, safety) — whether vs why/cost/risk.
2. Per-step reliability compounds — raise step reliability + recovery and reduce step count; success is the loop, not model IQ (Phase 5.06).
3. Safety is a first-class success criterion — unsafe completion is failure (05/07).
4. Prefer programmatic outcome checkers; calibrate any LLM-judge; the dataset is tasks-in-an-environment, scored from traces (08).
5. Gate changes offline in CI (success + safety + cost) and measure online, attributing failures to a stage and feeding them back.

15. Artifact Checklist

• A task set (goals + environment + programmatic success criteria + unsafe-temptation tasks).
• An outcome + trajectory + safety eval report.
• A compounding demonstration (success vs steps; per-step-reliability lift).
• A safety gate (unsafe completion = fail).
• A CI regression gate + an online-feedback plan.

Up: Phase 10 Index · Next: Phase 11 — AI Coding Platforms