Agent Observability

Phase 10 · Document 08 · Agents and Tools Prev: 07 — Browser and Research Agents · Up: Phase 10 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

An agent is a multi-step, non-deterministic, branching process — it might take 3 steps or 30, call any tool in any order, loop, recover, or wander. You cannot debug, improve, cost-control, or trust it without seeing the whole trajectory. A single final answer tells you nothing about why it succeeded or failed: which tool was mis-called, where it looped, which step burned the tokens, where it got injected (07). Agent observability — tracing every step, tool call, and decision — is what turns an opaque, flaky agent into a debuggable, improvable system. It's the prerequisite for evaluation (09), the tool you reach for in every incident, and (because agents are where cost and risk spike, 00) non-negotiable in production. It extends Phase 7 observability (Phase 7.08) to the multi-step agent world.

2. Core Concept

Plain-English primer: trace the whole trajectory, not just the answer

For a single LLM call, you log inputs/outputs/tokens (Phase 7.08). An agent is many calls + tool executions in a loop (01), so observability means capturing the trajectory — the full sequence of steps — as a trace of nested spans:

TRACE: "research competitor pricing" (root span: task)
 ├─ span: LLM call (step 1)        in/out tokens, latency, the reasoning + tool_use proposed
 ├─ span: tool.search("…")         args, result (truncated), latency, success/error
 ├─ span: LLM call (step 2)        …
 ├─ span: tool.fetch("url")        … (was this injected? what came back?)
 ├─ span: subagent "pricing"       nested trace (its own steps) [03/07]
 └─ span: LLM call (final)         answer, stop_reason, total tokens/cost

Each step is a span (a timed unit of work with attributes); spans nest into a trace (the whole task); OpenTelemetry is the standard, and subagents nest as child traces (03, 07). This is distributed tracing applied to the agent loop.

What to capture per step

• LLM call spans: the prompt/context (or a reference), the model's reasoning + proposed tool_use, tokens (in/out/cached/reasoning), latency, model/version, stop_reason.
• Tool spans: tool name, arguments, result (truncated/redacted), latency, success/error, whether it required approval and the decision (05).
• Decision/control: step number, why it stopped, loop-detection signals, retry counts.
• Cost/latency: per-step and cumulative tokens + dollars + wall-clock — agents accumulate fast (Phase 7.09).
• Safety events: approvals/denials, sandbox blocks, injection flags (05, 07).
• Outcome: task success/failure (if known), final answer.

Agent-specific metrics (beyond per-call)

Per-call metrics (Phase 7.08) aren't enough; agents need trajectory-level metrics:

• Steps per task (and distribution) — a spike signals wandering/looping.
• Tool-call success/error rate and valid-call rate (01) — per-step reliability that compounds (Phase 5.06).
• Loop/repeat rate — same tool+args repeated (a stuck agent).
• Tokens/cost per task (not per call) — the real unit economics (Phase 7.09).
• Task success / resolution rate — the outcome that matters (09).
• Stop-reason distribution — how often it finishes vs hits a step/token/time cap (a healthy agent mostly finishes).
• Safety metrics — approvals, denials, sandbox blocks, injection attempts caught (05).

Loop and anomaly detection

Because agents can run away (00), observability feeds detectors:

• Loop detection — flag/halt when the same tool+args repeat N times or steps exceed a budget (also a stop condition, 00).
• Cost/step anomalies — alert when a task's tokens/steps exceed the normal distribution (runaway/abuse).
• Error clustering — repeated tool errors → a broken tool/schema (01). These turn passive logs into active guardrails.

Make traces inspectable (the human side)

Raw spans aren't enough; engineers (and PMs) need to read a trajectory to debug. Agent-observability tools (Langfuse, LangSmith, Phoenix/Arize, Braintrust, Helicone, W&B Weave, plus OpenTelemetry GenAI conventions) render the step tree, show each prompt/tool call/result, and link to evals (09). The ability to replay and inspect a single failed run step-by-step is the most valuable debugging capability you can build.

Privacy and cost of observability itself

Traces contain prompts, tool results, and possibly PII/secrets — redact and sample (Phase 7.08, Phase 14.06). Full-fidelity tracing of every step has storage/cost overhead; sample in high volume, keep full traces for errors/flagged runs.

3. Mental Model

   agent = multi-step branching loop → observe the TRAJECTORY, not just the answer
   TRACE (task) ⊃ SPANS (steps): LLM-call spans (reasoning+tool_use, tokens, latency) +
        tool spans (name/args/result/success/error/approval) + subagent traces nested [03/07]   (OpenTelemetry)

   TRAJECTORY METRICS: steps/task · tool success & valid-call rate [01] · LOOP/repeat rate ·
        tokens/cost PER TASK [7.09] · TASK SUCCESS [09] · stop-reason mix · safety events [05]

   detectors: LOOP detection · cost/step anomaly · error clustering → active guardrails [00]
   inspect: replay a single failed run step-by-step (Langfuse/LangSmith/Phoenix/Braintrust + OTel)
   REDACT + SAMPLE (prompts/results contain PII/secrets) [7.08/14.06]

Mnemonic: trace the whole trajectory as nested spans (LLM calls + tool calls + subagents); measure per-task (steps, tool-success, loops, cost, success), feed detectors (loops/anomalies/errors), and make a failed run replayable step-by-step — redacted and sampled.

4. Hitchhiker's Guide

What to look for first: can you replay a single failed run step-by-step (every prompt, tool call, args, result)? And do you have tokens/cost per task + steps per task? Those make agents debuggable and cost-visible.

What to ignore at first: a full custom observability platform — adopt Langfuse/LangSmith/Phoenix + OpenTelemetry rather than building from scratch.

What misleads beginners:

• Logging only the final answer. You can't debug why a multi-step agent failed without the trajectory.
• Per-call metrics only. Agents need per-task metrics (steps, cost/task, success) — per-call hides the loop (Phase 7.08).
• No loop/anomaly detection. Runaway agents (cost + actions) go unnoticed until the bill/incident (00).
• Tracing raw PII/secrets. Redact + sample; traces hold prompts/results (Phase 14.06).
• Not nesting subagents. Multi-agent traces are unreadable if subagent steps aren't child spans (03).

How experts reason: they trace every step as spans (OTel), capture reasoning + tool calls + results + safety events, compute trajectory metrics (steps, tool-success, loops, cost/task, success, stop-reason mix), wire detectors (loops/anomalies/error-clusters) as guardrails, make runs replayable, and redact + sample. They treat the trace as the input to evaluation (09) and the first stop in any incident.

What matters in production: replayable per-run traces, per-task cost/steps/success dashboards, loop/anomaly alerts, safety-event visibility, and PII-safe sampled storage.

How to debug an agent failure: open the trace → step through: did it retrieve/choose the right tool? did a tool error and was it recovered (01)? did it loop (repeat detector)? where did tokens go? was it injected (07)? The trajectory localizes the failure that the final answer hides.

Questions to ask: can I replay a failed run step-by-step? do I have steps + tokens/cost per task? loop/anomaly detection? subagents nested? PII redacted + sampled? does the trace feed eval (09)?

What silently gets expensive/unreliable: answer-only logging (undebuggable), no per-task cost/steps (runaway spend), no loop detection (infinite loops), PII in traces (compliance), and un-nested multi-agent traces (unreadable).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Observe the trajectory, not just the answer — an agent is many steps; the final output hides why it failed.
• A trace is a tree of spans (LLM-call spans + tool spans + nested subagent traces); OpenTelemetry is the standard (Phase 7.08).
• Capture per step: reasoning + tool_use, tool args/results (truncated/redacted), tokens/latency, success/error, approvals, stop_reason, safety events (05).
• Use trajectory metrics: steps/task, tool-success/valid-call rate, loop rate, tokens/cost per task, task success, stop-reason mix.
• Wire detectors (loop detection, cost/step anomaly, error clustering) as active guardrails (00).
• Make failed runs replayable step-by-step — the single most valuable agent-debugging capability.
• Redact + sample — traces hold prompts/results/PII (Phase 14.06); keep full traces for errors/flagged runs.
• The trace is the input to evaluation (09) and the first stop in any incident.

9. Observations from Real Systems

• Langfuse, LangSmith, Phoenix/Arize, Braintrust, Helicone, W&B Weave all center on trace + replay + eval for agents — the category exists because trajectory visibility is essential.
• OpenTelemetry GenAI conventions are standardizing agent/LLM spans so traces are portable across tools.
• Multi-agent systems (orchestrator–worker, 03/07) require nested traces (subagent runs as child spans) or they're impossible to debug (Anthropic multi-agent).
• The first move in any agent incident is opening the trace and stepping through — loops, mis-calls, injections, and cost spikes are all visible there.
• Cost-per-task observability is what catches runaway agents before the monthly bill (Phase 7.09).

10. Common Misconceptions

11. Engineering Decision Framework

INSTRUMENT AN AGENT:
 1. TRACE every run as nested SPANS (OTel): LLM-call spans (reasoning+tool_use, tokens, latency, stop_reason)
    + tool spans (name/args/result/success/error/approval) + subagent CHILD traces [03/07].
 2. METRICS (per TASK): steps/task · tool-success & valid-call rate [01] · loop rate · tokens/cost per task [7.09]
    · task success [09] · stop-reason mix · safety events [05].
 3. DETECTORS: loop detection · cost/step anomaly · error clustering → alerts/guardrails [00].
 4. REPLAY: render a failed run step-by-step (adopt Langfuse/LangSmith/Phoenix).
 5. PRIVACY/COST: REDACT PII/secrets; SAMPLE at volume; keep full traces for errors/flagged runs [7.08/14.06].
 6. FEED EVAL: traces are the dataset for trajectory/outcome evaluation [09].

12. Hands-On Lab

Goal

Add tracing to an agent so you can replay a failed run step-by-step, compute per-task metrics, and detect a loop.

Prerequisites

• An agent from 01/03; pip install langfuse (or Phoenix/OTel); a task that sometimes loops/fails.

Steps

1. Instrument spans: wrap each LLM call and each tool execution in a span (Langfuse/OTel) capturing reasoning + tool_use, tool name/args/result (truncated), tokens, latency, success/error, stop_reason. Nest any subagents as child traces (03).
2. Run + replay: run several tasks; open the trace UI and step through one run — see the full trajectory (every prompt/tool call/result).
3. Per-task metrics: compute steps/task, tool success rate, tokens & cost per task, stop-reason distribution across runs (Phase 7.09).
4. Loop detection: create a task that loops (e.g., a tool that keeps "failing"); implement a detector that flags same tool+args repeated N times and halts; verify the trace shows the loop and the detector fired (00).
5. Cost anomaly: flag any run whose tokens exceed the normal distribution; show it catches a runaway.
6. Redaction: inject a fake secret into a tool result and confirm your tracing redacts it (Phase 14.06).

Expected output

A traced agent with a replayable trajectory per run, per-task metrics (steps/cost/success/stop-reason), a working loop detector, and redacted traces — demonstrating you can debug and cost-control a multi-step agent.

Debugging tips

• Trace unreadable for multi-agent → subagents not nested as child spans.
• Can't find the cost → not capturing per-step tokens; add to spans.

Extension task

Wire the traces into evaluation (09) — score trajectories and outcomes from the captured runs; add error clustering across runs.

Production extension

Adopt Langfuse/LangSmith/Phoenix + OTel; sample at volume (full traces for errors); alert on loop/cost anomalies; export safety events to audit (05, Phase 14.06).

What to measure

Steps/task, tool success rate, tokens/cost per task, stop-reason mix, loop-detector firing, redaction correctness, replay usefulness.

Deliverables

• A traced agent with replayable per-run trajectories.
• Per-task metrics (steps/cost/success/stop-reason).
• A working loop/anomaly detector + redacted traces.

13. Verification Questions

Basic

1. Why isn't logging the final answer enough for an agent?
2. What is a trace vs a span, and how do subagents fit?
3. Name four per-task (trajectory) metrics.

Applied 4. How would observability help you find why an agent's cost spiked? 5. How do you detect (and stop) an agent stuck in a loop?

Debugging 6. A multi-agent run is impossible to follow. What's wrong with the tracing? 7. An agent failed and you only have the final answer. What do you wish you'd captured?

System design 8. Design agent observability: spans, per-task metrics, detectors, replay, redaction/sampling, eval feed.

Startup / product 9. Why is replayable trajectory tracing a prerequisite for trusting and improving an agent product?

14. Takeaways

1. Observe the whole trajectory — an agent is many steps; the final answer hides why it failed.
2. Trace = nested spans (LLM calls + tool calls + subagents), OTel-standard; capture reasoning, args/results, tokens, success/error, safety events.
3. Measure per task (steps, tool-success, loops, cost, success, stop-reason), not just per call.
4. Wire detectors (loop/anomaly/error-cluster) as guardrails; make failed runs replayable.
5. Redact + sample; the trace is the input to evaluation (09) and the first stop in every incident.

15. Artifact Checklist

• A traced agent (nested spans: LLM calls + tool calls + subagents).
• Per-task metrics (steps, tool-success, cost/task, success, stop-reason).
• A loop/anomaly detector wired as a guardrail.
• Replayable failed-run trajectories.
• Redaction + sampling; traces feeding 09.

Up: Phase 10 Index · Next: 09 — Agent Evaluation