Lab 01 — Agent Reliability, Cost & Latency Calculator

Phase 00 · Lab 01 · Phase README · Warmup

The problem

Before you build an agent, four numbers decide the architecture — and every one of them is arithmetic you can do on a whiteboard in the interview:

  1. Reliability compounds. Ten steps that each work 95% of the time succeed end to end only \(0.95^{10} \approx 0.60\) of the time. How many autonomous steps can you afford?
  2. Cost has a shape. A ReAct loop resends the whole scratchpad every step, so its input tokens grow quadratically. ReWOO plans once and executes without re-feeding the model, so its tokens grow linearly. Which is cheaper, and by how much?
  3. Latency lives in the tail. The mean lies; p95/p99 page you. Does the sequential step chain fit the budget at p95?
  4. Workflow or agent? The "least-agentic thing that works" — a known, non-branching path is a workflow, not an agent, no matter how fashionable agents are.

You will implement all four as small, exact, testable functions.

What you build

FunctionWhat it computesThe lesson
compound_reliability(p, n)\(p^n\)why long autonomous chains fail
max_autonomous_steps(p, target)\(\lfloor \log target / \log p \rfloor\)your step budget
reliability_with_retries(p, r)\(1-(1-p)^{r+1}\)the other reliability lever
react_cost / rewoo_costtokens + dollars per strategyquadratic vs linear token growth
cost_per_resolved_taskunit_cost / success_ratethe number the business cares about
percentile, sequential_latency, fits_budgetthe latency tail + budget checkwhy the mean lies
workflow_vs_agent(profile)a Decisionleast-agentic that works

File map

FileRole
lab.pyyour implementation (fill the # TODOs)
solution.pyreference implementation + worked example (python solution.py)
test_lab.pythe proof — 22 tests covering math, edges, quadratic-vs-linear, tails, decisions
requirements.txtpytest only

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

  • You can derive \(0.95^{10}\approx0.60\) and explain what it means for architecture without looking it up.
  • You can explain why ReAct input tokens are quadratic and ReWOO's are linear, pointing at the exact term in react_cost that grows.
  • Your percentile reports the tail correctly and you can say why the mean is a trap.
  • workflow_vs_agent recommends a workflow for a known, non-branching path — and you can defend "most things called agents should be workflows."
  • All 22 tests pass under both lab and solution.

How this maps to the real stack

  • The 0.95^n curve is the reason production agents checkpoint (Phase 08), verify with critics (Phase 07), gate high-impact actions behind a human (Phase 10), and keep step counts low. It is the single most-cited number in agent design reviews.
  • react_cost vs rewoo_cost is the real tradeoff behind LangGraph (interleaved, ReAct- like, flexible, pricier) vs a ReWOO / plan-execute graph (cheaper, less adaptive). Real frameworks let you pick per node; this lab is why you'd pick.
  • cost_per_resolved_task is the metric a FinOps / unit-economics review uses (Phase 14); cloud cost dashboards and LLM gateways (LiteLLM, Helicone, LangSmith) report the numerator, and you divide by the eval success rate (Phase 11) to get what matters.
  • percentile + fits_budget is what an SLO is made of; every serving system (vLLM, Triton, a FastAPI agent service in Phase 12) is judged on p95/p99, not the mean.
  • workflow_vs_agent is the discipline Anthropic's "Building Effective Agents" post and every senior reviewer preach: prefer the simplest composition (a chain, a router, a workflow) and only reach for an autonomous agent when the task genuinely needs runtime flexibility.

Limits of the miniature. Real steps are not independent (a good plan makes later steps more reliable; a bad one, less), token counts come from a real tokenizer (Phase 04), latency has queueing and network variance beyond a clean percentile, and prices change. The point is the shape of each relationship, which does not change.

Extensions (your own machine)

  • Add a Monte-Carlo simulator: sample each step's success from random.Random(seed), run 10k trials, and confirm the empirical end-to-end rate matches compound_reliability.
  • Model parallel tool calls (ReWOO can fan out independent evidence): latency becomes the max of a group, not the sum. Add parallel_latency(groups).
  • Pull real prices from a model's pricing page and compare a ReAct vs ReWOO agent for a 10-step research task; produce the $/resolved-task table for three models.

Interview / resume signal

"Sized an agent architecture from first principles — compounding reliability (0.95^n), ReAct-vs-ReWOO token economics (quadratic vs linear context growth), p95 latency budgets, and a workflow-vs-agent decision rule — turning 'should this be an agent?' into a defensible number instead of a vibe."