« Phase 14 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 14 Warmup — Cost, Latency & Observability

Who this is for: you did the earlier phases. You can build an agent but haven't had to make one cheap, fast, and debuggable at scale. By the end you'll have built the gateway that routes, caches, meters, and traces — the production-readiness layer the enterprise JDs require.

Cost and latency are engineered, not observed
The token cost model, recapped
Model routing: cheapest-that-works
Cascades: escalate on low confidence
Exact caching
Semantic caching
Prompt-prefix and KV caching
Cost metering and $/resolved-task
Latency: TTFT, tails, and streaming
Observability: traces, not just logs
Degradation ladders
Common misconceptions
Lab walkthrough
Success criteria
Interview Q&A
References

1. Cost and latency are engineered, not observed

At one request a day, cost and latency don't matter. At a hundred thousand, they are the business: the difference between a 70% and a 20% gross margin, between a 400 ms and a 4 s p95, is made entirely of engineering choices — which model, cache or not, stream or not, how many steps. Phase 00 gave you the arithmetic; this phase gives you the controls. The mental shift: you don't "monitor cost" and hope, you build the margin — routing, caching, budgets — and instrument it so you can see it move.

2. The token cost model, recapped

Every LLM call bills input tokens (the prompt you send) and output tokens (what it generates), per million, and output usually costs 3–5× input (generating is dearer than reading). So a request's cost is input/1e6 * price_in + output/1e6 * price_out. Two levers fall out immediately: send fewer input tokens (caching, prefix reuse, context compression from Phase 04, avoiding ReAct's quadratic scratchpad from Phase 00) and generate fewer output tokens (terse tool protocols, shorter reasoning). The lab's CostMeter accumulates exactly this per request, model, and tenant.

3. Model routing: cheapest-that-works

You rarely need the flagship model for every request. Routing sends each request to the cheapest model that meets its quality bar: a classification or extraction task goes to a small fast model; a hard reasoning task goes to a big one. In the lab, a Router holds a registry of ModelSpecs (price, quality, latency) and picks the cheapest whose quality ≥ the task's min_quality. This single move can cut cost several-fold, because in most workloads the majority of requests are easy. The art is classifying the request's difficulty cheaply — a small router model, a heuristic, or the cascade below.

4. Cascades: escalate on low confidence

A cascade is routing over time: try the cheap model first, check the answer's confidence, and escalate to a stronger model only if it's not good enough. In the lab, route_cascade(task, judge) runs the cheap model, applies an injected judge(answer) → confidence, and if it's below threshold retries on the next model up. The economics are compelling: if the cheap model handles 80% of requests acceptably, you pay flagship prices on only 20% while keeping quality high. The cost is added latency on the escalated 20% (two calls) and the need for a reliable confidence signal — which is where Phase 11's evaluation and calibration come back in. (RouteLLM and FrugalGPT are the published versions of this idea.)

5. Exact caching

The cheapest request is the one you don't make. An exact cache keys on the prompt (and model, params): identical prompt → return the stored answer, no model call, ~0 cost and instant latency. The catch is that LLM inputs are rarely byte-identical (a timestamp, a user name, whitespace), so exact caching hits mostly on truly repeated calls (a fixed system-prompt eval, a popular FAQ). It needs a TTL (time-to-live) so stale answers expire — the lab's ExactCache uses the injected clock for deterministic expiry. Exact caching is free correctness for the hits it gets; the question is the hit rate.

6. Semantic caching

To catch near-duplicates ("what time do you open?" vs "when are you open?"), a semantic cache keys on the meaning: embed the query, and if its cosine similarity to a cached query exceeds a threshold, return that answer. The lab's SemanticCache uses a stdlib hashing embedder + cosine. The power is a much higher hit rate on paraphrase-heavy traffic (support, search); the danger is a false hit — two queries that are close in embedding space but want different answers ("cancel my order" vs "cancel my subscription"). So the threshold is a precision/recall dial you tune with real traffic, and high-stakes queries should bypass the semantic cache. GPTCache is the reference implementation.

7. Prompt-prefix and KV caching

Agent prompts share huge leading chunks — the same system prompt and tool schemas on every call (and, in a ReAct loop, a growing shared scratchpad). Prompt-prefix caching (offered by Anthropic and OpenAI) lets the provider reuse the computation for a shared prefix, so you pay reduced input price on the cached portion. Under the hood this is KV-cache reuse: the attention keys/values for the prefix tokens are computed once and reused, which is also why it cuts latency (less prefill). The lab's shared_prefix_tokens(a, b) counts the reusable leading overlap so you can reason about the saving. Design implication: put the stable content (system, tools, few-shot) at the front of the prompt and the variable content at the back, to maximize the cacheable prefix.

8. Cost metering and $/resolved-task

You can't manage what you don't measure. A cost meter accumulates tokens and dollars per request, per model, and per tenant (for chargeback and noisy-neighbor detection — Phase 13). But the number that matters, from Phase 00, is $/resolved-task = $/attempt ÷ success_rate: a cheap model that fails half the time is not cheap. So you join the cost meter (this phase) with the eval success rate (Phase 11) to get the true unit cost, and that's what a margin conversation divides by. The lab's CostMeter.report() gives $/request; multiply through your eval to get $/resolved-task.

9. Latency: TTFT, tails, and streaming

Two latencies matter: time-to-first-token (TTFT) — how long until the user sees anything — and total — how long until the answer is complete. Streaming (Phase 12) makes TTFT the felt latency, which is why chat UIs stream. And, from Phase 00, you design to the tail (p95/p99), not the mean: agent loops are sequential so latencies sum, and one slow step or a cache miss lands a user in the tail. Caching cuts both cost and tail latency (a hit is instant); routing to a faster model cuts latency at some quality cost. The lab's tracer records per-span durations so you can see where the milliseconds go.

10. Observability: traces, not just logs

When an agent request is slow, wrong, or expensive, logs tell you what happened but not how the pieces related. A trace does: it's a tree of spans (a unit of work with a start, end, and attributes), so one request produces a span for the llm-call inside the tool-call inside the retrieval inside the request, with durations and token counts on each. The lab's Tracer builds exactly this nested Span tree. The industry standard is OpenTelemetry, which now has GenAI semantic conventions — standard attribute names like gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — so your traces interoperate with LangSmith, Langfuse, Helicone, Datadog, and the rest. Instrumenting with these from the start is what lets you answer "what did this agent do, for whom, at what cost, and where did the time go?" for any request.

11. Degradation ladders

Under budget pressure or load, a mature platform degrades gracefully instead of failing. A degradation ladder steps down: best model → cheaper model → cached/approximate answer → a canned response — each rung cheaper and faster, none of them an error page. The lab's serve_under_budget(task, remaining_budget) walks this ladder as the budget shrinks. This is the production instinct that "a slightly worse answer now beats a perfect answer that blew the budget or timed out" — and it's a great thing to raise in a design review, because most candidates only think about the happy path.

12. Common misconceptions

"Caching LLM output is easy." Exact hits are rare (inputs vary); semantic hits risk false positives; both need TTLs and careful invalidation. It's subtler than a dict.
"Just use the best model." At scale that's a margin-killer; route the easy majority to cheap models and cascade the hard minority.
"Cost per request is the metric." It's $/resolved-task — cost divided by the eval success rate.
"The average latency is fine." Users live in the p95/p99 tail; design to it.
"Logs are enough observability." Logs don't show how spans relate or where time went; you need traces (OTel).
"Put variable content first in the prompt." Put stable content first to maximize the cacheable prefix.

13. Lab walkthrough

Open lab-01-agent-gateway-cache-router/ and fill the TODOs: tokenize/hashing_embed/cosine/shared_prefix_tokens; the Router (cheapest-meeting- quality) and route_cascade (escalate on the injected judge); ExactCache and SemanticCache (TTL via the injected clock, threshold cosine); the CostMeter (+ $/request); the Tracer (nested spans with durations); serve_under_budget (the degradation ladder); and the Gateway that wires cache → route/cascade → meter → trace. Run LAB_MODULE=solution pytest -v first, then match it. solution.py's main() shows a cache hit avoiding a call, a cascade escalating, the cost report, a rendered trace tree, and the ladder stepping down.

14. Success criteria

You can explain routing vs cascade and when each fires.
You can explain exact vs semantic vs prefix caching and their staleness/false-hit tradeoffs.
You can compute $/resolved-task from the meter and an eval success rate.
You can read a span tree and say where the latency went.
All 37 tests pass under lab and solution.

15. Interview Q&A

Q: An agent's cost is too high at scale. What levers do you pull? A: First profile tokens (often ReAct's quadratic scratchpad — Phase 00). Then: route the easy majority of requests to a cheaper model and cascade only the hard ones to the flagship; add exact + semantic caching for repeated and near-duplicate queries; enable prompt-prefix caching by putting stable content (system, tools) at the front; trim output tokens. Measure $/resolved-task, not $/attempt, so a cheap-but-failing model doesn't look like a win.

Q: Exact vs semantic caching — tradeoffs? A: Exact caches on the literal prompt: zero false hits, but a low hit rate because inputs vary. Semantic caches on embedding similarity: a much higher hit rate on paraphrase traffic, but risk of a false hit where two close queries want different answers, so the threshold is a precision/recall dial and high-stakes queries should bypass it. Both need TTLs.

Q: How do you observe an agent in production? A: Distributed tracing — OpenTelemetry with the GenAI semantic conventions — so each request is a tree of spans (retrieval, llm-call, tool-call) with durations and token/cost attributes, exported to a backend (LangSmith/Langfuse/Helicone/ Datadog). Logs tell you what happened; traces tell you how the pieces related and where the time and money went, which is what you need at 2 a.m.

Q: What's a cascade and when is it worth it? A: Try a cheap model, check confidence, escalate to a stronger model only on low confidence. Worth it when a large fraction of requests are handled acceptably by the cheap model — you pay flagship prices on only the hard minority. The costs are extra latency on escalations and needing a reliable confidence signal (tie to Phase 11 eval).

Q: What's a degradation ladder? A: Under budget or load pressure, step down gracefully — best model → cheaper → cached/approximate → canned — instead of erroring. A slightly worse answer that ships beats a perfect one that times out or blows the budget. It's the production instinct most candidates skip.

16. References

OpenTelemetry — GenAI semantic conventions. https://opentelemetry.io/docs/specs/semconv/gen-ai/
Anthropic prompt caching. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching · OpenAI prompt caching. https://platform.openai.com/docs/guides/prompt-caching
Chen et al., FrugalGPT (2023). https://arxiv.org/abs/2305.05176
Ong et al., RouteLLM (2024). https://arxiv.org/abs/2406.18665
GPTCache (semantic caching). https://github.com/zilliztech/GPTCache
LangSmith / Langfuse / Helicone — LLM observability platforms.
Dean & Barroso, The Tail at Scale, CACM 2013.

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems