« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 14 — Cost, Latency & Observability: Caching, Routing, Tracing & Metering

Answers these JD lines: Cohere's "reliability, latency, cost, observability"; Citi's "optimize cost, latency, prompts, caching, and vector indexes" and "MLOps"; the production- readiness bar in every enterprise role in jd.md.

Why this phase exists

Phase 00 said cost and latency are engineered, not observed. This phase builds the engine. Every agent request costs money (tokens) and time (latency), and at scale both are dominated by choices you make in code: which model handles the request, whether the answer was already cached, how you traced and metered it. An agent platform that ships without cost telemetry finds out from the invoice; one that ships without tracing can't answer "what did this agent do, for whom, at what cost?" when it's paged. This phase makes the money and the milliseconds visible and controllable.

Concept map

• Model routing & cascades: pick the cheapest model that meets the task's quality bar; escalate to a stronger model only when a confidence check fails (a cascade).
• Caching: an exact cache (same prompt → same answer, with a TTL), a semantic cache (near-duplicate query → cached answer by cosine), and prompt-prefix reuse (shared leading tokens cut input cost).
• Cost metering: accumulate input/output tokens and dollars per request/model/tenant; report $/request and connect to Phase 00's $/resolved-task.
• Tracing: OpenTelemetry-GenAI-style nested spans (llm-call, tool-call, retrieval) with durations, so a run is debuggable.
• Degradation ladder: under a shrinking budget or load, step down best → cheaper → cached → canned, gracefully, instead of failing.

The lab

Integrated scenario

A support agent handles 100k requests/day; the bill is climbing. You profile: 40% of queries are near-duplicates (a semantic cache serves them for ~$0), most of the rest are simple enough for a cheap model (route by quality bar), and only the hard 10% need the flagship (a cascade catches those via a confidence check). You add a cost meter reporting $/resolved-task per tenant and OTel spans so you can see where the latency goes. The bill drops 60%, p95 improves because cache hits are instant, and when a tenant complains about a slow request you have the trace. That is the difference between a demo and a platform, and it's Cohere's exact JD.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py.
• You can explain routing vs cascade and when each fires.
• You can explain exact vs semantic vs prefix caching and their staleness tradeoffs.
• You can read an OTel span tree and compute $/resolved-task.

Key takeaways

• Cost and latency are levers you pull (routing, caching, budgets), not numbers you watch.
• Instrument tokens, cost, and traces on day one — the alternative is finding out from finance.
• Cache by exactness and meaning and prefix; each cuts a different slice of cost/latency.
• A cheap model that fails costs more per resolved task than an expensive one that works (Phase 00).