« Phase 14 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 14 — Principal Deep Dive: Cost & Latency as a Production System

The Deep Dive traced the mechanisms. This doc is about the platform they compose into: where the cost/latency controls sit in a real agent gateway, how they scale, what fails and how far the blast radius reaches, and which "looks wrong" decisions are load-bearing. The through-line: at 100k requests/day, cost and latency are not metrics you watch — they are the margin and the SLA, built out of routing, caching, and metering, and instrumented so you can see them move.

The gateway is a pipeline of cost/latency stages

The architecture is a single request pipeline: cache lookup → route/cascade → meter → trace → (degrade under pressure). The ordering is the design. Cache first because a hit short-circuits everything downstream at ~0 cost and instant latency — you never want to route or meter a request you could have skipped. Route/cascade next to pick the cheapest model that clears the bar. Meter and trace wrap the whole thing so every request is attributed and observable whether it hit, routed, or escalated. Degrade last, as a pressure-release valve, when the budget or latency envelope is shrinking. Getting the order wrong — metering before caching, say — means you pay the observability cost on requests you should have never processed, and at 100k/day that overhead is itself a line item.

Scaling and the cost envelope

There is no single "cost number"; your envelope is a composition of hit rates and routing mix.

  • The cache hit rate is your biggest lever and it is workload-shaped. Exact caching earns little on high-variance inputs and a lot on repeated evals/FAQs; semantic caching earns a lot on paraphrase-heavy support traffic and risks false hits on operationally-distinct near-duplicates. You do not "add caching" — you profile the traffic, estimate the hit rate per cache type, and size the store for the hot set. A 40% semantic hit rate on support traffic is a 40% cost cut and a p95 improvement, because hits are instant.
  • The routing mix is the second lever. If 80% of requests clear the cheap model's bar, you pay flagship prices on 20%, and the blended cost is 0.8·cheap + 0.2·flagship, not flagship. The cascade buys this at the price of added latency on the escalated fraction — two sequential calls — so the mix is a cost/latency tradeoff, not a free lunch.
  • The tail dominates the SLA. Because agent loops are sequential, per-step latencies sum, and the p99 is set by the worst step on the worst path — a cold cache, an escalation, a retry. You size and alarm to p95/p99, not the mean, because the mean is exactly where slow requests hide.

The capacity math that matters is the unit economics: $/resolved-task = $/attempt ÷ success_rate. A cheaper model that fails more is more expensive per real result. This is the formula a principal puts on the whiteboard when someone proposes "just use the cheap model to save money."

Failure modes and blast radius

The interesting failures here are not crashes — they are quiet cost and correctness regressions that surface from finance or a paged tail.

  • The surprise invoice. An agent ships with no cost telemetry; three weeks later finance asks why the bill 10×'d. The usual root cause is a ReAct loop re-reading a growing scratchpad (quadratic token growth), or the flagship on every trivial request, or no caching. Blast radius: the whole product's margin, and it is invisible until the invoice because nobody metered tokens. The guard is $/request and $/resolved-task on a dashboard before ship.
  • The semantic-cache false hit. "Cancel my order" is served the cached "cancel my subscription" answer because they are one token apart in embedding space. Blast radius: any high-stakes query routed through the semantic cache, and the failure is a wrong action, not a slow one. The guard is a threshold tuned on real traffic plus a bypass for high-stakes intents.
  • The p50 dashboard that lied. Mean 0.7 s, everyone relaxed; p99 6 s, and the biggest customer always hits a cold cache. Blast radius: your most important customers, who live in the tail the mean erased. The guard is designing SLOs at p95/p99.
  • Prefix-cache defeated by layout. Someone puts a per-request timestamp at the front of the prompt, invalidating the shared prefix on every call and silently doubling input cost. Blast radius: every request through that prompt template. The guard is a layout rule: stable content first, variable content last.

Cross-cutting concerns

Cost. Cost is engineered at three layers that compose: the token model (fewer in, fewer out), the routing mix (cheapest-that-works + cascade), and the cache hit rate (exact + semantic + prefix). The principal skill is naming which layer a given saving comes from and what it trades — caching trades staleness, routing trades quality headroom, cascades trade latency.

Latency. TTFT is the felt latency (why chat UIs stream); total latency is the SLA. Caching cuts both (a hit is instant); routing to a faster model cuts latency at some quality cost; the tail is where you actually design. These are not independent knobs — a cache hit improves cost and p95 simultaneously, which is why the hit rate is the highest-leverage number in the system.

Observability. Traces, metrics, and logs answer three different questions and belong in three stores. A span tree answers "where did the time and money go on this request"; metrics answer "what is the p99 and the $/request right now"; logs answer "what exactly did this step say." Trying to answer "why was this request slow" from a latency dashboard is the anti-pattern — the dashboard has the aggregate, the trace has the causal structure. OTel GenAI conventions are what make the three interoperate across LangSmith/Langfuse/Helicone/Datadog instead of siloing per vendor.

Multi-tenancy (with Phase 13). The cost meter is per tenant for a reason: chargeback and noisy-neighbor detection. A tenant within its request quota can still blow the model bill, so per-tenant $/month budgets are a distinct control from per-tenant rate limits — one bounds spend, the other bounds throughput.

The "looks wrong but is intentional" decisions

  • Cache before route. Checking the cache before deciding which model to use looks like premature optimization; it is the point — the cheapest request is the one you never make, so the cache must sit upstream of every downstream cost.
  • Two cache types instead of one. Running both exact and semantic looks redundant. They occupy different key spaces with opposite failure profiles (exact: never wrong, low hit rate; semantic: high hit rate, false-hit risk), so they cover disjoint slices of traffic.
  • Optimize $/resolved-task, not $/attempt. Dividing cost by success rate looks like it raises the reported cost; it makes it true. A cheap-but-failing model looks like a win on $/attempt and a loss on $/resolved-task, and the second number is the one margin is made of.
  • Degrade instead of fail. Serving a cached or canned answer under budget pressure looks like shipping a worse product; it is the mature choice — a slightly worse answer that ships beats a perfect one that times out or blows the budget.

Where this fits the platform decision

Every production agent platform — Cohere's "reliability, latency, cost, observability," Citi's "optimize cost, latency, prompts, caching, and vector indexes," the production-readiness bar in every role in jd.md — solves the same problem: make the money and the milliseconds visible and controllable. The principal move is to treat cost and latency as engineered outputs of routing, caching, metering, and tracing, instrumented from day one, and to defend the unit economics with $/resolved-task in a FinOps review — instead of discovering the number in a Slack from finance.