« Phase 14 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 14 — Deep Dive: The Mechanics of Cost, Latency & Traces

The load-bearing idea of this phase is that cost and latency are engineered, not observed — they are functions of concrete decisions (which model, cache or not, how many steps), and each decision is a data structure with an algorithm you can trace. This doc walks those structures — the router's selection, three cache mechanisms with different key spaces, the cascade's control flow, the span tree, and the cost accumulator — at the level someone implementing them has to reason about, because that is exactly what Lab 01 makes you build.

The cost model is one arithmetic identity

Every call bills cost = input_tokens/1e6 * price_in + output_tokens/1e6 * price_out, with price_out typically 3–5× price_in because generating is dearer than prefilling. CostMeter accumulates this per request, per model, and per tenant. Two levers fall straight out of the two terms: send fewer input tokens (caching, prefix reuse, context compression) and generate fewer output tokens (terse tool protocols, shorter reasoning). The asymmetry in price_out is why a cache hit — which pays zero of both terms — dominates any per-token micro-optimization: the cheapest request is the one you never make.

Routing: cheapest-that-meets-the-bar is a filter-then-min

Router holds a registry of ModelSpec(price, quality, latency). route(task) filters the registry to specs whose quality >= task.min_quality, then selects the minimum-price survivor — min(candidates, key=lambda m: m.price). That is the whole mechanism, and its correctness rests on one invariant: quality is a floor, price is the objective. You never pay for quality above the bar. Complexity is O(models), trivial, because the registry is small; the hard part is not the selection but classifying the task's difficulty cheaply to set min_quality — a small router model, a heuristic, or the cascade below. The naive alternative — the flagship on every request — fails economically: most workloads are majority-easy, so you pay flagship prices for a small model's job on 80% of traffic.

The cascade: routing over time, gated by a confidence signal

route_cascade(task, judge) is routing across attempts instead of across models-at-once. It runs the cheap model, applies an injected judge(answer) → confidence, and if confidence < threshold escalates to the next model up, repeating until acceptance or the top of the ladder. The control flow is a loop over an ascending-quality list with an early exit on the first accepted answer. The economics: if the cheap model clears the bar on 80% of requests, you pay flagship prices on only 20%. The costs are precise and worth naming — added latency on the escalated fraction (two sequential calls, so latencies sum) and dependence on a reliable confidence signal, which is where evaluation and calibration (Phase 11) re-enter. A miscalibrated judge either escalates too eagerly (you lose the savings) or accepts bad cheap answers (you lose quality). judge is injected so the lab is deterministic and the confidence model is a swappable seam.

Three caches, three key spaces, three failure modes

The caches are not one idea tuned three ways — they occupy three different key spaces, and the key space is the tradeoff.

Exact cache keys on the literal tuple (prompt, model, params). A hit returns the stored answer at ~0 cost and instant latency; the guarantee is zero false hits because byte-identical inputs demand identical outputs. Its weakness is the hit rate: LLM inputs rarely repeat byte-for-byte (a timestamp, a username, whitespace all miss), so exact caching earns its keep on truly repeated calls — a fixed system-prompt eval, a popular FAQ. It needs a TTL: each entry stores an insertion tick, and get treats now - inserted >= ttl as a miss (and evicts). The clock is injected, so expiry is deterministic — a wall clock would make the test non-reproducible.

Semantic cache trades the key space from bytes to meaning. It embeds the query with the stdlib hashing embedder and, on lookup, returns a cached answer iff cosine(query_vec, cached_vec) >= threshold. This lifts the hit rate dramatically on paraphrase-heavy traffic ("what time do you open" ≈ "when are you open"), at the cost of a genuinely new failure mode: the false hit. Two queries close in embedding space may want different answers — "cancel my order" vs "cancel my subscription" are one token apart and semantically adjacent but operationally opposite. The threshold is therefore a precision/recall dial you tune on real traffic, and high-stakes queries must bypass the semantic cache entirely. The mechanism worth internalizing: exact caching can never be wrong and is usually a miss; semantic caching is usually a hit and can be dangerously wrong — they are duals, and mature platforms run both (exact first, semantic second).

Prefix / KV cache keys on the shared leading prefix of the prompt. shared_prefix_tokens(a, b) counts the reusable leading overlap. The saving is real because of how transformer inference works: the attention keys/values for the prefix tokens are computed once during prefill and reused, so the provider charges a reduced input price on the cached portion and prefill latency drops. The design implication is a data-layout rule: put stable content (system prompt, tool schemas, few-shot) at the front and variable content at the back, because the cache reuses a prefix, not an arbitrary substring — a single early-varying byte invalidates everything after it.

The hashing embedder underneath the semantic cache

The semantic cache needs vectors, and the lab supplies a dependency-free one: feature hashing. Each token maps via a stable SHA-256 hash to a bucket h % dim, and a sign drawn from a second slice of the hash ((h // dim) % 2) is accumulated into that bucket. The sign is the non-obvious part — it makes two different tokens that collide into the same bucket cancel on average rather than always reinforce, which keeps the collision noise unbiased. Cosine over these vectors measures direction, ignoring magnitude, which is why it is the right similarity for the cache. It is not a learned embedding — its "semantics" are pure word overlap — but it is deterministic and offline, and it is the exact plug point where a real embedding model drops in.

Tracing: a span tree, not a log stream

Tracer builds a tree of Spans, each a unit of work with start, end, and attributes. One request produces a span for the llm-call nested inside the tool-call nested inside the retrieval nested inside the request — parent/child by containment, with durations and token/cost attributes on each node. This is structurally different from logs: logs are a flat, time-ordered stream that tells you what happened; a span tree tells you how the pieces relate and where the time went. Because agent steps are sequential, the parent's duration is (at least) the sum of its children's, so the tree localizes the slow step immediately — retrieval 40 ms, tool 1.2 s, llm 800 ms, and "oh, it retried twice." The attribute names follow OpenTelemetry's GenAI conventions (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) so the tree interoperates with real backends.

Why the mean lies, and the ladder that respects the tail

Latency has two numbers: TTFT (time to first token — the felt latency, which streaming optimizes) and total. The mechanism that trips people up is the tail: you design to p95/p99, not the mean, because agent loops are sequential so per-step latencies sum, and one cold cache miss or one slow step lands a user in the tail. A p50 dashboard shows 0.7 s and everyone is happy while the biggest customer, who always hits a cold cache, sits at 6 s. serve_under_budget(task, remaining_budget) is the degradation ladder that respects this: as the budget shrinks it steps best → cheaper → cached/approximate → canned, each rung cheaper and faster, none an error page. The control flow is a descending walk over rungs, choosing the first the remaining budget affords — the production instinct that a slightly worse answer that ships beats a perfect one that times out.

What to hold onto

Every control here moves one of two numbers, and the real metric ties them together: $/resolved-task = $/attempt ÷ success_rate. Routing, caching, and cascades move the numerator; evals move the denominator; the meter measures both. Cache by exactness and meaning and prefix because each cuts a different slice; trace as a tree because relationships live in the structure; and design to the tail because the mean is where slow requests go to hide.