« Phase 14 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 14 — Core Contributor Notes: How the Real Systems Are Built

This is the maintainer's-eye view of the systems our miniature imitates — OpenTelemetry's GenAI conventions, provider prompt caching and the KV cache underneath it, GPTCache, the RouteLLM/FrugalGPT line of work, and the tracing backends. The interesting engineering is in the seams. Where I describe a pattern rather than a verified internal, I say so, and I do not invent exact numbers or URLs.

Prompt caching: the KV cache is the real object

The lab's shared_prefix_tokens counts a reusable prefix; the reason that saves money is a real server-side data structure. During transformer inference, each token's attention keys and values are computed per layer; autoregressive decoding caches them (the "KV cache") so each new token attends to stored K/V instead of recomputing the whole sequence. Prefix caching extends this across requests: if two prompts share a leading prefix, the K/V for that prefix is identical and can be reused, skipping prefill for those tokens. That is why the saving is both cost (fewer prefill FLOPs billed) and latency (less time to first token).

The two big providers expose this with a revealing design difference:

• Anthropic makes it explicit: you mark cache breakpoints (cache_control) on the parts of the prompt you expect to reuse — typically the system prompt, tool schemas, and large shared context. Reads on a cache hit are billed at a small fraction of the base input price (on the order of a tenth), while the write that populates the cache costs a premium over base input. There is a minimum cacheable prefix length (on the order of a thousand tokens) and a short idle TTL (a few minutes) that refreshes on use. The committer's takeaway: you are choosing what to cache, so prompt layout is your responsibility.
• OpenAI makes it automatic: prefixes above roughly a thousand tokens are cached without any code change, matched by leading prefix, with a discount on the cached input tokens. No breakpoints, less control, zero effort.

Both enforce the same physics our lab teaches: caching reuses a prefix, so a single early-varying byte (a timestamp, a request id at the front) invalidates everything after it. Put stable content first. The server-side implementations that make this fast — vLLM's PagedAttention with prefix caching, SGLang's RadixAttention (a radix tree over KV blocks) — are the real machinery behind the provider's billing line; our lab models only the token-overlap accounting, not the block allocator.

Semantic caching: GPTCache's architecture is the reference

Our SemanticCache is a hashing embedder + cosine + a threshold. GPTCache is the productionized version, and its architecture names the seams we collapsed:

• A pluggable embedding function (OpenAI embeddings, a local sentence-transformer, ONNX) — our hashing embedder is the offline stand-in for this slot.
• A vector store (FAISS, Milvus, and others) for the cached query embeddings, because at scale you cannot linear-scan every cached vector — you need an ANN index, which our O(n) cosine loop ignores.
• A similarity evaluation step that decides hit/miss, plus eviction (LRU/FIFO) and a TTL. The threshold is the precision/recall dial, and the sharp edge every GPTCache user learns is the false hit: two queries close in embedding space that want different answers. The mitigation in production is the same as the lab's lesson — tune the threshold on real traffic and route high-stakes intents around the semantic cache. Our miniature keeps the mechanism (embed, cosine, threshold) and drops the ANN index and eviction policy.

Routing and cascades: RouteLLM and FrugalGPT

The lab's Router (cheapest-that-meets-quality) and route_cascade (escalate on low confidence) are the teaching-sized versions of a real research line:

• FrugalGPT (Chen et al., 2023) is the cascade generalized: an LLM cascade tries models in ascending cost, a learned scoring function judges whether the current answer is good enough, and it stops at the first acceptable answer — plus prompt adaptation and query concatenation to cut tokens. Our injected judge(answer) → confidence is precisely FrugalGPT's scorer, abstracted so the lab stays deterministic. The paper's result is the phase's thesis: you can hit flagship-level quality at a fraction of the cost by paying the flagship only on the hard minority.
• RouteLLM (Ong et al., 2024) is routing-at-once rather than over-time: a trained router decides, before calling, whether a query needs the strong or the weak model, learned from preference data (with approaches like a BERT classifier or matrix factorization over model-vs-query strength). The seam our lab hides is exactly this: we set min_quality by hand; the hard, valuable part in production is classifying difficulty cheaply and accurately, because a router that misjudges either overspends or underdelivers. RouteLLM is what that classifier looks like when it is learned rather than heuristic.

Observability: OTel GenAI conventions and the proxy-vs-SDK split

The lab's Tracer builds a nested Span tree with GenAI-style attributes. The real standard is OpenTelemetry's GenAI semantic conventions — agreed attribute names like gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.operation.name, and a gen_ai span/event shape. These conventions are still evolving (parts are marked experimental), which is worth knowing: pinning to them buys interoperability but the exact attribute set moves. The value is that a trace emitted with these names is readable by any conformant backend rather than locked to one vendor's SDK.

The backends split along an instrumentation-architecture line a committer should recognize:

• LangSmith and Langfuse are SDK-instrumented: you wrap your calls (or use their auto-instrumentation), and spans are emitted from inside your process. More detail, more code coupling.
• Helicone is (canonically) a proxy: you point your base URL at it and it observes requests passing through, so you get cost/latency telemetry with a one-line change and no in-code spans — at the cost of seeing only what crosses the proxy boundary (it doesn't see your in-process tool calls as nested spans unless you also instrument them).

That proxy-vs-SDK tradeoff — effortless-but-shallow vs detailed-but-coupled — is the real design decision behind "how do we get traces," and our in-process Tracer models the SDK side.

The $/resolved-task metric is not standard, and that is the point

No provider bills you in $/resolved-task — they bill tokens. The metric $/attempt ÷ success_rate is a composed number you build by joining the cost meter (this phase) with the eval success rate (Phase 11). The reason it matters to a maintainer is that every vendor dashboard reports $/request or tokens, which flatters a cheap-but-failing model; the true unit cost only appears when you divide by the fraction of attempts that actually resolved the task. Building that join is a platform responsibility, not a feature you can buy.

What our miniature deliberately simplifies

• Token-overlap prefix accounting instead of a real KV-block allocator — we count reusable leading tokens; real systems (vLLM PagedAttention, SGLang RadixAttention) manage KV blocks and share them across requests.
• O(n) cosine over a hashing embedder instead of an ANN index over learned embeddings — GPTCache uses FAISS/Milvus and a real embedding model; our embedder's "semantics" are word overlap.
• An injected judge instead of a trained scorer/router — FrugalGPT learns the scorer, RouteLLM learns the router; we abstract both behind a deterministic function.
• An in-process span tree instead of an OTel exporter to a backend — same tree shape, none of the sampling, batching, or wire protocol.
• An injected integer clock instead of wall time — deterministic TTL expiry and durations; real caches evict on real elapsed time and real backends timestamp with wall clocks.

Know the seams — the KV cache under prompt caching, the ANN index and false-hit dial in GPTCache, the learned scorer/router in FrugalGPT/RouteLLM, and the proxy-vs-SDK split in tracing — and the real systems' docs read as confirmation of the shapes this lab already made you build.