« Phase 14 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 14 — Staff Engineer Notes: Owning Cost, Latency & Observability

Anyone can call a model and read the latency off a dashboard. The gap between using an LLM and being trusted to own a platform's cost and latency is judgment about things nobody hands you a default for: which model each request runs on, what you cache and where the false hits will bite, what you actually measure, and — the one that gets escalated to you — why the bill 10×'d and what you do about it without tanking quality. This doc is about that judgment and the signal that proves you have it.

The decisions a staff engineer actually owns here

Instrumentation is a day-one decision, not a day-ninety one. The single most avoidable incident in this space is the surprise invoice, and it is always the same story: someone shipped, felt great, and finance sent a Slack three weeks later. You own the rule that tokens-per-request and $/resolved-task are on a dashboard before the agent ships, and that every request emits a trace. "I had cost telemetry from day one" is a sentence you get to say in a review instead of a postmortem.

The metric is $/resolved-task, and defending it is your job. The junior optimization is $/attempt — cheaper model, fewer tokens — which quietly ships a worse model that tanks the success rate and the margin. You own dividing cost by the eval success rate (Phase 11) so a cheap-but-failing model shows up as the loss it is. This is the move that connects the cost conversation to the quality conversation, which is what a real FinOps-for-AI discussion is.

The caching strategy is a correctness decision, not just a cost one. Exact caching is safe and low-hit; semantic caching is high-hit and can be wrong. You own the threshold, tuned on real traffic, and the policy that high-stakes intents ("cancel my order," anything that triggers an action) bypass the semantic cache. A false hit is a wrong answer served confidently, and owning that boundary is the difference between a cost win and an incident.

The routing/cascade mix is a cost/latency/quality tradeoff you set. You own the min_quality bars, the cascade threshold, and the confidence signal behind it — and the knowledge that a cascade buys cost savings at the price of added latency on the escalated fraction. Miscalibrate the judge and you either lose the savings or ship bad cheap answers.

The decision framework, out loud

  • Traffic is majority-easy and paraphrase-heavy (support, search) → route the easy majority to a cheap model, run a semantic cache with a tuned threshold, cascade the hard minority. Biggest wins, biggest false-hit risk to manage.
  • Traffic is high-variance and high-stakes (each request unique, wrong answers costly) → exact cache only (or none), no semantic cache on the risky intents, and spend on the model that clears the bar rather than cascading.
  • Prompts share large stable prefixes (system + tools + few-shot on every call) → lay out stable content first and lean on prompt-prefix caching; this cuts input cost and TTFT for free.
  • Budget or load is spiking → a degradation ladder (best → cheaper → cached → canned) so you shed cost gracefully instead of erroring — a slightly worse answer that ships beats a perfect one that times out.

Naming the tradeoff each lever makes is the signal. "Just use the best model" and "just add a cache" are the anti-signals.

Code-review red flags

  • The flagship model hardcoded on every request. A margin-killer at scale. Ask for the routing bar and the cheap-model fraction.
  • A semantic cache with no threshold discipline and no high-stakes bypass. A false hit waiting to serve the wrong answer. Demand the threshold's provenance (tuned on what traffic?) and the bypass list.
  • $/attempt presented as the cost metric. Ask for $/resolved-task — cost divided by eval success rate — or the "saving" may be a quality regression in disguise.
  • Mean latency quoted as the SLO. Users live at p95/p99; agent loops sum latencies. Design to the tail or the biggest customer lives in it.
  • Variable content (a timestamp, a request id) at the front of the prompt. It defeats prefix caching on every call and silently doubles input cost. Stable content first.
  • A cache with no TTL, or treated as a plain dict. LLM caching needs expiry and false-hit handling; it is subtler than a dict.
  • Logs-only observability with no traces. Logs say what happened; only a span tree says how the pieces related and where the time went. At 2 a.m. you want the tree.
  • README.md chapter links, an untraced tool call, or a ReAct loop re-reading a growing scratchpad (quadratic tokens) — small tells the author hasn't internalized the cost/latency model.

War stories worth carrying

  • The 10× invoice. A ReAct loop re-read its scratchpad every step (quadratic token growth) and nobody metered tokens. The fix was one afternoon — once someone looked — but it ran for weeks because there was no cost telemetry. Instrument on commit one.
  • The semantic-cache false hit. "Cancel my order" was served the cached "cancel my subscription" answer because the two are one token apart in embedding space. High-stakes queries must bypass the semantic cache; the threshold alone is not enough.
  • The p50 dashboard that lied. Mean 0.7 s, everyone happy; p99 6 s, and the biggest customer always hit a cold cache. SLOs live at the tail, not the mean.
  • The cheap-model "saving" that lost money. A cost cut swapped in a smaller model that dropped the success rate; $/attempt fell but $/resolved-task rose because more attempts failed. The number that told the truth was the one divided by the eval.

The interview / architecture-review signal

What a reviewer listens for: do you treat cost and latency as engineered outputs, and do you know the real unit cost? The winning move is specific and unprompted — route the easy majority to a cheap model, cascade the hard minority, cache by exactness and meaning and prefix, meter $/resolved-task per tenant, trace with OTel GenAI spans, and degrade gracefully under pressure. Then the depth: explain why the semantic cache can be wrong, why you design to p99 not the mean, why stable content goes first in the prompt, and why a cheap model that fails is not cheap. Candidates who demo an agent are common; the person who talks about what it costs to run one at 100k/day — and has been in the room when finance asked why the bill 10×'d — is who gets handed "make this production-ready."

Closing takeaways

  1. Instrument on commit one. Tokens-per-request and $/resolved-task on a dashboard before ship; the alternative is finding out from finance.
  2. Optimize $/resolved-task, not $/attempt. A cheap model that fails is more expensive per real result than a pricier one that works.
  3. Cache by exactness, meaning, and prefix — and own the false hit. Each cuts a different slice; the semantic cache is the one that can be wrong, so tune it and bypass high-stakes intents.
  4. Design to the tail. p95/p99, not the mean; agent loops sum latencies and slow requests hide in the mean.
  5. Degrade, don't fail. A ladder from best to canned sheds cost gracefully; a slightly worse answer that ships beats a perfect one that times out.