« Phase 14 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 14 — Hitchhiker's Guide
30-second mental model
Cost and latency are engineered: route the easy majority to cheap models, cascade the
hard minority to the flagship, cache exact + semantic + prefix, meter $/resolved-task,
and trace everything with OpenTelemetry GenAI spans. Under pressure, degrade gracefully
(best → cheaper → cached → canned) instead of failing. Instrument from day one or find out from the
invoice.
The facts to tattoo on your arm
| Fact | Why |
|---|---|
| output tokens 3–5× input price | generating is dearer than reading |
| route to cheapest-meeting-quality | most requests are easy |
| cascade = cheap → check → escalate | pay flagship only on the hard minority |
| exact cache: rare hits, zero false | inputs vary |
| semantic cache: high hits, false-hit risk | tune the cosine threshold |
| prefix cache: stable content first | reuse the KV cache → cheaper + faster prefill |
metric = $/resolved-task | cost ÷ eval success rate |
| design to p95/p99 | the mean lies; loops sum latencies |
| traces > logs | spans show how the pieces relate |
Framework one-liners
- RouteLLM / FrugalGPT — cascade/route by difficulty for cost.
- GPTCache — semantic caching.
- Anthropic/OpenAI prompt caching — reduced input price on a shared prefix (KV reuse).
- OpenTelemetry GenAI — standard span attributes (
gen_ai.usage.input_tokens, …). - LangSmith / Langfuse / Helicone — LLM tracing + cost dashboards.
War stories
- The 10× invoice. A ReAct loop re-read its scratchpad (quadratic); nobody metered tokens. One afternoon's fix once someone looked.
- The semantic-cache false hit. "cancel my order" served the cached "cancel my subscription" answer. High-stakes queries must bypass the semantic cache.
- The p50 dashboard that lied. Mean 0.7 s, everyone happy; p99 6 s, the biggest customer always hit a cold cache. SLOs live at the tail.
Vocabulary
routing / cascade · exact / semantic / prefix (KV) cache · TTL · cost meter /
$/resolved-task · span / trace / OTel GenAI · TTFT · p95/p99 tail · degradation
ladder.
Beginner mistakes
- Using the flagship model for every request.
- Treating LLM caching as a trivial dict (no TTL, no false-hit handling).
- Comparing agents on
$/attemptinstead of$/resolved-task. - Quoting mean latency instead of p95/p99.
- Putting variable content before stable content (kills prefix caching).
- Logs-only observability with no traces.