Lab 01 — Agent Gateway: Router, Cache, Cost Meter & Tracer

Phase 14 · Lab 01 · Phase README · Warmup

The problem

At scale, an agent's cost and latency are dominated by choices in code: which model, cached or not, how you traced and metered it. Build the gateway that makes those choices — a model router + cascade, exact/semantic/prefix caching, a cost meter, an OpenTelemetry-style tracer, and a degradation ladder — so cost and latency become engineered, and observable, instead of a surprise on the invoice.

What you build

PieceWhat it does
Router / route_cascadepick the cheapest model meeting the quality bar; escalate on a low-confidence judge
ExactCache / SemanticCacheliteral-prompt cache (TTL) + meaning-based cache (cosine threshold)
shared_prefix_tokenscount reusable leading tokens (prompt-prefix / KV caching saving)
CostMeter / CostReportaccumulate tokens + dollars per request/model/tenant; $/request
Tracer / Spannested OTel-style spans with durations from an injected clock
serve_under_budgetthe degradation ladder (best → cheaper → cached → canned)
Gatewaywires cache → route/cascade → meter → trace

File map

FileRole
lab.pyyour implementation (fill the # TODOs)
solution.pyreference + a main() (cache hit, cascade escalation, cost report, trace tree, ladder)
test_lab.py37 tests: routing, cascade, caches, prefix, meter, spans, degradation, determinism
requirements.txtpytest only

Run it

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

  • Router picks the cheapest model meeting min_quality; the cascade escalates only on low confidence.
  • Exact cache hits/misses/expires by TTL; the semantic cache hits above threshold and misses below.
  • The cost meter accumulates per model/tenant and yields $/request; you can extend it to $/resolved-task.
  • The tracer builds a nested span tree with correct durations from the injected clock.
  • The degradation ladder steps down as the budget shrinks.
  • All 37 tests pass under both lab and solution.

How this maps to the real stack

  • Routing/cascade ≈ RouteLLM / FrugalGPT and an LLM gateway's model-routing policy.
  • Exact/semantic caching ≈ a Redis exact cache + GPTCache; prefix caching ≈ Anthropic/OpenAI prompt caching (KV reuse) — put stable content first to maximize it.
  • The tracer ≈ OpenTelemetry with the GenAI semantic conventions (gen_ai.usage.*), exported to LangSmith / Langfuse / Helicone / Datadog.
  • The cost meter ≈ a FinOps/chargeback pipeline; join with Phase 11's eval success rate for $/resolved-task.

Limits. Real routing needs a difficulty classifier; real semantic caching needs a tuned threshold and eviction; real tracing exports over OTLP to a backend. The mechanisms are these.

Extensions (your own machine)

  • Add a difficulty classifier in front of the router (a cheap model or heuristic).
  • Export the spans as real OTLP and view them in a tracing UI.
  • Compute $/resolved-task by joining the meter with a Phase 11 eval run over a golden set.

Interview / resume signal

"Built an agent gateway that engineers cost and latency — model routing and confidence-based cascades, exact/semantic/prompt-prefix caching, per-tenant cost metering toward $/resolved-task, OpenTelemetry-style tracing, and a graceful degradation ladder — the production-readiness layer for an LLM platform."