« Phase 12 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 12 — Principal Deep Dive: Production Agent Services — Async, Streaming, State & Events

The mechanisms are the easy part. The hard part is the topology they sit inside, the capacity math that sizes it, and the failure modes that decide whether a Tuesday deploy is invisible or an incident. This document is the architecture view: where the boxes go, how many of each you need, what breaks first, and which of your "looks wrong" decisions are actually load-bearing.

The reference topology

A production agent service is five tiers, and the lab is the middle one with the network amputated:

clients ──► load balancer / ingress ──► N ASGI worker processes ──► external state ──► upstreams
(browser,   (TLS, routing,             (uvicorn, one event loop    (Redis: sessions,   (LLM API,
 mobile,     health-based              per core; FastAPI app;       cache, idempotency,  tools, DBs,
 partners)   routing, sticky?)         semaphore, streaming)        pub/sub)             vector store)

Every design choice in the phase is about keeping a request cheap while it is in a worker and durable when it is out of one. Streaming keeps first-byte latency low. The semaphore keeps a worker from stampeding the upstream. External state keeps the worker disposable so the load balancer can route freely and the deploy can recycle instances. The event bus lets the platform do work that isn't request/response without coupling producers to consumers. You are not building an app; you are building the part of a distributed system that happens to have an LLM at the far edge.

Capacity: the only math that matters at review time

Little's Law, L = λW, sizes the whole thing. L is concurrent requests in the system, λ is arrival rate, W is average time-in-system. An agent request is I/O-bound and long: assume W = 8 s. At λ = 100 req/s, L = 800 — eight hundred requests must be simultaneously in flight. In a thread-per-request model that is 800 threads (~800 MB of stack, brutal context-switch overhead); on an event loop it is 800 coroutines on one thread, a few MB. That single comparison is why async is the architecture, not an optimization.

Now size the fleet. One event loop saturates roughly one core, so a box with C cores runs C uvicorn workers (Gunicorn or --workers C). Async scales concurrency within a core; workers scale across cores. If one loop comfortably holds ~500 in-flight I/O-bound coroutines, an 8-core box holds ~4000, and L = 800 fits on two cores with headroom. The ceiling is almost never CPU or threads — it is the upstream. Your LLM has a requests-per-minute and tokens-per-minute quota. If the provider allows 600 RPM (10 concurrent 8-second calls sustained) then your global useful concurrency to that model is ~10, no matter how many coroutines you can hold. That is the number the semaphore must encode. Sizing the semaphore to your loop's capacity instead of your quota is the classic mistake: you accept 800 concurrent calls, the provider returns 429s, retries pile on, and the loop drowns in work that will never succeed. The cap belongs at the scarcest downstream resource, and per worker it is quota_concurrency / worker_count (minus headroom for other workers and other callers).

Latency has two components that streaming decouples. Total generation time W is fixed by the model. Time-to-first-token is a separate, much smaller number. Buffering couples them — perceived latency becomes W. Streaming breaks the coupling — perceived latency becomes TTFT (~300 ms), and W becomes throughput you feel rather than a wall you stare at. This is why streaming is a latency-architecture decision, not a front-end nicety, and why the wire discipline (a terminal done event) earns its keep: it is how the client distinguishes "finished cleanly" from "connection dropped," which decides whether a retry is safe.

Failure modes and blast radius

Reason about each mechanism by what happens when it is absent or wrong, and how far the damage spreads.

  • Blocking call in an async path. Blast radius: the entire worker, every concurrent request on it, not just the offending one. One synchronous requests.get or time.sleep freezes the loop for its full duration; p99 spikes across hundreds of unrelated requests; health checks may miss it because the loop still answers between stalls. This is the single highest-leverage bug in the phase — one line, whole-worker impact — which is why "never block the loop" is the cardinal rule and why CPU-heavy work must go to a thread/process pool.
  • Unbounded wait queue. Blast radius: the worker, via OOM. Without a max_queue bound, a spike parks unlimited coroutines on the semaphore; each holds request state and an open connection; memory climbs until the OOM killer takes the whole worker, dropping every in-flight request including the ones that were about to succeed. A bounded queue plus a fast 503 converts an outage into graceful degradation — you shed the marginal request to save the ones already admitted.
  • Ungraceful shutdown. Blast radius: every in-flight stream on the recycled instance, on every deploy. Exit immediately on SIGTERM and you sever half-streamed answers mid-sentence and abandon side-effecting tool calls partway. Since deploys are routine, this is a continuous low-grade reliability tax that users experience as "it cut off again." Graceful drain (flip readiness to draining, stop admitting, await in-flight to a deadline, then exit) makes the deploy invisible.
  • In-process state. Blast radius: correctness, intermittently and unreproducibly. A local dict works in dev and on a single instance, then "randomly forgets" in prod because the load balancer routed the next turn elsewhere. The bug is inherent to horizontal scale, not a race you can patch.
  • No idempotency key. Blast radius: money and trust. A retried side-effecting request runs twice — double-charge, double-email, double agent run at $2 a pop. No downstream transaction prevents it because the duplication happens above the database.

The "looks wrong but is intentional" decisions

Four choices routinely draw a "why would you do that?" in review and are all correct.

Shedding load with a fast 503. It looks like giving up. It is the opposite: a service that returns 503 in a millisecond and stays up beats one that accepts everything and collapses, taking admitted requests down with it. Load-shedding is a reliability feature; the 503 is you defending a limit you chose rather than letting the upstream choose it for you with 429s. Pair it with a Retry-After and the client backs off instead of hammering.

Injecting the clock. Reading time.time() for TTLs looks simpler. But a service that reads the wall clock is neither deterministically testable nor replay-safe: you cannot prove "expires at exactly tick 5" without sleeping, and sleeping tests are slow and flaky. Injecting an integer now() makes expiry exact and the test a synchronous assertion. The same discipline carried the durability work in Phase 08 — time is an input, not an ambient syscall.

Per-subscriber queues in the event bus. Allocating a separate queue per subscriber looks wasteful next to one shared queue. It is the difference between fan-out and steal: a shared queue is a competing-consumers work queue where the first reader removes the event, so a shared queue silently drops the event to every other subscriber. The "waste" is the semantics.

Idempotency check outside the semaphore. Checking the cache before acquiring a permit looks like a layering violation. It is intentional: a replay must not consume scarce upstream concurrency. The cheap path (cache hit) must not queue behind the expensive path.

Cross-cutting concerns a principal owns

Cost of idle-holding connections. Streaming means a connection stays open for the full W seconds, mostly idle. On serverless with per-request billing that is expensive; this is exactly the problem Vercel Fluid compute (and Lambda's concurrency model, Cloud Run) targets — one instance multiplexes many concurrent invocations because the work is I/O-bound, so idle wait time is shared rather than billed per lonely request. Sizing here is "concurrent invocations per instance," and it is the same Little's Law number.

Observability. You cannot operate what you cannot see. The metrics that matter are the ones the lab exposes: in_flight (are we near the cap?), queued (is backpressure building?), rejected (are we shedding, and how much?), max_in_flight (did the cap ever bind?), plus TTFT and drain duration. These are the SLIs; alerts on rising queued and rejected catch a spike before it is an outage. An agent service without in_flight/queued gauges is flying blind into its own saturation.

Multi-tenant fairness (the Phase 13 seam). A single global semaphore is fair per worker but tenant-blind: one abusive tenant can consume the entire concurrency budget and starve everyone. The mechanism here is the foundation; the next layer is per-tenant quotas — a semaphore or token bucket keyed by tenant, so one tenant's spike sheds its own load, not the platform's. Phase 13 is where the global cap becomes a fairness scheduler. Recognizing that this cap is single-pool — and saying so before someone asks — is the principal-level tell.