« Phase 12 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 12 — Core Contributor Notes: Production Agent Services — Async, Streaming, State & Events

The lab is a stdlib miniature of a real stack: Starlette/FastAPI on uvicorn over ASGI, asyncio's primitives, Redis for state and pub/sub, the browser's EventSource, and Kubernetes lifecycle. This document is the maintainer's-eye view of what those systems actually do under the hood, the source-level decisions that shape their APIs, and exactly which of those the lab deliberately does not reproduce.

StreamingResponse is a loop over ASGI send

FastAPI is a thin layer over Starlette; Starlette is an ASGI application. An ASGI app is a coroutine app(scope, receive, send) where scope describes the connection, receive is an awaitable that yields client events, and send is an awaitable that pushes server events. An HTTP response is not a return value — it is a sequence of messages you send. First a http.response.start message carrying status and headers, then one or more http.response.body messages, each {"type": "http.response.body", "body": b"...", "more_body": True}, and a final chunk with more_body false (or omitted) that ends the response.

StreamingResponse(generator, media_type="text/event-stream") is, essentially, send the start message, then async for chunk in generator: await send({..., "body": chunk, "more_body": True}), then a terminal empty body. That is the real machinery under our format_sse plus stream_agent: the generator yields SSE-framed bytes, the response object relays each as one http.response.body. The lab collapses the ASGI server and the transport into a direct async for, but the shape — an async generator feeding a per-chunk send loop — is exactly Starlette's.

The non-obvious part is backpressure through await send. uvicorn writes chunks onto an asyncio transport that has a high-water mark. When a slow client (or a saturated network) lets the OS socket buffer fill, the transport pauses writing and await send(...) stops returning until the buffer drains. Because the generator only produces the next chunk after send returns, a slow consumer transparently slows the producer — flow control with no explicit signaling. This is why you should never eagerly buffer a whole response in a streaming handler: the point of the design is that memory stays bounded because the generator is throttled by the client. Our lab has no transport, so it has no client-driven backpressure — a real gap to name, because in production a slow reader is a legitimate way an agent stream ties up a worker for its full duration.

asyncio primitives: futures, a ready deque, and a selector

The default loop on Unix is a selector event loop wrapping epoll/kqueue; on Windows it is a proactor loop over IOCP. The core objects are small. A Future is a box with a state, a result slot, and a list of done-callbacks; completing it calls loop.call_soon on each callback, which appends to the loop's ready deque. A Task is a Future that drives a coroutine: each step runs the coroutine to its next await, which surfaces a Future the task subscribes to, so the task resumes when that Future completes.

Semaphore is _value: int plus _waiters, a deque of Futures; acquire fast-paths when _value > 0 else parks a Future, release increments and wakes the oldest waiter. Queue is a deque plus _getters and _putters (deques of Futures) and a maxsize; put on a full bounded queue parks a putter, get on an empty queue parks a getter — this is where a bounded asyncio.Queue gives you backpressure for free. Event is a bool plus _waiters; set wakes all waiters, and once set, wait() returns immediately — edge behavior that maps directly onto our drain: the idle event, once set by the last request, lets shutdown return without re-blocking. Our lab's TTLCache, EventBus, and AgentService are hand-rolled over these exact primitives, so understanding the primitives is understanding the lab.

A sharp edge worth internalizing: asyncio.gather is not structured concurrency. With gather, if one awaitable raises, the first exception propagates to the caller but the other tasks are not cancelled — they keep running, orphaned, and their exceptions may be logged as "never retrieved." asyncio.TaskGroup (3.11+) fixes this: on any child failure it cancels the siblings and raises an ExceptionGroup, giving you all-or-nothing semantics and no leaked tasks. In production agent code that fans out to multiple tools, gather can leave a failed fan-out with tools still executing side effects; prefer TaskGroup when a failure should abort the batch. The lab uses gather for its burst tests because it wants independent survivors, which is the legitimate case for it.

uvicorn lifespan and graceful shutdown

ASGI defines a lifespan scope separate from HTTP. The server sends lifespan.startup before serving and lifespan.shutdown when stopping; FastAPI's lifespan context manager (the modern replacement for the old startup/shutdown events) runs setup before the yield and teardown after. This is where you open and close your Redis pool, and where you would flip a readiness flag.

On SIGTERM/SIGINT, uvicorn's signal handler sets an internal should_exit. The server then stops accepting new connections, and its shutdown drains outstanding requests up to --timeout-graceful-shutdown seconds before force-closing remaining connections and firing lifespan shutdown. That timeout is exactly our drain deadline. The full production sequence layers Kubernetes on top: a preStop hook plus the pod's terminationGracePeriodSeconds give the load balancer time to observe the failing readiness probe and stop routing before SIGTERM lands, so in-flight requests finish and no new ones arrive. Our shutdown() models the middle of that: set _shutting_down (readiness now reports draining and new work gets a 503), then await an idle event the last request sets. What the lab omits is the deadline — real drains bound the wait so a stuck request cannot block a deploy forever, and the orchestrator ultimately SIGKILLs past the grace period.

Redis pub/sub is fire-and-forget; know when you need Streams

Our EventBus mirrors Redis pub/sub precisely in semantics, including its limitation. Redis PUBLISH/SUBSCRIBE is at-most-once and non-persistent: a message is delivered only to clients subscribed at that instant; there is no buffering for absent subscribers, no acknowledgment, no replay. A subscriber that disconnects and reconnects has a gap it can never recover. Our in-memory bus has the same shape — events published before a subscriber subscribes are gone, and everything is lost on process restart. That is fine for ephemeral fan-out (live metrics, a logger tailing events) and wrong for anything that must not be missed.

When you need durability, delivery guarantees, or replay, the real system reaches for Redis Streams (XADD/XREAD, consumer groups with XACK for at-least-once and load-balanced consumption) or a real broker — Kafka (partitioned, retained log, offset-based replay), NATS JetStream, RabbitMQ. The distinction the lab is teaching is exactly the pub/sub-vs-work-queue split: per-subscriber fan-out (everyone gets every event) versus competing consumers (each message to one worker). Redis Streams consumer groups are the competing-consumers pattern; plain pub/sub is the fan-out pattern. Picking the wrong one either duplicates or drops work.

SSE, EventSource, and the proxy that eats your stream

The browser's EventSource is a built-in SSE client, and its ergonomics shape the wire format. It auto-reconnects on a dropped connection; on reconnect it sends the last id: value it saw back as a Last-Event-ID request header, so a server that stamps id: on events can resume from the client's position — the SSE-native answer to "don't replay tokens the client already received." The retry: field lets the server set the reconnect backoff. EventSource dispatches by the event: name to listeners the page registered, which is why naming a terminal done event matters: the client can distinguish a clean end from a drop and decide whether to reconnect. Our format_sse produces exactly the bytes EventSource parses, minus id:/retry:, which the lab leaves as a noted extension.

The sharpest production gotcha is not in the format — it is the proxy in the middle. Reverse proxies and CDNs buffer responses by default (nginx proxy_buffering on), which defeats streaming entirely: the proxy holds your chunks and delivers the whole response at once, so TTFT collapses back to total time and the "streaming" endpoint feels buffered. The fixes are proxy-side (proxy_buffering off) or the X-Accel-Buffering: no response header that tells nginx not to buffer this response; you also disable response compression buffering on the stream. Every team that ships SSE learns this once, usually in production, because it works perfectly in local dev where there is no proxy.

What the miniature deliberately simplifies

Naming the simplifications is the maintainer's discipline. No network or ASGI — direct coroutine calls stand in for uvicorn and the transport, so there is no client-driven backpressure and no real socket lifecycle. The idempotency store is a plain map check-then-set, not atomic — real dedupe needs SET key val NX EX ttl (or a Lua script) so truly-concurrent duplicates cannot both miss; the lab implements sequential replay and documents the race. The semaphore is per-process — a distributed cap across many workers needs a shared limiter (a Redis token bucket, or per-worker shares of a global quota). The event bus is in-process — Redis/Kafka add the network, persistence, and delivery semantics. The clock is injected — production reads a monotonic clock, and TTLs live in Redis with server-side expiry. Every one of these is a correct simplification for a deterministic, network-free test, and every one is a real thing you add back when the miniature meets production. Knowing precisely which line moves is the difference between "I built the toy" and "I understand the system the toy models."