« Phase 12 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 12 Warmup — Production Agent Services: Async, Streaming, State & Events

Who this is for: you can write Python and you've built an agent loop (Phases 00–01), but you've only ever run it in a script or a notebook. You have never had to serve it to a thousand users at once, stream its tokens to a browser, dedupe a retried request, or drain it cleanly on a deploy. By the end you will have built the systems layer that turns "an agent that runs" into "an agent service" — the layer Citi and Docker's JDs call FastAPI, asyncio, event-driven design, state, cache, and pub/sub. No framework, no network, no API key: asyncio is standard library, the model is an injected async generator, and time is an injected integer tick.

What a production agent service actually is
Why agents are I/O-bound — and why that means async
The event loop from first principles
Coroutines, await, and the async generator
Streaming: SSE vs WebSocket vs chunked, and why TTFT matters
The SSE wire format, precisely
Concurrency limits and backpressure: the semaphore and the queue
Idempotency: deduping retries and double-submits
State and cache: stateless services need external state
Event-driven architecture and pub/sub
Health, readiness, graceful drain, and deployment
Common misconceptions
Lab walkthrough
Success criteria
Interview Q&A
References

1. What a production agent service actually is

In Phases 00–11 you built the agent: a loop over a model, with tools, validation, retrieval, memory, durability, sandboxing, guardrails, and evals. In all of those, the agent ran in one process handling one task. That is not how anyone uses it. In production the agent sits behind an HTTP endpoint, and a service wraps it:

   many clients                       ONE process (few threads)
   ┌──────────┐   POST /agent/stream   ┌─────────────────────────────────────┐
   │ browser  │ ─────────────────────► │  async framework (FastAPI/uvicorn)  │
   │ mobile   │   (streams back SSE)   │   ├─ concurrency cap (semaphore)     │
   │ another  │ ◄───────────────────── │   ├─ idempotency (dedupe retries)    │
   │  service │                        │   ├─ state/cache (Redis, TTL)        │
   └──────────┘                        │   ├─ event bus (pub/sub, webhooks)   │
                                       │   └─ health + graceful drain         │
                                       └──────────────────────┬──────────────┘
                                                              ▼  (I/O-bound waits)
                                                        LLM API · tools · DB

Everything new in this phase is a distributed-systems concern that happens to have an LLM inside it. That framing — from the track overview, "agents are distributed systems" — is the whole phase: retries, idempotency, backpressure, tenancy (Phase 13), tracing (Phase 14) are the parts that page you at 2 a.m., and the async/streaming/state/events machinery is how you build a service that survives real traffic. The good news, if you have a backend background, is that you already know most of this; we are just wiring it around an agent.

The single most important reframing: an agent request is a long, I/O-bound network call, not a CPU computation. That one fact justifies every design decision below.

2. Why agents are I/O-bound — and why that means async

Take an agent request apart and time it. A ReAct step (Phase 01) is: send a prompt to the LLM, wait for it to generate tokens, run a tool (an HTTP call to search / a database query), wait for the result, repeat. Of the wall-clock time, the overwhelming majority is waiting on something else's network — the LLM provider, a search API, a database. Your CPU is idle the entire time. A single agent request might take 8 seconds and use 20 milliseconds of your CPU.

This is the definition of an I/O-bound workload, and it is the exact opposite of a CPU-bound one (matrix multiply, video encode) where the CPU is the bottleneck. The distinction decides your concurrency model:

CPU-bound → you need parallelism: more cores actually doing work at once (multiprocessing, threads that release the GIL, a GPU). Adding concurrency without cores buys nothing.
I/O-bound → you need concurrency: a way to have thousands of requests waiting at the same time on one core, because none of them is using the CPU while they wait. This is what async gives you.

Here is the punchline. Suppose each request waits 8 seconds on I/O. A synchronous, thread-per- request server that wants to handle 1,000 concurrent requests needs ~1,000 threads, each sleeping on a socket, each costing ~1 MB of stack and a scheduler slot — memory and context-switch overhead for threads that do nothing but wait. An async server runs those 1,000 requests as 1,000 cheap coroutines on one thread, and while each awaits its socket the single event loop runs the others. Same one core, ~1,000× the concurrency, a fraction of the memory. That is why every serious LLM service — FastAPI, aiohttp, Node, Go's goroutines — is built on an async or lightweight-thread model. Agents are the most I/O-bound workload in software (they wait on the slowest thing in the stack, a generating LLM), so async is not an optimization here; it is the architecture.

Little's Law makes the capacity concrete. In steady state the average number of requests in the system is

$$L = \lambda W,$$

arrival rate $\lambda$ times average time-in-system $W$. At $\lambda = 100$ requests/second and $W = 8$ seconds, $L = 800$ requests must be in flight simultaneously. A blocking model needs 800 threads for that; an async model needs 800 tiny coroutines on one loop. This is the number you put on the whiteboard when someone asks "how many concurrent agents can one instance hold."

3. The event loop from first principles

"Async" sounds like magic; it is not. An event loop is a plain while True: over a queue of ready callbacks:

loop:
    run every callback that is READY now (until it hits an await and yields control)
    ask the OS: which of my sockets/timers became ready? (select/epoll/kqueue)
    schedule the callbacks waiting on those, mark them READY
    repeat

That's it. There is one thread. A coroutine runs until it hits an await on something not yet ready (a socket with no bytes, a timer not yet elapsed) — at that point it yields control back to the loop, which runs another ready coroutine. When the awaited thing becomes ready, the loop resumes the parked coroutine right where it left off. No coroutine is ever interrupted; it only pauses voluntarily at an await. This is cooperative scheduling, and it has two consequences you must internalize:

Concurrency is not parallelism. On one loop, exactly one coroutine executes Python bytecode at any instant. You get the illusion of many things at once because they take turns at every await, and the turns are fast because everyone is mostly waiting on I/O. Two agents "running concurrently" are really taking turns while each waits on its LLM. There is no parallel CPU work unless you add processes.
One blocking call freezes everything. If a coroutine runs time.sleep(5) or a synchronous requests.get(...) or a heavy CPU loop, it never yields, so the entire loop—every other request—stalls for those 5 seconds. This is the #1 async bug: a single blocking call in a hot path tanks the whole service's latency. The rule is: in an async service, never block the loop; use await versions of everything, and push genuine CPU work to a thread/process pool.

In the lab, the event loop is real (asyncio.run starts one), and the only thing our injected model does to yield is await asyncio.sleep(0) — which schedules the coroutine to resume on the next loop tick without any real delay. That single sleep(0) is what lets concurrent requests interleave deterministically so a test can prove the semaphore caps them (§7). It is the purest possible demonstration of cooperative yielding: "I have nothing to do this instant, run someone else."

4. Coroutines, await, and the async generator

Three language constructs carry the whole phase:

A coroutine is what async def produces. Calling foo() where foo is async def does not run it — it returns a coroutine object, a suspendable computation. It runs only when awaited (await foo()) or scheduled on the loop (asyncio.create_task(foo()) / asyncio.gather(...)). This trips everyone up once: an async def you forget to await silently does nothing.

await means "suspend me until this finishes, and let the loop run others meanwhile." You can only await inside an async def. await asyncio.gather(a(), b(), c()) runs three coroutines concurrently and resumes when all three are done — the primitive we use to fire many agent requests at once.

An async generator — an async def containing yield — is the streaming primitive, and the heart of this lab. A normal generator (def + yield) produces a synchronous stream you pull with for x in gen. An async generator produces an asynchronous stream you pull with async for x in gen, and crucially it can await between yields — so it can wait for the next LLM token before yielding it. That is exactly the shape of a streaming LLM client:

async def model(prompt):        # the INJECTED streaming LLM stand-in
    for token in ...:
        await ...               # wait for the next token off the wire
        yield token             # hand it to the consumer, then suspend

async def stream_agent(prompt, model):
    async for token in model(prompt):        # pull tokens as they arrive
        yield {"event": "token", "data": token}
    yield {"event": "done", "data": {"tokens": n}}

stream_agent is itself an async generator that transforms one stream (raw tokens) into another (SSE-shaped events) without ever buffering the whole answer — the token you generated is handed onward the instant it exists. Injecting model as an async generator is the same determinism seam as every other lab: the non-deterministic streaming LLM becomes a pure, scripted async generator, so the test is a plain asyncio.run(...) with an exact expected event list.

5. Streaming: SSE vs WebSocket vs chunked, and why TTFT matters

Why stream at all? Because of time-to-first-token (TTFT). An LLM generating a 500-token answer takes, say, 6 seconds end to end. If you buffer the whole thing and return it as one JSON blob, the user stares at a spinner for 6 seconds — the perceived latency is 6 seconds. If you stream tokens as they generate, the first token lands in ~300 ms and text flows visibly after that — the perceived latency is 300 ms even though the total is unchanged. Every chat UI you've used streams for exactly this reason: streaming does not make the agent faster; it makes the wait feel like a fraction of what it is, and for a multi-second agent that is the difference between "feels broken" and "feels alive." (It also lets the user interrupt a wrong answer early, and lets you start downstream work on partial output.)

There are three ways to push incremental output over HTTP; know when each fits:

Mechanism	Direction	Shape	Best for
Chunked transfer-encoding	server → client	raw bytes, no framing	streaming a file/one big body incrementally
Server-Sent Events (SSE)	server → client only	text frames with named events, auto-reconnect, over plain HTTP	token streaming to a browser/UI — the default for LLM chat
WebSocket	full duplex (both ways)	binary/text frames, its own protocol/upgrade	interactive/bidirectional (voice, collaborative, live tool approval)

For agent token streaming, SSE is the right default: it is one-directional (the server pushes tokens; the client already sent its prompt in the POST), rides on a normal HTTP response (no protocol upgrade, works through proxies and load balancers that choke on WebSocket upgrades), has a dead-simple text format, and the browser's built-in EventSource handles reconnection and event dispatch for free. You reach for WebSocket only when you need the client to stream to the server mid-response too — real-time voice (Phase 16), live barge-in, collaborative editing. The lab builds the SSE path because it is what an agent chat endpoint uses; format_sse produces the exact wire bytes a FastAPI StreamingResponse(..., media_type="text/event-stream") would send.

6. The SSE wire format, precisely

SSE is almost insultingly simple, which is why it won for token streaming. The response has Content-Type: text/event-stream, and the body is a sequence of frames separated by blank lines. Each frame is one or more field: value lines. The fields that matter:

event: token
data: Par

event: token
data: is

event: done
data: {"tokens": 2}

Rules, exactly:

A frame ends at a blank line (\n\n). That blank line is what tells the client "this event is complete, dispatch it" — forget it and the browser buffers forever.
event: names the event; the browser's EventSource fires a listener registered for that name (es.addEventListener("token", ...), es.addEventListener("done", ...)). Omit it and the event is the default message type.
data: carries the payload. Multi-line data repeats the data: field — the client concatenates them with newlines. That's why format_sse splits the payload on \n and emits a data: line for each; structured payloads are JSON-encoded onto (usually) one line.
Optional id: sets a last-event-ID the client echoes on reconnect (Last-Event-ID header) so the server can resume — the SSE-native answer to "don't replay tokens the client already saw."
Optional retry: sets the client's reconnect delay in milliseconds.

That is the entire protocol. format_sse({"event": "token", "data": "Par"}) → "event: token\ndata: Par\n\n". The lesson: the wire format is trivial; the discipline is emitting a terminal event (done) so the client can distinguish "the stream ended cleanly" from "the connection dropped mid-answer" — a distinction that matters enormously when you're deciding whether to retry (§8).

7. Concurrency limits and backpressure: the semaphore and the queue

Async lets you run 10,000 requests concurrently on one loop. That is also a problem: every one of those requests wants to call your LLM provider, which has a rate limit (requests/minute, tokens/minute) and will start returning 429 Too Many Requests — or your own database will melt under 10,000 simultaneous connections. Unbounded concurrency turns one traffic spike into a total outage. You must cap concurrency and, past the cap, apply backpressure.

The tool for the cap is a semaphore: a counter initialized to max_concurrency. A request must acquire (decrement) before entering the critical section and release (increment) after; when the counter is 0, further acquires block (asynchronously — the coroutine parks on the loop, it does not spin) until someone releases. So at most max_concurrency requests are ever in the LLM-calling section at once, no matter how many arrive:

async with self._sem:          # at most max_concurrency inside at any instant
    ... call the model ...      # everyone else awaits their turn here

That waiting is backpressure: when the downstream (the LLM) can't keep up, the pressure propagates back up to callers as latency, instead of overwhelming the downstream. But you can't let the wait queue grow forever either — a mob of requests all parked on the semaphore is memory you don't have and latency no user will wait through. So you bound the queue and, past it, shed load: reject new requests with 503 Service Unavailable (in the lab, ServiceUnavailable) rather than accept work you can't do. Load-shedding is a feature, not a failure — a service that returns a fast 503 and stays up beats one that accepts everything and falls over. This is the same reliability instinct as Phase 00's step budget: know your limit and defend it with a number.

The lab wires exactly this: AgentService.handle bumps a queued counter, awaits the semaphore (the backpressure), and if queued already exceeds max_queue it raises ServiceUnavailable (the load-shed). metrics() exposes in_flight, queued, completed, rejected, and the peak max_in_flight ever seen — and a test asserts max_in_flight reaches but never exceeds max_concurrency, proving the cap holds under a burst of concurrent requests.

8. Idempotency: deduping retries and double-submits

Here is a fact of networked systems: requests get sent more than once. A client times out and retries; a mobile user double-taps "send"; a load balancer replays; an at-least-once queue redelivers. If the request has a side effect — charge a card, send an email, kick off a 20-tool, $2 agent run — doing it twice is a bug that costs money or spams a customer. The canonical fix, straight out of payments APIs (Stripe's Idempotency-Key header), is the idempotency key: the client attaches a unique key to the request; the server, before doing the work, checks whether it has already processed that key; if so it returns the stored prior result instead of re-running.

POST /agent  Idempotency-Key: req-abc123
  ├─ key seen before?  ──► return the CACHED response (replay). Do NOT re-run the agent.
  └─ new key          ──► run the agent once, STORE result under the key (with a TTL), return it.

Two subtleties make it production-grade:

Store the result, keyed, with a TTL. You don't keep idempotency records forever — you keep them long enough to cover the retry window (minutes to a day), then let them expire so the key can be reused and the store doesn't grow unbounded. This is why idempotency is a TTL-cache problem (§9), and why the lab's idempotency store is a TTLCache with the injected clock: a test advances the fake clock past the TTL and asserts the request re-runs.
A replay must not re-invoke the model. The whole point is to not do the expensive, side-effecting work again. The lab proves this precisely: the injected model counts its invocations, two handle calls with the same key are made, and the test asserts the model was called once — the second returns the cached Response marked replayed=True, with no model call and without even acquiring the semaphore.

This is the edge-of-the-service twin of Phase 08's durability lesson. There, retries with backoff made a durable workflow safe under at-least-once execution, and activities had to be idempotent to be safe to replay. Here, the idempotency key makes the whole request safe to retry: "run exactly once" reconstructed on top of an "at-least-once" network. Same principle, one layer out. (The honest limitation, worth stating in an interview: a pure cache-check races if two identical requests arrive truly concurrently — both miss, both run. Real systems close that with an atomic "reserve the key" on first sight, e.g. SET key NX in Redis, so the second sees "in progress." The lab implements the sequential-replay version and notes the race.)

9. State and cache: stateless services need external state

Modern services are deployed stateless: every instance is identical and disposable, because a deploy, a crash, or an autoscaler kills and respawns instances constantly, and a load balancer sends a user's next request to any instance, not the one that served the last. If instance A holds a user's conversation in a local Python dict, then when the next request lands on instance B — or A is recycled — that state is gone. So the rule is: the process holds no state that must survive a request. State that must persist — conversation history, session data, an agent's scratchpad across turns, a cached tool result, an idempotency record — lives in an external store every instance shares: Redis for fast ephemeral/session state, Postgres for durable records, a vector store for memory (Phase 05/06).

The workhorse pattern is a TTL cache: a key→value store where each entry carries a time-to-live and expires automatically. It's exactly Redis's SETEX key ttl value / GET key. TTLs matter because agent state is mostly ephemeral and self-correcting: a session that's idle for an hour is abandoned; a cached search result older than five minutes is stale; an idempotency record older than a day is past its retry window. Letting entries expire instead of leaking is what keeps an unbounded stream of sessions from becoming an unbounded memory bill.

The lab's TTLCache(now) is that store, with the one detail that makes it testable: the clock is injected as an integer now() tick, never time.time(). set(key, val, ttl) stamps an expiry at now() + ttl; get returns the value while now() < expiry and a MISS sentinel once now() >= expiry, dropping the dead entry on read (lazy eviction). Because time is a knob the test turns, expiry is exact: set with ttl=5 at tick 0, still there at tick 4, gone at tick 5. (This is the same injected-clock discipline as Phase 08 — a service that reads the wall clock is neither testable nor replay-safe, so we inject time everywhere it appears.) A MISS sentinel distinct from None matters because a stored None is a legitimate hit — the same reason Redis distinguishes a nil reply from a stored empty value.

10. Event-driven architecture and pub/sub

So far every request is synchronous request/response: the caller waits for the answer. But much of what an agent platform does is better modeled as events: "a job finished," "a document was indexed," "a tool needs human approval," "an agent emitted a trace span." An event-driven architecture decouples the thing that produces an event from the things that consume it. The producer publishes to a topic and moves on; any number of subscribers on that topic receive the event independently. The producer neither knows nor waits on its consumers.

Publish/subscribe (pub/sub) is the pattern, and its superpower is fan-out: one event, N independent consumers. When an agent finishes a job and publishes job.completed, a logger writes it to the audit trail, a webhook dispatcher POSTs it to the customer's callback URL, a metrics sink increments a counter, and a notifier emails the user — four consumers, zero coupling, each addable without touching the producer. This is how you build the parts of an agent platform that aren't a direct answer to a user: async job pipelines, webhooks, streaming logs, the agent-as-a-consumer-of-its-own-events. It is also how an agent becomes a consumer: an agent that reacts to document.uploaded by summarizing it is a subscriber, not a request handler.

The mechanics that make fan-out correct:

Each subscriber gets its own queue. If all subscribers shared one queue, the first to read would steal the event from the others (that's a work queue / competing-consumers pattern, which is a different tool — for load-balancing one job across a worker pool). For fan-out, every subscriber has a private queue, publish enqueues into all of them, so each receives every event. Knowing when you want fan-out (pub/sub) vs. competing-consumers (a task queue) is a real design decision.
Delivery order is deterministic. publish delivers to subscribers in a defined order (subscription order), and within one subscriber events arrive in publish order (FIFO queue). The lab asserts two subscribers both receive [event1, event2] in order.

The lab's EventBus is the in-process miniature: subscribe(topic) returns a Subscription (an async iterator over its own asyncio.Queue); publish(topic, event) fans out to every subscriber; async for event in subscription: consumes until close(). Swap the in-process queues for Redis pub/sub, Kafka, NATS, or an SSE hub and the shape is identical — the abstraction is what transfers, and the lab makes you build it so the broker stops being magic.

11. Health, readiness, graceful drain, and deployment

A service that can't tell the platform how it's doing gets killed at the worst moment. Two probes every orchestrator (Kubernetes, a load balancer, a serverless platform) expects:

Liveness / health — "am I alive?" If it fails, restart me. A crashed event loop or a deadlock fails liveness.
Readiness — "should you send me traffic right now?" A service that is starting up, overloaded, or draining is alive but not ready; the load balancer stops routing new requests to it without killing it. The lab's health() returns {"status": ..., "metrics": ...}; status == "draining" is exactly a readiness signal to route new work elsewhere.

Graceful drain is the lifecycle event that separates a real service from a script. On every deploy and every scale-down, your instance receives a shutdown signal (SIGTERM). If you exit immediately, you sever every in-flight request — including half-streamed agent responses a user is watching, and side-effecting tool calls mid-execution. The correct sequence is a graceful drain:

receive SIGTERM
  1. flip to "draining": stop accepting NEW requests (readiness fails, LB reroutes; new -> 503)
  2. WAIT for in-flight requests to finish (up to a deadline)
  3. then exit

The lab's shutdown() is precisely this: set _shutting_down (so handle/stream reject new work with ServiceUnavailable and health() reports draining), then await an idle event that the last completing request sets — so shutdown() returns only once in_flight == 0. A test creates three in-flight requests and asserts shutdown() does not return until all three finish, with completed == 3. That's the difference between a deploy that drops connections and one nobody notices.

Where this runs. In production the async service is FastAPI (or Starlette/aiohttp) on an ASGI server like uvicorn (often behind Gunicorn managing multiple uvicorn workers — one event loop per worker per core). Increasingly agent services run serverless — Vercel Fluid compute, AWS Lambda, Cloud Run — where a single instance handles many concurrent invocations precisely because the work is I/O-bound (Fluid compute's whole pitch), and the platform autoscales instances by concurrency/CPU. In all of these your job is the same: an async service that streams, caps concurrency, keeps state external, and drains cleanly — which is exactly what the lab builds, minus the network. The README's Extensions section shows the @app.post + StreamingResponse sketch that maps your stdlib version onto real FastAPI.

12. Common misconceptions

"Async makes my code faster / parallel." No. Async gives you concurrency on one thread for I/O-bound work — it lets many requests wait at once. It runs no Python in parallel (one coroutine executes at a time) and does nothing for CPU-bound work; a heavy CPU loop in a coroutine blocks the whole loop. Parallelism needs processes/cores.
"Streaming is just a UI nicety." It changes perceived latency from total-time to time-to-first-token, which for a multi-second agent is the difference between "broken" and "alive" — and it enables early interruption and downstream work on partial output. It's an architecture decision, not CSS.
"A stateless service means my service has no state." It means the process holds no state that must survive a request; the state moved to an external store (Redis/DB) so any instance can serve any request and a redeploy loses nothing. Stateless services have more state discipline, not less.
"One blocking call is fine, it's just one request." In an async service one synchronous requests.get / time.sleep / CPU loop stalls the entire event loop — every concurrent request waits on it. The whole model assumes you never block the loop.
"Idempotency is a database transaction thing." It's a request-level dedupe key that makes retries safe over an at-least-once network, distinct from (and layered above) DB transactions. A retried POST without an idempotency key is a duplicate side effect no transaction prevents.
"More concurrency is always better." Unbounded concurrency stampedes your rate-limited LLM and your DB into 429s and outages. The cap (semaphore) + backpressure + load-shedding is what keeps a spike from becoming an outage; a fast 503 beats a slow collapse.
"Pub/sub and a task queue are the same." Fan-out (pub/sub) delivers every event to every subscriber; a work queue (competing consumers) delivers each job to one worker. Picking the wrong one either duplicates work or drops it.

13. Lab walkthrough

Open lab-01-async-agent-service/ and fill the TODOs top to bottom — the file is ordered to match this warmup:

Streaming — format_sse (event line + one data: line per payload line + terminating blank line; JSON-encode non-strings), then stream_agent (an async generator: a token event per model token, then a done event with the count), then sse_stream (compose the two). Confirm the event list is [token, token, done].
State/cache — TTLCache.set/get with the injected clock: stamp now()+ttl, and treat now() >= expiry as a MISS (dropping the dead entry). The tests probe the exact expiry tick and the stored-None-vs-MISS distinction.
Event bus — EventBus.subscribe (register a new Subscription) and publish (fan out to every subscriber in order). The Subscription plumbing is given; you wire the fan-out.
The service — AgentService.handle: the draining check, the idempotency cache-hit replay (no model call), the load-shed on a full queue, then acquire the semaphore, drain stream_agent into text, store under the key, return. Then stream (the streaming twin) and shutdown (flip draining, await idle). __init__, _enter/_exit, metrics, and health are given — the interesting control flow is yours.

Run LAB_MODULE=solution pytest test_lab.py -v first to see green, then make your lab.py match. Finish by reading solution.py's worked example — it runs concurrent streaming under a semaphore, an idempotent replay, a TTL expiry on a fake clock, and a pub/sub fan-out, then prints the metrics and an SSE wire frame.

14. Success criteria

You can explain why agents are I/O-bound and why that makes async the architecture, and state Little's Law for "how many concurrent agents fit on one instance."
You can explain concurrency ≠ parallelism, and name the one bug (blocking the loop) that breaks an async service.
You can write the SSE wire format from memory and say why a terminal done event matters.
Your stream_agent yields tokens then done, and a test asserts the exact event list.
Your semaphore caps max_in_flight at max_concurrency under a burst, and a full queue sheds load with a 503 — and you can defend "a fast 503 beats a slow collapse."
Your idempotent replay returns the cached result without re-invoking the model (the call-count test), and expires with the TTL on the fake clock.
Your TTLCache expires at the exact tick and distinguishes a stored None from a MISS.
Your EventBus fans one event out to multiple subscribers in deterministic order.
Your shutdown() drains in-flight requests before returning and rejects new work.
All 26 lab tests pass under both lab and solution.

15. Interview Q&A

Q: Why is async the right model for an agent service, and what's the limit of one instance? A: Agent requests are almost entirely waiting on I/O — a generating LLM, a tool's HTTP call, a DB — so they're I/O-bound, and the CPU sits idle per request. Async runs thousands of those waits concurrently as cheap coroutines on one event-loop thread, versus a thread-per-request model that needs thousands of ~1 MB threads. By Little's Law, at 100 req/s and 8 s each you have ~800 in flight at once; async holds that as 800 coroutines on one loop. The limit isn't threads, it's the downstream (LLM rate limits, DB connections), which is why you cap concurrency.

Q: Concurrency vs parallelism — and the classic async bug? A: Concurrency is many tasks making progress by taking turns; parallelism is many tasks running at the same instant on multiple cores. An event loop gives concurrency on one thread — exactly one coroutine runs Python at a time, they interleave at awaits. It does nothing for CPU-bound work. The classic bug is blocking the loop: one synchronous call (requests.get, time.sleep, a CPU loop) never yields, so every concurrent request stalls behind it. Fix: await everything, offload real CPU work to a process pool.

Q: Why stream tokens, and SSE vs WebSocket? A: Streaming cuts perceived latency to time-to-first-token — the total generation time is unchanged, but the user sees text in ~300 ms instead of staring at a spinner for 6 s, and can interrupt early. For token streaming SSE is the default: server-to-client only (the client already sent its prompt), rides plain HTTP (no upgrade, proxy-friendly), trivial text format, and EventSource gives reconnection for free. Use WebSocket only when the client must stream to the server mid-response too — voice, live approval, collaboration.

Q: How do you make a retried agent request safe? A: An idempotency key. The client attaches a unique key; before running I check a keyed store — if the key was seen, I return the cached prior result (a replay) without re-running the agent or re-invoking the model; if it's new, I run once and store the result under the key with a TTL covering the retry window. It's the Stripe Idempotency-Key pattern and the edge-of-service twin of Phase 08's "activities must be idempotent under at-least-once execution." The subtlety: truly-concurrent duplicates race, so production reserves the key atomically (SET NX) on first sight.

Q: What does backpressure mean here, and what happens when you're overloaded? A: Async lets unbounded requests arrive, but the downstream LLM is rate-limited, so I cap concurrency with a semaphore — at most N requests call the model at once; the rest await their turn, and that waiting is backpressure (overload propagates back as latency instead of crushing the LLM). I also bound the wait queue and shed load past it — a fast 503 — because a mob parked on the semaphore is memory and latency I can't afford. A service that stays up shedding load beats one that accepts everything and collapses.

Q: Why is your service "stateless," and where does the state go? A: Instances are disposable — deploys, crashes, and autoscaling recycle them, and a load balancer sends the next request to any instance — so the process holds no state that must survive a request. Conversation history, sessions, cached tool results, and idempotency records live in an external store (Redis for fast/ephemeral with TTLs, Postgres for durable). Statelessness is what lets any instance serve any request and a redeploy lose nothing; it takes more state discipline, not less.

Q: Walk me through a graceful shutdown. A: On SIGTERM: flip to draining so readiness fails and the load balancer stops routing new traffic (new requests get a 503), wait for in-flight requests to finish up to a deadline — including half-streamed responses and mid-flight tool calls — then exit. Exiting immediately severs live connections and can leave side effects half-done. In the lab shutdown() sets the draining flag and awaits an idle event the last completing request sets, so it returns only at in_flight == 0.

Q: Pub/sub fan-out vs a task queue — when each? A: Fan-out (pub/sub) delivers every event to every subscriber — one job-completed event drives a logger, a webhook, a metrics sink, a notifier independently. A work queue (competing consumers) delivers each job to exactly one worker — for load-balancing work across a pool. Fan-out needs a per-subscriber queue; a work queue is one shared queue. Pick fan-out for event notification, a task queue for distributing work; mixing them up either duplicates or drops jobs.

16. References

Python asyncio docs — the event loop, coroutines, tasks, synchronization primitives (Semaphore, Event, Queue). https://docs.python.org/3/library/asyncio.html
PEP 525 — Asynchronous Generators (the async def + yield streaming primitive). https://peps.python.org/pep-0525/
WHATWG HTML — Server-Sent Events / the EventSource API and text/event-stream wire format. https://html.spec.whatwg.org/multipage/server-sent-events.html
MDN — Using Server-Sent Events. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
FastAPI — StreamingResponse and async endpoints. https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse
Starlette — EventSourceResponse / streaming responses (FastAPI's ASGI core). https://www.starlette.io/responses/
Stripe — Idempotent requests (the Idempotency-Key pattern). https://docs.stripe.com/api/idempotent_requests
Redis — SETEX / key expiration (TTL) and Pub/Sub. https://redis.io/docs/latest/commands/setex/ · https://redis.io/docs/latest/develop/interact/pubsub/
Little's Law and the tail-at-scale framing — Dean & Barroso, The Tail at Scale, CACM 2013. https://research.google/pubs/the-tail-at-scale/
Vercel — Fluid compute (many concurrent invocations per instance for I/O-bound / AI workloads). https://vercel.com/docs/fluid-compute
Kubernetes — liveness/readiness probes and graceful termination (SIGTERM, preStop). https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Nygard, Release It! (2nd ed.) — backpressure, load-shedding, bulkheads, the circuit breaker.

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems