« Phase 12 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 12 — Staff Engineer Notes: Production Agent Services — Async, Streaming, State & Events

Anyone can stand up a FastAPI endpoint that streams tokens. The gap between that person and the one an org trusts to own an agent-serving platform is not more framework knowledge — it is judgment about the four or five decisions that decide whether the service survives a bad Tuesday. This document is about those decisions, the review instincts that catch the failures early, and the exact thing an interviewer is listening for.

What separates "uses FastAPI" from "owns the platform"

The person who uses FastAPI reaches for async def because the tutorial did, keeps conversation in a module-level dict because it worked locally, and streams because chat UIs stream. Everything works in the demo. The person who owns the platform can tell you, unprompted, why the request is I/O-bound and therefore why async is the architecture and not a performance tweak; why the process must hold no state that outlives a request; where the concurrency cap belongs and what number it takes; and what happens to a half-streamed answer when the pod gets SIGTERM'd mid-deploy. The tell is not that they know these mechanisms exist — it is that they reason about the failure first and the happy path second. Ownership is a failure-mode posture.

The decisions a staff engineer actually owns

Async vs sync worker model. The default is async, because agent requests are 99% wait. But the moment there is real CPU in the path — local embedding, reranking, image work, heavy JSON transforms — you own the decision to push it off the loop (thread or process pool) or run a hybrid. The failure of getting this wrong is not slow; it is a whole worker frozen. Owning it means you can name which calls in the path are CPU-bound and where they go.

Where state lives. This is a per-datum decision, not a blanket one. Conversation history and sessions: external, Redis-shaped, with a TTL. Idempotency records: external, TTL covering the retry window, atomic first-write. Cache: external, TTL sized to staleness tolerance. Nothing that must survive a request stays in the process. The staff move is to enumerate every piece of state in the request and say, for each, where it lives and why — and to catch the one someone quietly left in a dict.

Concurrency caps and where they sit. The cap is not "protect the box" — the box can hold thousands of coroutines. The cap protects the scarcest downstream: the LLM's RPM/TPM quota, the database connection pool. You own the number (downstream_quota / worker_count, with headroom) and the placement (at the scarce resource), and you own the decision to bound the wait queue and shed past it. A cap sized to the loop instead of the quota is the most common sizing error at this level.

Drain deadline. Graceful drain is table stakes; the deadline is the judgment. Too short and you cut off long agent runs; too long and a stuck request blocks every deploy and the orchestrator SIGKILLs you anyway, defeating the point. You own the number, and you own aligning it with the orchestrator's grace period and the load balancer's deregistration timing so traffic stops arriving before SIGTERM lands.

Decision frameworks worth memorizing

SSE vs WebSocket vs polling. Default to SSE for token streaming: the flow is server-to-client only (the client already sent its prompt in the POST), it rides plain HTTP so it survives proxies and load balancers, the format is trivial, and EventSource gives reconnection free. Reach for WebSocket only when the client must stream to the server mid-response — live voice, barge-in, collaborative editing, live tool approval. Fall back to polling only when infrastructure forbids long-lived connections. The red flag is a WebSocket used for one-way token streaming: it is strictly more operational surface (its own protocol, upgrade handshake, proxy incompatibilities) for zero benefit.

Semaphore vs external rate-limiter. A per-process semaphore caps one worker's concurrency and is perfect when one worker owns its slice of the quota. The moment many workers or many services share a single provider quota, an in-process semaphore is blind to the others and the aggregate blows the limit — you need a distributed limiter (a Redis token bucket, a gateway). Framework: local semaphore for per-worker concurrency, external limiter for a shared global budget. Name which you have.

Pub/sub vs task queue. Fan-out (pub/sub, per-subscriber queues): one event, every subscriber gets it — logger, webhook, metrics, notifier, all independent. Competing consumers (task queue, one shared queue): one job, exactly one worker — for distributing work across a pool. Choosing fan-out for work distribution runs every job N times; choosing a work queue for notification means only one of your four consumers sees the event. The question that disambiguates: "should every consumer react, or should exactly one?"

Buffer vs stream. Stream when perceived latency matters and the output is incremental (chat). Buffer when the client needs the whole result atomically (a structured tool result a caller parses as one JSON object) or the output is small. The red flag is buffering a multi-second generation and returning one blob — TTFT equals total time and it feels broken.

Code-review red flags (the fast scan)

  • A synchronous call in an async def — requests.get, time.sleep, open().read() of a large file, a heavy loop. One line, whole-worker blast radius. This is the first thing to grep for.
  • State in a module-level dict, a class attribute, or an lru_cache that must survive a request. Works on one instance, "randomly forgets" behind a load balancer.
  • A side-effecting POST with no idempotency key. A retry double-charges; no downstream transaction saves you.
  • An unbounded queue or an unbounded semaphore wait — Queue() with no maxsize, no max_queue shed path. A spike becomes OOM.
  • A streaming endpoint with no terminal event and no Retry-After/deadline story. The client cannot tell "done" from "dropped."
  • asyncio.gather where a failure should abort the batch (leaks running siblings) — prefer TaskGroup.
  • Reading time.time() for TTL/expiry logic — untestable and non-deterministic; inject the clock.
  • SSE behind a proxy with buffering on — streaming that silently isn't. Ask how they verified TTFT in the real topology, not localhost.
  • A single global semaphore presented as multi-tenant fairness — it is tenant-blind; one tenant starves the rest.

War stories that teach the lesson

The 3x spike outage. No concurrency cap. Every request hit the LLM at once; the provider returned 429s; retries piled on; the loop drowned in work that would never succeed. A Semaphore sized to the quota, a bounded queue, and a fast 503 past it turned a total outage into graceful degradation. The lesson: an unbounded async service is a self-inflicted DDoS on your own upstream.

The time.sleep that tanked p99 for everyone. One handler did a synchronous time.sleep(retry_after) "just to back off." It froze the whole loop; every concurrent request's latency spiked. One character — await asyncio.sleep — fixed it. The lesson: in async, one blocking line is never "just one request."

The double-charged customer. A mobile client retried a timed-out "resolve + bill" over flaky network; no idempotency key, so the agent ran and billed twice. A key plus an atomic SET NX reserve made the retry a replay. The lesson: retries are a fact of networks, not an edge case.

The deploy that cut off streams. Old pods exited immediately on SIGTERM, severing every half-streamed answer. Graceful drain — stop admitting, await in-flight to a deadline, then exit — made deploys invisible. The lesson: routine operations are where reliability is won or lost.

The signal an interviewer is listening for

When the prompt is "design an agent API that serves thousands," the junior answer lists FastAPI features. The staff answer reframes first: "An agent request is a long I/O-bound network call, so this is a distributed-systems problem with an LLM at the edge." From that one sentence everything follows — async as architecture, streaming for TTFT, external state for disposability, a cap at the scarce downstream, load-shedding over collapse, idempotency over at-least-once, drain over abrupt exit. The interviewer is listening for whether you lead with the failure modes and the tradeoffs or with the feature list; whether you can put a number on capacity (Little's Law, L = λW) and on the cap (quota over workers); and whether you volunteer the limits of your own design (the single-pool semaphore is tenant-blind; the naive dedupe races). Naming the weakness before you are asked is the loudest seniority signal there is.

Closing takeaways

  1. Async is the architecture, not an optimization. The request is I/O-bound; one blocking call freezes the whole worker. Lead every design with that reframe.
  2. The process holds nothing that must survive a request. Enumerate state datum by datum and put each in an external store with a TTL; disposability is what makes horizontal scale and invisible deploys possible.
  3. Cap at the scarce downstream, then shed. Size the semaphore to the LLM quota, bound the queue, and defend the limit with a fast 503 — a service that stays up shedding load beats one that accepts everything and collapses.
  4. Make retries safe and deploys invisible. An idempotency key with an atomic reserve turns at-least-once into exactly-once; a bounded graceful drain turns every deploy into a non-event.
  5. Own the numbers and name the limits. Capacity, cap, drain deadline, TTLs — put a defensible number on each, and say out loud where your design is single-pool, non-atomic, or proxy-fragile. The judgment, not the framework, is what they are hiring.