« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 12 — Production Agent Services: FastAPI, Async, Streaming, State & Events

Answers these JD lines: Citi's "Python, FastAPI, asyncio, event-driven design, microservices, APIs" and its "context/prompt/intent engineering served behind a real API"; Docker's Staff "Agentic Platform" role — "event-driven agent workflows, state, cache, pub/sub" and secure serving; and the "streaming," "real-time," and "APIs" language threaded through Cohere, Wolters Kluwer, and Redcan in jd.md. This is the phase that turns "an agent that runs" into "an agent service that serves thousands."

Why this phase exists

Phases 00–11 built the agent and everything that makes it correct: the loop, tools, MCP, retrieval, orchestration, durability, sandboxing, security, evals. Every one of them ran the agent in one process handling one task. Nobody uses an agent that way. In production the agent lives behind an HTTP endpoint under real traffic, and a service wraps it — and that service is a distributed-systems problem that happens to have an LLM inside.

An agent request is a long, I/O-bound network call: it spends almost all its time waiting on the LLM to stream tokens, on tools, on a database, using almost no CPU. That single fact drives everything here:

1. Async, because agents are I/O-bound. Thousands of requests can wait at once on one event-loop thread as cheap coroutines — the concurrency model for I/O-bound work. Concurrency is not parallelism; the loop runs one coroutine at a time and never blocks.
2. Streaming, because perceived latency is time-to-first-token. You stream tokens over SSE so a multi-second answer feels instant and the user can interrupt — a change to the architecture, not the CSS.
3. State outside the process, because instances are disposable. A stateless service keeps conversation, session, cache, and idempotency records in an external store (Redis-shaped, with TTLs) so any instance serves any request and a deploy loses nothing.
4. Events, because a platform is more than request/response. Pub/sub fans one event out to many independent consumers (logger, webhook, metrics) — the decoupling that builds job pipelines, webhooks, and agent-as-consumer.
5. Limits and lifecycle, because traffic spikes and deploys happen. A semaphore caps concurrency to protect a rate-limited LLM; backpressure and load-shedding keep a spike from becoming an outage; idempotency makes retries safe; graceful drain makes deploys invisible.

Build these once, from the standard library, and every framework (FastAPI, Redis, a broker, Kubernetes, serverless) stops being magic and starts being ergonomics.

Concept map

• Async & the event loop: I/O-bound → concurrency on one thread; coroutines, await, asyncio.gather; the async generator as the streaming primitive; concurrency ≠ parallelism; never block the loop; Little's Law (L = λW) for capacity.
• Streaming: SSE vs WebSocket vs chunked; time-to-first-token (TTFT); the text/event-stream wire format (event:/data:/blank-line); a terminal done event.
• Concurrency & backpressure: an asyncio.Semaphore(max_concurrency); the wait queue is backpressure; bound it and shed load (a fast 503) past the limit.
• Idempotency: an idempotency key dedupes retries/double-submits; replay the cached result without re-invoking the model; store keyed with a TTL (Stripe pattern; Phase 08 twin).
• State/cache: stateless service + external state; a TTLCache with an injected clock; Redis SETEX/GET; session/conversation state.
• Event-driven / pub-sub: topics, subscribers, fan-out; per-subscriber queues; pub/sub vs a competing-consumers task queue; webhooks; agent-as-consumer.
• Lifecycle: health/readiness; graceful drain (shutdown waits for in-flight); FastAPI + uvicorn; serverless / Fluid compute; autoscaling.

The lab

Integrated scenario (how this shows up at work)

A support-agent product must serve chat to a web UI, a mobile app, and a partner's backend. The web UI wants streaming so the answer feels instant — you expose an SSE endpoint that maps a StreamingResponse onto your stream_agent. The mobile app retries aggressively on flaky networks — you require an idempotency key so a retried "resolve ticket" replays the cached result instead of running the $2 agent twice. Conversation history can't live in the process (three instances behind a load balancer, redeployed hourly) — it lives in Redis with a TTL. When a ticket resolves, a ticket.resolved event fans out to a webhook to the partner, a metrics counter, and an audit log — no coupling. Black-Friday traffic 5×'s — a semaphore caps calls to your rate-limited LLM, and past the queue limit you shed load with a fast 503 so the service stays up. And every deploy drains in-flight streams before the old instance exits, so nobody's answer gets cut off. That entire paragraph is this phase, and you can build every noun in it.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py (26 tests).
• You can explain why agents are I/O-bound and why async (not parallelism) is the model.
• You can write the SSE wire format from memory and say why you stream (TTFT).
• You can show the semaphore capping max_in_flight and load-shedding past the queue.
• You can trace an idempotent replay that does not re-invoke the model.
• You can explain a stateless service, a TTL cache, pub/sub fan-out, and a graceful drain.

Key takeaways

• An agent in production is an I/O-bound network service; async is the architecture, not an optimization. Concurrency is not parallelism, and one blocking call freezes the whole loop.
• Streaming is an architecture decision: it changes perceived latency to time-to-first-token and enables interruption — worth the wire-format discipline (event:/data:/blank line + a terminal done).
• Retries are a fact of networks; an idempotency key + a TTL store makes "run exactly once" hold on top of at-least-once delivery, without re-invoking the model.
• Stateless service, external state: instances are disposable, so conversation/session/cache/ idempotency state lives in Redis-shaped stores with TTLs, on an injected clock.
• Decouple with events: pub/sub fan-out builds the platform parts that aren't request/response (webhooks, pipelines, agent-as-consumer).
• Defend your limits: cap concurrency, apply backpressure, shed load, and drain gracefully — a service that stays up shedding load beats one that accepts everything and collapses.