« Phase 12 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 12 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the derivations; this is the stuff you say in the meeting.

30-second mental model

An agent in production is an I/O-bound network service — it spends its life waiting on a generating LLM and tools, using almost no CPU. So you serve it async: thousands of requests await at once on one event-loop thread (concurrency, not parallelism — never block the loop). You stream tokens over SSE because perceived latency is time-to-first-token, not total time. You keep state outside the process (Redis + TTL) because instances are disposable. You cap concurrency with a semaphore and shed load past the queue to protect a rate-limited LLM. You make retries safe with an idempotency key. You fan events out with pub/sub. And you drain in-flight requests on deploy so nobody's stream gets cut. Build all of it from asyncio; the frameworks are ergonomics over these exact mechanisms.

The numbers to tattoo on your arm

Framework one-liners

• FastAPI / Starlette — async endpoints; StreamingResponse(gen, media_type="text/event-stream") streams your async generator as SSE. Depends for injection, lifespan for startup/drain.
• uvicorn / Gunicorn — the ASGI server; one event loop per worker per core; --workers N scales across cores, async scales within a core.
• Redis — SETEX key ttl val / GET is your TTLCache; PUBLISH/SUBSCRIBE is your EventBus; SET key val NX EX is the atomic idempotency-key reserve.
• asyncio — gather (run many), Semaphore (cap), Queue (backpressure/fan-out), Event (drain signal), async for over an async generator (streaming).
• Serverless (Vercel Fluid / Lambda / Cloud Run) — one instance serves many concurrent invocations because the work is I/O-bound; the platform autoscales on concurrency.
• All of them — sugar over event loop + await + a semaphore + an external store + a broker.

War stories

• The service that fell over at 3× traffic. No concurrency cap. Every request hit the LLM at once; the provider returned 429s, retries piled on, the loop drowned. A Semaphore(20) plus a bounded queue and a fast 503 past it turned a total outage into graceful degradation.
• The time.sleep that tanked p99 for everyone. One endpoint did a synchronous time.sleep(retry_after) inside a coroutine "just to back off." It blocked the whole event loop; every concurrent request's latency spiked. await asyncio.sleep fixed it in one line.
• The double-charged customer. A mobile client retried a timed-out "resolve + bill" request; no idempotency key, so the agent ran twice and billed twice. An Idempotency-Key header + a Redis SET NX reserve made the retry a replay.
• The deploy that cut off streams. Old instances exited on SIGTERM immediately, severing every half-streamed answer mid-sentence. A graceful drain (stop new, await in-flight) made deploys invisible.
• The conversation that "randomly forgot" itself. History lived in a local dict; the load balancer sent turn 2 to a different instance. Moving it to Redis with a TTL fixed the "amnesia."

Vocabulary

I/O-bound (waits, doesn't compute) · event loop (one-thread cooperative scheduler) · coroutine / await · async generator (async def + yield, the streaming primitive) · concurrency ≠ parallelism · block the loop (the cardinal sin) · SSE / EventSource / text/event-stream · TTFT (time-to-first-token) · semaphore (concurrency cap) · backpressure / load-shedding / 503 · idempotency key (dedupe retries) · TTL cache / SETEX (external state) · stateless service · pub/sub / fan-out / topic / subscriber · readiness vs liveness · graceful drain · Little's Law (L = λW).

Beginner mistakes

1. Thinking async makes code parallel or faster — it gives I/O concurrency on one thread, nothing for CPU-bound work.
2. Blocking the loop with a synchronous call (requests, time.sleep, a CPU loop) — freezes every concurrent request.
3. Buffering the whole answer instead of streaming — TTFT = total latency, feels broken.
4. Unbounded concurrency — stampedes your rate-limited LLM into 429s; no semaphore, no backpressure.
5. Accepting all load under a spike instead of shedding it — a slow collapse beats no fast 503.
6. Keeping conversation/session state in a process dict — lost on the next deploy or a different instance.
7. No idempotency key — a retried side-effecting request runs (and bills) twice.
8. Reading time.time() for TTLs/expiry — untestable and non-deterministic; inject the clock.
9. Exiting immediately on SIGTERM — severs in-flight streams; drain first.
10. Confusing pub/sub fan-out with a work queue — duplicates or drops jobs.