Lab 02 — Continuous Batching Simulator + Goodput (Python)

Phase: 07 — LLM Serving | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–8 hours Language: Python (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 5–6.

Run

python solution.py

0. The mission

Simulate an LLM server under Poisson request load and prove two things with numbers: continuous batching beats static, and goodput peaks at an interior batch depth (so "maximize batch" is wrong, "maximize goodput" is right — WARMUP Ch. 6).

The model is a discrete-iteration simulator: each iteration advances every in-batch sequence one decode step (or processes a prefill); the per-iteration time grows with batch size (more work per step → higher TPOT), which is what creates the latency-throughput dial.

1. What the output shows

== static vs continuous batching (Poisson load) ==
policy       thru(tok/s)  p50_TTFT  p99_TTFT  p50_TPOT completed
static               255     35774     75020        31       400
continuous           378        33      1669        31       400
  -> continuous: 1.5x throughput, 45x lower p99 TTFT

== goodput vs batch depth (SLO: TTFT<300ms, TPOT<40ms) ==
 max_batch  throughput   goodput
         4         209         3   <- batch too small: saturates -> TTFT SLO fails
         8         337        36   <- still saturating
        16         378       321       <- both SLOs met: goodput ~= throughput
        32         368         0   <- TPOT SLO blown (iter time 47ms > 40ms)
        64         350         0   <- deeper batch, worse goodput
goodput-optimal batch depth: 16
  • Static suffers batch-formation delay (waits for a whole batch to arrive) and head-of-line blocking (the batch runs at its longest member's length while short requests waste their slots) → enormous TTFT. Continuous admits and retires per iteration → here 1.5× throughput and 45× lower p99 TTFT (WARMUP Ch. 5). At moderate load the headline win is latency; under saturation it becomes throughput.
  • Goodput sweep — the headline result: raw throughput is highest at batch 32, but goodput collapses to zero there because per-token time (47 ms) blows the TPOT SLO. Too-small batches saturate and blow the TTFT SLO. Goodput peaks at an interior depth (16) — proof the objective is goodput, not max batch (WARMUP Ch. 6).

The per-iteration time scales with the configured max_batch (real engines graph-capture/pad decode kernels to the batch width), which is what makes a larger batch genuinely cost every user's TPOT. Production continuous-vs-static gains are often larger (3–10×) once variable arrivals and prefill interference — beyond this model — are included.

2. Reading order (solution.py)

  1. Request / Poisson arrival generation.
  2. simulate_static — fixed batches, run to completion.
  3. simulate_continuous — the iteration loop: retire finished, admit waiting, step the batch; per-iteration time as a function of batch size.
  4. The goodput accounting: a request's tokens count only if its TTFT and TPOT met the SLO.

3. Extensions

  1. Chunked prefill: split long prefills across iterations so they don't stall decodes; measure the p99 TPOT improvement under a bimodal (short+long prompt) workload (WARMUP Ch. 5).
  2. Prefix-cache integration: requests sharing a system prompt skip prefill (use Lab 01's idea) — watch TTFT collapse on cache hits.
  3. SLO-aware admission: reject/queue requests when admitting them would blow the batch's TPOT SLO; show goodput protected under overload.
  4. Prefill/decode disaggregation: separate prefill and decode "pools"; model the KV hand-off cost; compare interference vs the chunked-prefill approach.

4. Common pitfalls

  1. Constant per-iteration time regardless of batch — then there's no latency-throughput dial and goodput never turns over. Per-iteration time must grow with batch size (the simulator models this).
  2. Counting throughput as goodput — goodput must filter by SLO compliance, or the whole lesson disappears.
  3. Poisson vs back-to-back arrivals — back-to-back inflates throughput unrealistically (WARMUP Ch. 6); the sim uses Poisson.
  4. Ignoring prefill cost — a long prefill should visibly stall the iteration it enters (motivating chunked prefill).

5. What this lab proves about you

You can quantify the two serving truths that matter most to the P&L: continuous batching's throughput win and the goodput-optimal operating point. You can walk an exec through "why we don't just crank the batch size" with a graph, and you can design admission/SLO policy from the goodput curve — the platform-leadership deliverable this JD hires for.