Continuous Batching

Phase 7 · Document 03 · Production Serving Prev: 02 — TGI, SGLang, TensorRT-LLM · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Continuous batching is the single technique that makes LLM serving economical. It's why one GPU can serve hundreds of users instead of one, and why vLLM/TGI/SGLang exist. Without it, your expensive GPU sits idle waiting for slow requests while fast ones finish; with it, the GPU stays saturated and throughput rises 3–5×. Every capacity plan, cost model, and "how many users per GPU" answer flows from understanding it. It's also where a subtle production reality lives: the same technique that maximizes throughput is what makes your per-request latency depend on your neighbors — the root of several incidents and of the non-determinism you read about.

2. Core Concept

Plain-English primer: why naïve batching wastes the GPU

Recall the economics (Phase 6.01, what-happens §1.D): decode is memory-bandwidth-bound, so a forward pass that reads all the weights can compute the next token for one user or for many users in almost the same time. Batching = computing several users' next tokens together in one pass, amortizing the weight-read. The question is how you form the batch.

Static (naïve) batching: collect N requests, run them as a fixed batch until all finish, then start the next batch. Two big wastes:

1. Tail waste: requests have different output lengths. A batch of 8 where one answer is 500 tokens and the rest are 20 means 7 slots sit idle for ~480 steps, waiting for the straggler.
2. Head-of-line waste: a new request that arrives 1ms after the batch starts must wait for the entire batch to finish before it can begin.

LLM output lengths are wildly variable and unknown in advance, so static batching leaves the GPU badly underused.

Continuous batching (a.k.a. in-flight / iteration-level batching)

The fix: make batching decisions every decode iteration, not every batch. The scheduler maintains a running set of sequences and, after each single-token step:

• removes any sequence that just finished (hit EOS / max_tokens) and frees its KV blocks (04), and
• admits waiting requests into the freed slots (they do a quick prefill, then join the decode loop).

So the batch is continuously refilled — finished requests leave immediately and new ones slot in at the next step, instead of everyone marching in lockstep. The GPU stays full of useful work. This is what vLLM, TGI, SGLang, and TensorRT-LLM all do (01, 02).

STATIC (lockstep):                        CONTINUOUS (iteration-level):
step1 [A B C D]                           step1 [A B C D]
step2 [A B C D]   B,C,D done, A long →    step2 [A B C D]  → B finishes
step3 [A _ _ _]   ← 3 idle slots          step3 [A E C D]  → E admitted into B's slot
step4 [A _ _ _]   ← wasted                 step4 [A E C F]  → D finished, F admitted
...   wait for A                           ...  GPU stays FULL of useful work

Prefill/decode interleaving and chunked prefill

A wrinkle: a brand-new request must first prefill its (possibly long) prompt — a big compute burst (Phase 2.07) that can stall the decode steps of everyone already running, spiking their latency. Chunked prefill (01 --enable-chunked-prefill) splits a long prefill into smaller chunks interleaved with ongoing decodes, so one big prompt doesn't freeze the batch — trading a little prefill latency for much smoother decode latency across users.

What it buys — and what it costs

• Throughput: ~3–5× over static batching on typical mixed workloads — the headline win.
• Utilization: GPU stays near-saturated; cost per token drops because the fixed weight-read is shared across more tokens.
• The cost: per-request latency now depends on batch composition — under heavy load your tokens interleave with many others, so p95/p99 latency rises with concurrency (the throughput↔latency trade-off). And batch composition changing step-to-step is a source of floating-point non-determinism.

3. Mental Model

   decode step ≈ "read all weights once" → can compute 1 token for MANY users (batching)
   BUT output lengths vary wildly + requests arrive anytime → static batching idles the GPU

   CONTINUOUS BATCHING = re-decide the batch EVERY token-step:
        finished seq → LEAVES (frees KV [04])      waiting req → JOINS (prefill, then decode)
        ⇒ GPU stays FULL of useful work  ⇒ 3–5× throughput  ⇒ cost/token ↓

   chunked prefill: slice a long new prompt's prefill so it doesn't STALL others' decode
   the trade: higher throughput, but per-request p95/p99 latency RISES with concurrency

Mnemonic: don't wait for the slowest request — refill the batch every step. Throughput up, but your latency depends on your neighbors.

4. Hitchhiker's Guide

What to look for first: is continuous batching on (it is, by default, in vLLM/TGI/SGLang) and what's your throughput vs concurrency curve. That curve is your capacity and cost model.

What to ignore at first: hand-tuning scheduler internals. The defaults are strong; tune --max-num-seqs and chunked prefill before anything deeper.

What misleads beginners:

• Measuring at batch 1. Continuous batching does nothing for a single request — its entire value is under concurrency. Batch-1 benchmarks hide the real behavior (01).
• Thinking throughput and latency move together. They trade off: more concurrency → more throughput and higher per-request p95.
• Ignoring prefill stalls. A few long prompts can spike everyone's latency without chunked prefill.
• Forgetting KV is the limit. Continuous batching admits new requests only while KV blocks are free — concurrency is capped by KV memory (04, Phase 6.02), not by the scheduler.

How experts reason: they treat the throughput–latency–concurrency surface as the core capacity model: pick a target p95, find the max concurrency that holds it, and sleep at that batch size. They enable chunked prefill for mixed/long-prompt traffic and size --max-num-seqs to the KV budget.

What matters in production: the knee of the throughput-vs-concurrency curve (where p95 starts to blow up), queue depth (num_requests_waiting) as the backpressure signal, and admission control so you queue/reject rather than thrash at the limit.

How to debug/verify: sweep concurrency and plot throughput + p50/p95; watch queue depth and KV usage (08). p95 exploding while throughput plateaus = you're past the knee.

Questions to ask: Is continuous batching on? What --max-num-seqs? Chunked prefill enabled? What's p95 at target concurrency? Where's the throughput knee?

What silently gets expensive/unreliable: running past the latency knee (p95 SLO breach), prefill stalls without chunking, and assuming batch-1 latency reflects production.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Continuous batching = iteration-level scheduling: re-decide the batch every decode step (finished leave, waiting join).
• It yields ~3–5× throughput over static batching on typical variable-length workloads.
• Its value appears only under concurrency — batch-1 latency is unchanged.
• Concurrency is capped by KV memory, not the scheduler (04, Phase 6.02).
• Throughput and latency trade off: more concurrency → more throughput and higher p95/p99.
• Chunked prefill stops a long new prompt from stalling everyone's decode.
• Batch composition changes step-to-step, contributing to FP non-determinism.
• It originated as iteration-level scheduling (Orca) and is standard in all modern engines.

9. Observations from Real Systems

• Every modern engine (vLLM, TGI, SGLang, TensorRT-LLM "in-flight batching") ships continuous batching as the default — it's table stakes (02).
• Provider throughput numbers are continuous-batching-at-high-concurrency figures — single-user speed is lower (Phase 4.03, Phase 5.10).
• Latency incidents frequently trace to running past the throughput knee or to prefill stalls from long prompts (10).
• The "why is the same prompt nondeterministic at temp 0" question partly answers here: batch composition shifts the FP reduction order (what-happens §7).
• Speculative decoding interacts with batching: its gains concentrate at low batch where the GPU is underused; at high batch continuous batching already fills it (Phase 6.07).

10. Common Misconceptions

11. Engineering Decision Framework

TUNE for your workload:
 1. Confirm continuous batching is ON (default in vLLM/TGI/SGLang).
 2. Sweep concurrency → plot throughput + p50/p95. Find the KNEE (p95 SLO breaks).  [08]
 3. Set --max-num-seqs near the knee, bounded by KV budget (Phase 6.02 / [04]).
 4. Mixed/long prompts spiking others' latency? → --enable-chunked-prefill.
 5. Add admission control (queue/reject) at the gateway so load past the knee degrades gracefully. [07]
 6. Low-batch latency-critical path? → consider speculative decoding (Phase 6.07).

12. Hands-On Lab

Goal

Measure the throughput-vs-latency-vs-concurrency surface and locate your knee, then show chunked prefill smooths long-prompt stalls.

Prerequisites

• A vLLM endpoint (01); pip install httpx.

Setup

vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 4096 \
  --max-num-seqs 256 --port 8000           # continuous batching is on by default

Steps

1. Sweep concurrency: for C in {1,2,4,8,16,32,64,128}, fire C simultaneous streaming requests (short prompt, ~200-token answer). Record aggregate tok/s and p50/p95 TTFT + TPOT.
2. Plot the surface: throughput vs C (rises then plateaus) and p95 vs C (flat then sharp). The knee is where p95 turns up while throughput flattens.
3. Prove static-batch waste (conceptually): include one very long (1,000-token) request among short ones; observe that with continuous batching the short ones still finish promptly (they're not stuck behind the long one).
4. Prefill stall vs chunked: send several 4k-token prompts concurrently with --enable-chunked-prefill off vs on (restart to toggle); compare the p95 TTFT of short requests that arrive during the big prefills.
5. Read the engine: during the sweep, watch vllm:num_requests_running/waiting and gpu_cache_usage_perc.

Expected output

A throughput/p95-vs-concurrency plot with a marked knee; evidence that short requests aren't blocked by long ones; and a chunked-prefill before/after on short-request TTFT.

Debugging tips

• Throughput never rises → batching off, KV-capped early (gpu_cache_usage), or you're CPU/network-bound on the client.
• p95 huge even at low C → prefill stalls; enable chunked prefill.

Extension task

Overlay the speculative-decoding result from Phase 6.07: show it helps at C=1 but its benefit shrinks as C grows (the GPU's already full).

Production extension

Set --max-num-seqs at the knee and add gateway admission control (queue beyond it); verify graceful degradation under a load spike (07).

What to measure

Throughput and p50/p95 TTFT/TPOT vs concurrency; the knee; chunked-prefill effect on short-request TTFT; KV usage/queue depth.

Deliverables

• A throughput/latency vs concurrency plot with the knee marked.
• A chunked-prefill before/after on short-request p95.
• A chosen --max-num-seqs + admission-control note tied to your SLO.

13. Verification Questions

Basic

1. Why does static batching waste the GPU on variable-length workloads?
2. What does continuous batching re-decide, and how often?
3. Why does batch-1 benchmarking hide its value?

Applied 4. Throughput rises with concurrency but p95 explodes past C=40. What is C=40 called and how do you use it? 5. Why does a long prompt hurt other users' latency, and what fixes it?

Debugging 6. Adding concurrency stops increasing throughput. Two causes. 7. Short requests are slow only when big prompts are in flight. Fix?

System design 8. Design a capacity model: given a p95 SLO, how do you choose --max-num-seqs and admission control?

Startup / product 9. How does continuous batching change your cost-per-token, and what's the latency cost you trade for it?

14. Takeaways

1. Continuous batching refills the batch every decode step — finished leave, waiting join.
2. It delivers ~3–5× throughput and lower cost/token by keeping the GPU saturated.
3. Its value is under concurrency; batch-1 is unchanged.
4. Concurrency is KV-capped; throughput and latency trade off along a curve with a knee.
5. Chunked prefill prevents long prompts from stalling the batch; run below the knee for your p95 SLO.

15. Artifact Checklist

• A throughput/p95-vs-concurrency plot with the knee.
• Evidence short requests aren't blocked by a long one.
• A chunked-prefill before/after on short-request TTFT.
• A chosen --max-num-seqs + admission-control plan tied to an SLO.
• (Optional) speculative-decoding-vs-concurrency overlay.

Up: Phase 7 Index · Next: 04 — PagedAttention