Lab 03 — vLLM Throughput Tuning

Goal: stand up vLLM, sweep its serving knobs, and produce a throughput-vs-p95-latency (goodput) curve with a chosen operating point that meets a defined SLO. This is the JD's "scalable inference pipelines using vLLM … with clear trade-off analysis" as an artifact.

Knowledge prereq: K04 — Serving Frameworks, K01 — Inference Internals, K06 §8 — Load testing.

Stack: vllm, the Lab 01 harness (or vllm's benchmark_serving.py), a CUDA GPU.

Steps

  1. Define an SLO first (non-negotiable — K01 §8): e.g., p95 TTFT < 500 ms, p95 TPOT < 50 ms. Pick a realistic input/output length (e.g., 1000-in / 256-out) — not 128/128, which hides decode cost.
  2. Baseline: vllm serve <model> --port 8000. Run the harness at concurrency = 1, 4, 16, 64, 128, 256. Record throughput + p95 TTFT/TPOT at each. You're tracing the load curve.
  3. Find the knee: where p95 latency starts climbing steeply as concurrency rises — that's where queueing dominates (K06 §8). Beyond it, throughput gains cost unacceptable latency.
  4. Sweep one knob at a time (K04 §8), re-running the curve:
    • --gpu-memory-utilization (0.85 → 0.95): more KV room → bigger effective batch.
    • --max-num-seqs: the batch cap (throughput↔latency dial).
    • --max-num-batched-tokens: prefill chunking / token budget per step.
    • --enable-prefix-caching: with a big shared system prompt — watch TTFT drop (K04 §5).
    • --quantization fp8/awq: combine with Lab 01.
    • --kv-cache-dtype fp8: more concurrency.
  5. Plot the goodput Pareto curve: throughput (x) vs p95 TPOT (y). Draw the SLO line. The best operating point is the highest throughput still under the SLO line.
  6. Verify GPU utilization: while loaded, check it's actually busy and efficient — not idle (K06 §4–5). Low utilization at high concurrency = a config problem to chase.
  7. Soak test: run the chosen config for 30+ min — watch for throughput decay, KV preemption, OOM (K04 §6).
  8. Write the tuning runbook (K04 §10): the chosen config, the curve, the operating point, and the headroom — the customer deliverable.

What you must be able to explain

  • Why throughput rises with batch until the compute roof or KV memory caps it (K01 §7).
  • Why you tune for goodput, not raw throughput (K06 §10).
  • What PagedAttention and continuous batching are doing under the hood while you sweep (K04 §3–4).

Result / résumé bullet

"Tuned a vLLM deployment to a 50 ms p95 TPOT SLO: mapped the throughput-vs-latency goodput frontier across batch, KV-utilization, and prefix-caching knobs, raising sustainable concurrency 2.7× while holding the SLO, and documented the operating point in a runbook."