vLLM — The Production Serving Engine

Phase 7 · Document 01 · Production Serving Prev: 00 — Serving Architecture · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

vLLM is the default open-source engine for serving LLMs at scale — the thing you reach for the moment Ollama/llama.cpp's low-concurrency ceiling (Phase 6.04) becomes the bottleneck. It bundles the three innovations that made self-hosting economical — PagedAttention (04), continuous batching (03), and automatic prefix caching (05) — behind an OpenAI-compatible API, so you can serve a 70B to hundreds of concurrent users on your own GPUs and have existing OpenAI client code just work. If you self-host, you will almost certainly run vLLM (or a close cousin, 02); knowing its flags, metrics, and memory model is core production-serving literacy.

2. Core Concept

Plain-English primer: what vLLM is and the problem it solved

vLLM is a Python/CUDA inference server: you give it a model (Hugging Face safetensors, Phase 6.03) and it loads it onto your GPU(s) and exposes an HTTP endpoint that turns prompts into tokens for many concurrent users at once.

The problem it solved: naïve serving pre-allocates a big contiguous KV-cache buffer per request sized for the maximum context, and batches requests statically (a batch starts and ends together). Both waste enormous GPU memory and time — most requests don't use max context, and fast requests wait for slow ones. vLLM fixed both:

• PagedAttention stores the KV cache in small fixed blocks (like OS memory pages) allocated on demand, so no memory is wasted on unused context and fragmentation nearly vanishes (04). This lets vLLM pack far more sequences into the same VRAM.
• Continuous batching adds new requests to (and removes finished ones from) the running batch every decode step, keeping the GPU saturated (03). Result: ~3–5× the throughput of static batching.
• Automatic prefix caching shares KV blocks across requests with a common prefix (e.g., the same system prompt) — server-side prompt caching (05).

Running it (OpenAI-compatible)

pip install vllm

# Start an OpenAI-compatible server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")  # key ignored
r = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role":"user","content":"Explain PagedAttention in 2 sentences."}])
print(r.choices[0].message.content)

The flags that matter (and the memory model behind them)

vLLM's memory math is straight from Phase 6.02: weights load once; the rest of the GPU memory becomes the KV-cache pool, which determines how many concurrent/long sequences fit.

Observability built in

vLLM exposes Prometheus metrics at /metrics — your window into the serving capacity that 08 builds on:

vllm:gpu_cache_usage_perc        ← KV-cache utilization (your capacity ceiling)
vllm:num_requests_running         ← current batch size
vllm:num_requests_waiting         ← queue depth (backpressure signal)
vllm:time_to_first_token_seconds  ← TTFT histogram
vllm:time_per_output_token_seconds← TPOT histogram
vllm:prompt_tokens_total / generation_tokens_total

3. Mental Model

   vllm serve MODEL  →  loads weights ONCE on GPU(s)  →  rest of VRAM = KV-CACHE POOL
        │                                                      │ (PagedAttention blocks [04])
        │  OpenAI-compatible /v1/chat/completions              │
        ▼                                                      ▼
   requests → SCHEDULER → CONTINUOUS BATCH (add/drop each step [03]) → stream tokens
                  │  prefix cache shares common-prefix blocks [05]
                  └─ /metrics: cache_usage · running · waiting · TTFT · TPOT  [08]

   capacity ceiling = KV pool ÷ per-seq KV(ctx)   →  set --max-model-len, --max-num-seqs,
                                                       --gpu-memory-utilization, --kv-cache-dtype

Mnemonic: vLLM = weights once + KV pool managed by PagedAttention + continuous batching + prefix cache, behind an OpenAI API. Tune the KV pool to set concurrency.

4. Hitchhiker's Guide

What to look for first: --max-model-len, --gpu-memory-utilization, and --max-num-seqs — together they set your KV pool and thus your concurrency ceiling. And /metrics to watch it.

What to ignore at first: multi-node pipeline parallelism, custom schedulers, and exotic quant until a single-node vLLM is saturated.

What misleads beginners:

• Setting --max-model-len to the model's max "to be safe." That reserves huge per-request KV and slashes concurrency (Phase 6.02). Set it to real need.
• Cranking --gpu-memory-utilization to 0.98. No headroom → OOM under load.
• Expecting GGUF. vLLM serves safetensors (AWQ/GPTQ/FP8), not GGUF (that's llama.cpp, Phase 6.03). vLLM has experimental GGUF support but it's not the path.
• Benchmarking batch 1. vLLM's win is throughput under concurrency, not single-stream latency (03).

How experts reason: they size the KV pool from the memory model, set --max-model-len/--max-num-seqs to the workload, enable prefix caching + chunked prefill, pick AWQ/FP8 to fit bigger models, and watch gpu_cache_usage_perc + num_requests_waiting as the capacity/backpressure signals. They scale out with tensor parallelism (within a node) before pipeline parallelism (across nodes).

What matters in production: the KV ceiling (concurrency), p95/p99 TTFT/TPOT, queue depth (waiting requests = need more capacity), graceful behavior at the limit (queue vs reject), and health/readiness probes for the load balancer.

How to debug/verify: /metrics first. High cache_usage + rising waiting = KV-capped → lower --max-model-len, enable KV quant, or add GPUs. Slow TTFT on long prompts = prefill-bound → chunked prefill.

Questions to ask: What's the served --max-model-len? Quantization? Tensor-parallel size? Is prefix caching on? What's gpu_cache_usage_perc under peak?

What silently gets expensive/unreliable: oversized --max-model-len (kills concurrency), no headroom (OOM), idle GPUs at low utilization, and no queue/latency alerts.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• vLLM bundles PagedAttention + continuous batching + prefix caching behind an OpenAI-compatible API.
• It serves safetensors (AWQ/GPTQ/FP8), not GGUF — GGUF is llama.cpp's lane (Phase 6.03).
• After weights load, remaining VRAM is the KV pool — --max-model-len × --max-num-seqs must fit it (Phase 6.02).
• Oversizing --max-model-len slashes concurrency — set it to real need.
• --tensor-parallel-size shards one model across GPUs in a node (NVLink); pipeline parallel spans nodes (what-happens §1.F).
• /metrics exposes gpu_cache_usage_perc (capacity) and num_requests_waiting (backpressure) — your two key signals (08).
• Continuous batching → ~3–5× throughput vs static (03).
• Enable prefix caching for repeated system prompts (05); KV quant (fp8) to fit more.

9. Observations from Real Systems

• vLLM is the most common self-hosted backend behind internal gateways and is offered by many inference-as-a-service providers under the hood (Phase 8).
• The OpenAI-compatible surface means a gateway/Cursor/BYOK client treats vLLM exactly like a cloud provider (Phase 6.04, Phase 11).
• Throughput claims in model/provider posts usually come from vLLM-class engines at high batch with continuous batching — read them with the batch caveat (Phase 4.03).
• Capacity incidents almost always show up as rising num_requests_waiting + gpu_cache_usage_perc near 1.0 — the textbook KV-ceiling signature (10).
• AWQ/FP8 on vLLM is a standard way to serve a bigger model on fewer GPUs (Phase 6.06).

10. Common Misconceptions

11. Engineering Decision Framework

DEPLOY a model on vLLM:
 1. Pick precision to FIT: AWQ/GPTQ/FP8 if needed (Phase 6.06); 1 GPU if it fits, else TP.
 2. Size the KV pool: choose --max-model-len = real context; --gpu-memory-utilization ~0.9;
    derive --max-num-seqs from KV math (Phase 6.02). Add --kv-cache-dtype fp8 if tight.
 3. Turn on: --enable-prefix-caching (shared prompts), --enable-chunked-prefill (long prompts).
 4. Scale: single GPU → tensor-parallel within node → pipeline-parallel across nodes.
 5. Observe: scrape /metrics; alert on gpu_cache_usage_perc≈1 and num_requests_waiting>0.   [08]
 6. Front with a gateway: routing, fallback, budgets, auth.                                  [07, Phase 8]

12. Hands-On Lab

Goal

Stand up vLLM, prove the throughput-vs-concurrency win, and find your KV-cache ceiling from /metrics.

Prerequisites

• A CUDA GPU (or a cloud GPU); pip install vllm openai httpx.

Setup

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --max-model-len 4096 --gpu-memory-utilization 0.85 \
  --enable-prefix-caching --enable-chunked-prefill

Steps

1. Sanity: call /v1/chat/completions with the OpenAI SDK (base_url=.../v1).
2. Concurrency sweep: fire 1, 8, 32, 64 concurrent streaming requests; record aggregate tok/s and p50/p95 TTFT. Throughput should climb, per-request TTFT degrade.
3. Watch capacity: during the 64-case, poll curl localhost:8000/metrics | grep -E "cache_usage|waiting|running"; note when num_requests_waiting rises and gpu_cache_usage_perc nears 1.0 — that's your ceiling.
4. Shrink the pool: restart with --max-model-len 1024; repeat 64 — more sequences fit (higher concurrency). Then try --max-model-len 16384 and show concurrency drops. Tie to Phase 6.02.
5. Prefix cache: send 32 requests sharing a 1,500-token system prompt with caching on vs off; compare TTFT (05).

Expected output

A table: concurrency → tok/s, p50/p95 TTFT, cache_usage, waiting; plus how --max-model-len trades context for concurrency; plus the prefix-cache TTFT delta.

Debugging tips

• Throughput flat → batching not engaging or already KV-capped (check metrics).
• OOM on startup → lower --gpu-memory-utilization or use a smaller/quantized model.

Extension task

Run a 7–8B model with --quantization awq and/or --kv-cache-dtype fp8; show how many more concurrent sequences fit (Phase 6.06).

Production extension

Add a second replica behind a load balancer and a tiny gateway with fallback; scrape both /metrics into Grafana (07, 08).

What to measure

tok/s and p50/p95 TTFT vs concurrency; KV-ceiling concurrency; --max-model-len effect; prefix-cache delta.

Deliverables

• A throughput-vs-concurrency table with the KV ceiling.
• A --max-model-len trade-off note (context vs concurrency).
• A prefix-cache TTFT comparison + chosen launch flags.

13. Verification Questions

Basic

1. What three innovations does vLLM bundle, and what does each do?
2. What weight format does vLLM serve (and not)?
3. After weights load, what is the rest of GPU memory used for?

Applied 4. Why does a large --max-model-len reduce concurrency? Tie it to the KV math. 5. Choose flags to serve a 32B on 2×24 GB GPUs for ~32 users at 4k context.

Debugging 6. /metrics shows gpu_cache_usage_perc≈1.0 and rising num_requests_waiting. Diagnosis and three fixes. 7. TTFT is huge only on long prompts. Which flag helps and why?

System design 8. Design a 2-replica vLLM deployment with tensor parallelism, prefix caching, autoscaling triggers, and gateway fallback.

Startup / product 9. How do --max-model-len, quantization, and prefix caching translate into gross margin?

14. Takeaways

1. vLLM is the self-hosting standard: PagedAttention + continuous batching + prefix caching behind an OpenAI-compatible API.
2. It serves safetensors (AWQ/GPTQ/FP8), not GGUF.
3. Remaining VRAM = KV pool; --max-model-len × --max-num-seqs set your concurrency ceiling — don't oversize context.
4. Scale with tensor parallelism (node) then pipeline parallelism (cross-node).
5. Watch gpu_cache_usage_perc + num_requests_waiting — the capacity and backpressure signals.

15. Artifact Checklist

• A running vLLM OpenAI-compatible endpoint + client snippet.
• A throughput-vs-concurrency table + identified KV ceiling.
• A --max-model-len / quant trade-off note.
• A prefix-cache TTFT comparison.
• The launch flags chosen for your workload, justified.

Up: Phase 7 Index · Next: 02 — TGI, SGLang, and TensorRT-LLM