Cheatsheet 09 — vLLM Commands
High-throughput GPU serving. Full: Phase 7.01. Lab: labs/lab-07-vllm-serving.
Install & serve (OpenAI-compatible)
pip install vllm
# Serve a model with an OpenAI-compatible API on :8000
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--dtype auto
# Call it like OpenAI
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"hi"}]}'
Key flags
| Flag | What | Notes |
|---|---|---|
--max-model-len | Max context length | Lower = more KV-cache room for batching |
--gpu-memory-utilization | Fraction of VRAM to use (0–1) | ~0.9; leave headroom |
--tensor-parallel-size N | Shard across N GPUs | For models bigger than one GPU |
| `--quantization awq | gptq | fp8` |
--max-num-seqs | Max concurrent sequences | Throughput vs latency tradeoff |
--enable-prefix-caching | Reuse shared prompt prefixes | Big win for RAG/system-prompt reuse (Phase 7.05) |
--enable-lora --lora-modules ... | Serve LoRA adapters | Multi-LoRA (Phase 13.07) |
--dtype | bf16/fp16/auto | Precision |
Why vLLM is fast (say this in interviews)
- PagedAttention — KV-cache managed in pages → little fragmentation → more concurrent sequences.
- Continuous batching — requests join/leave the batch every step → high GPU utilization.
- Prefix caching — shared prefixes (system prompt, RAG context) computed once.
Tuning for your goal
| Goal | Lever |
|---|---|
| Higher throughput | ↑ max-num-seqs, ↓ max-model-len, enable prefix caching |
| Lower latency (TTFT/TPOT) | ↓ batch pressure, smaller/quantized model, speculative decoding |
| Bigger model than 1 GPU | --tensor-parallel-size |
| Lower VRAM | quantization (awq/gptq/fp8), lower max-model-len |
Measure (always under load)
- TTFT, TPOT/ITL, p95/p99, throughput (tok/s), GPU utilization. Load-test, don't trust single-request numbers (Phase 12.06).
Next: 10 — llama.cpp Commands · Full: Phase 7.01