Lab 07 — vLLM Serving & Benchmark

Needs an NVIDIA GPU (or a cloud GPU). Serve a model with vLLM and benchmark throughput/latency under load. Concepts: Phase 7.01, cheatsheet 09.

Goal

Serve an open model with vLLM (OpenAI-compatible), then measure TTFT/TPOT/p95 and throughput vs concurrency — proving you understand continuous batching + KV-cache.

Prerequisites

pip install vllm

Steps

  1. Serve: 
    vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192 \
    --gpu-memory-utilization 0.90 --enable-prefix-caching
  2. Smoke test (OpenAI-compatible): 
    curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"hi"}]}'
  3. Load test: drive concurrency 1, 4, 16, 64 (use lab-12 load_test.py or vllm's benchmark scripts). Record TTFT, TPOT, p95, throughput, GPU util at each level.
  4. Tune: vary --max-model-len, --max-num-seqs, quantization (--quantization awq); re-measure.
  5. Prefix caching: send a shared 2k-token system prompt across requests with/without --enable-prefix-caching; measure the TTFT/cost difference.

Deliverables

  • Throughput vs concurrency curve (find the knee).
  • p95 latency vs concurrency.
  • A quant comparison (FP16 vs AWQ/GPTQ): quality (sanity) vs VRAM vs speed.
  • A short writeup explaining results via prefill/decode + KV-cache + continuous batching.

Why it matters (interview)

Serving/infra roles want load-tested numbers and the mechanism behind them (interview-prep 03–04). Cross-check your memory estimate with lab-06.