Lab 01 — Quantize & Benchmark

Goal: take one model, produce it in FP16, INT8, and INT4, and generate a single table of latency, throughput, memory, and accuracy for each — plus a Pareto plot. This is the JD's "quantization … with clear trade-off analysis" turned into an artifact you can show.

Knowledge prereq: K02 — Quantization, K06 — Profiling & Benchmarking.

Stack: transformers, vllm (or auto-gptq/autoawq/llm-compressor), lm-eval (or a small custom eval), a CUDA GPU (A10/L4/A100-class is plenty for a 7–8B model).

Steps

  1. Pick a model you can run: meta-llama/Meta-Llama-3-8B-Instruct or mistralai/Mistral-7B-Instruct. Note its config: params, n_kv_heads (GQA?), layers — you'll cite these.

  2. Baseline (FP16): serve with vLLM, run run_benchmark.py (below). Record p50/p95 TTFT and TPOT, throughput (tok/s) at a fixed concurrency, and peak VRAM.

  3. Accuracy baseline: run a small eval — lm-eval on a slice of MMLU/GSM8K, or 50 domain questions with an LLM-judge. Record the score. This is your floor.

  4. INT8: produce an INT8 weight-only (or W8A8/FP8) version (vLLM --quantization fp8, or AWQ/GPTQ via autoawq/auto-gptq, or llm-compressor). Re-run steps 2–3.

  5. INT4: produce a W4 version (AWQ or GPTQ, group size 128). Re-run steps 2–3.

  6. Tabulate (this is the deliverable):

    PrecisionVRAMp95 TTFTp95 TPOTThroughputMMLU (or domain eval)Δ accuracy
    FP16……………baseline
    INT8/FP8………………
    INT4………………

  7. Plot the Pareto frontier: throughput (or $/token) on x, accuracy on y. Mark which precision you'd recommend for (a) a chat workload, (b) a math/code workload. Justify using K02 §11–12.

  8. Write the conclusion (3–5 sentences): which precision wins for which use case, the accuracy cost, and why — the honest report a customer would get.

What you must be able to explain

  • Why INT8 roughly doubles decode throughput (memory-bound: half the weight bytes → K01 §4).
  • Why INT4 needs group-wise scales and more accuracy care (K02 §3,9).
  • Which task you'd worry about most under quantization (reasoning/math/code cliffs — K02 §11).

Result / résumé bullet

"Benchmarked Llama-3-8B across FP16/INT8/INT4 (AWQ) on an A100: INT8 delivered 1.9× decode throughput and 0.5× VRAM for −0.4 pts MMLU; INT4 reached 3.4× throughput at −2.1 pts, with a documented Pareto frontier and per-workload recommendation."

See run_benchmark.py for a runnable serving-benchmark harness.