Cheatsheet 06 — Quantization
Storing weights in fewer bits to fit/run models cheaply. Full: Phase 6.06.
The idea
Weights are normally FP16 (2 bytes). Quantization stores them in fewer bits (INT8, INT4…) → less memory, faster (bandwidth-bound) decode, small quality loss. Memory ≈ params × bytes-per-param.
Memory rule of thumb (model weights only)
| Precision | Bytes/param | 7B model | 70B model |
|---|---|---|---|
| FP16/BF16 | 2.0 | ~14 GB | ~140 GB |
| INT8 (Q8) | ~1.0 | ~7 GB | ~70 GB |
| INT4 (Q4) | ~0.5 | ~4 GB | ~35 GB |
Add KV-cache + activations on top (grows with context × batch). Rule:
tok/s ≈ memory bandwidth ÷ model bytes.
Formats & methods
| Name | What | Where |
|---|---|---|
| GGUF (Q4_K_M, Q5_K_M, Q8_0…) | k-quant formats for llama.cpp/Ollama | Local CPU/GPU/Mac (cheatsheet 10) |
| AWQ | Activation-aware weight quant (4-bit, good quality) | GPU serving (vLLM) |
| GPTQ | Post-training quant (4-bit) | GPU serving |
| FP8 | 8-bit float (newer GPUs) | High-throughput serving |
| QAT | Quantization-aware training (best quality, costly) | When quality-critical |
| bitsandbytes (NF4) | 4-bit for QLoRA fine-tuning | Phase 13.02 |
Choosing a GGUF quant (local)
- Q4_K_M — best size/quality balance; the default to try.
- Q5_K_M / Q6_K — higher quality, more memory.
- Q8_0 — near-lossless, ~1 byte/param.
- Q3/Q2 — only if desperate for memory; quality drops noticeably.
Tradeoffs
- ✅ Less VRAM/RAM, faster decode, run bigger models on smaller hardware.
- ⚠️ Some quality loss (grows below 4-bit); worse for tasks needing precision (math/code can degrade).
- 🔁 Composes with distillation — distill a small model then quantize it (Phase 13.04).
Interview lines
- "Quantization trades a little quality for big memory/latency wins because decode is bandwidth-bound — fewer bytes per param means more tokens/sec."
- "For local I default to Q4_K_M GGUF; for GPU serving, AWQ/GPTQ 4-bit or FP8; for fine-tuning on a budget, QLoRA (NF4)."
Next: 07 — Inference Parameters · Full: Phase 6.06