Hitchhiker's Guide — Quantization & Model Compression

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about making models small and fast."

The 30-second mental model

Weights are billions of fp16 numbers that waste most of their bits. Quantization snaps them to a small integer grid — x̂ = (q − z)·s — and stores the codes plus the scale. It wins on memory (int4 = ¼ the GPUs) and bandwidth (decode is bandwidth-bound, fewer bytes = more tok/s). It's lossy, and the loss isn't uniform: a few outlier/salient channels carry the signal, so the whole field (GPTQ, AWQ, SmoothQuant, NF4) is about where to spend the bits you have left. Two moves do most of the work: per-channel (free) and calibrated int4 (GPTQ/AWQ).

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "Will a 70B fit on one 80GB GPU?"
fp16 140 → no; int8 70 → barely (no KV room); int4 35 → yes (+ KV headroom)

# "How much faster does int4 decode vs fp16?"
decode ∝ bandwidth / weight_bytes → int4 ≈ 4× fewer bytes ≈ ~4× tok/s ceiling

# "One outlier of 100 in a [-1,1] int8 channel — what happens?"
s = 100/127 ≈ 0.8 → the [-1,1] bulk gets ~3 usable levels → use per-channel + AWQ

# "KV-cache too big at 100 users?"
int8 K/V halves 2·L·T·d·bytes·batch → 2× the concurrency for a small ppl hit

The framework one-liners (where these live in real tools)

# bitsandbytes — easiest path (weight-only int8 / 4-bit NF4)
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(name, load_in_4bit=True)          # NF4 base (QLoRA)
m = AutoModelForCausalLM.from_pretrained(name, load_in_8bit=True)          # LLM.int8()

# AWQ — activation-aware int4 for serving
from awq import AutoAWQForCausalLM
model.quantize(tokenizer, quant_config={"w_bit": 4, "q_group_size": 128, "version": "GEMM"})

# GPTQ — Hessian error-feedback int4
from auto_gptq import AutoGPTQForCausalLM   # needs a calibration dataset

# llama.cpp / GGUF — CPU/edge:  ./quantize model.gguf model-q4.gguf Q4_K_M
# TensorRT-LLM — SmoothQuant W8A8, fp8 on Hopper
# Check the win:  model.get_memory_footprint()

War stories

• The int4 that broke the eval. Team shipped naive int4 (per-tensor, min/max) because it "fit." Quality cratered — the outlier channels were crushed. Switching to AWQ int4 with group_size=128 recovered nearly all of it at the same size. The fix was calibration and granularity, not more bits.
• The "quantized so it's faster" that wasn't. A team quantized weights to int4 but kept the mixed-precision dequant path unfused; the kernel overhead ate the bandwidth win. Quantization helps only if the runtime fuses dequant into the matmul — measure tok/s, not just memory.
• The KV-cache they forgot to quantize. Weights fit after int4, but the service still OOM'd at peak concurrency — the KV-cache (Phase 00's wall) was the binding constraint. int8 K/V fixed it. Quantize the thing that's actually full.
• The activation int8 surprise. Weight int8 was free; switching to W8A8 (int8 activations too) tanked accuracy until they added SmoothQuant. Activations are the hard side — never assume the weight result transfers.

Vocabulary (rapid-fire)

• scale / zero-point — reals-per-step; the code that means 0.0.
• affine vs symmetric — fit [xmin,xmax] (activations) vs centred-on-0 z=0 (weights).
• per-tensor / per-channel / per-group — 1 scale total / per row / per block of 64–128.
• PTQ / QAT — quantize after training (calibrate) / during (fake-quant + straight-through).
• outlier / salient channel — the 10–100× dimensions that carry the signal.
• GPTQ — per-layer Hessian error-feedback weight quant; PTQ, no retraining.
• AWQ — scale salient weight columns by act**α to keep their bits.
• SmoothQuant — migrate activation difficulty into weights for int8 W8A8.
• LLM.int8() — int8 with an fp16 path for outlier dimensions.
• NF4 — 4-bit grid on normal quantiles (QLoRA); fp8 — 8-bit float, native on Hopper.

Beginner mistakes

• Quoting weight memory and forgetting to quantize the KV-cache (the real wall).
• Using per-tensor when per-channel is nearly free and far better.
• Naive min/max int4 and being surprised it's bad — int4 needs per-group + GPTQ/AWQ.
• Clipping outliers as if they were noise — they're the signal.
• Assuming weight int8 results transfer to activation int8 (they don't; that's the hard side).
• Reporting a precision without a measured eval number — quality vs bits is a cliff, not a slope.
• Quantizing weights but leaving the dequant kernel unfused and seeing no speedup.

The one thing to take away

Quantization is the highest-leverage inference knob — it attacks memory and bandwidth at once. The whole skill is where to spend the bits: go per-channel for free, use a calibrated method (GPTQ/AWQ) for int4, protect the outliers instead of clipping them, and always quote a measured quality number for the precision you ship.