Phase 06 — Quantization & Model Compression

The phase that turns Phase 00's memory and bandwidth arithmetic into a lever you can pull. You proved decode is memory-bandwidth-bound and the KV-cache is the inference wall; quantization is the single most effective response to both — it shrinks the bytes you store and the bytes you move per token. This is the phase that gets a 70B model onto one GPU, and it is asked, in some form, in nearly every senior inference interview.

Why this phase exists

A frozen model is a giant table of numbers stored in fp16 (2 bytes each). Almost none of those bits carry information the model actually uses — and you pay for every one of them twice: once in memory (weights + KV-cache must fit in HBM) and once, per token, forever, in bandwidth (decode re-reads every weight from HBM, and decode is bandwidth-bound, so fewer bytes per weight = more tokens per second). Quantization — storing the weights as int8 or int4 and reconstructing them on the fly — is the highest-leverage compression we have:

1. Memory — int8 halves the weights, int4 quarters them. 70B × 2 = 140 GB (two GPUs) becomes 70B × 0.5 = 35 GB (one GPU). This is an architecture change, not a micro-optimization.
2. Bandwidth / speed — decode throughput ≈ bandwidth / weight_bytes (Phase 00). Halve the bytes, roughly double the tokens/second. Quantization is a latency technique, not just a memory one.
3. The catch — fewer bits is a lossy compression, and the loss is not uniform: a handful of outlier / salient channels carry most of the signal, and a naive quantizer crushes them. The whole research field (GPTQ, AWQ, SmoothQuant, LLM.int8(), NF4) is about where to spend the bits you have left.

Get fluent here and "can we fit this / make it faster?" stops being a vendor question and becomes a calculation you do on a whiteboard.

Concept map

                 ┌──────────────────────────────────────────┐
                 │  fp16 weights waste memory AND bandwidth   │
                 └──────────────────────────────────────────┘
                       │                         │
            ┌──────────┘                ┌────────┘
            ▼                           ▼
     map reals → int grid        why it's lossy: OUTLIERS
     scale · zero-point          a few channels carry the signal
     round + CLAMP                       │
            │                  ┌─────────┴──────────┐
            ▼                  ▼                    ▼
   GRANULARITY            WEIGHT-SIDE          ACTIVATION-SIDE
   per-tensor             GPTQ (Hessian,       SmoothQuant (migrate
   per-channel ◀─ soul    error feedback)      difficulty W↔X)
   per-group              AWQ (scale salient ◀─ soul
            │              channels by act**α)        │
            └───────────────┬───────────────────────┘
                            ▼
                 PTQ (calibrate) vs QAT (train)
                 int8 · int4 · NF4 quality cliffs
                 KV-cache quant · the runtimes
            (bitsandbytes · GPTQ · AWQ · GGUF · TensorRT)

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. Its two soul tests — per-channel beats per-tensor and AWQ beats naive on a salient channel — are exactly the two facts you'll be asked to defend.

Integrated scenario ideas

• Fit the 70B: take Phase 00's weight_memory_bytes, quantize to int8 then int4, and show the GPU count drop from 2 → 1. Then add the KV-cache and decide whether to quantize that too.
• Defend a precision choice: a customer wants "the smallest model that passes our eval." Quantize a candidate to int8 and int4, measure the error/quality gap, and write the ADR with the cliff you found ("int8 was free; int4 cost 0.6 perplexity, which fails the eval").
• Explain the outlier problem to a new hire: show that one salient channel forces a coarse per-tensor scale, then fix it with per-channel and again with AWQ — three lines of reasoning that cover half the field.
• Choose a runtime: bitsandbytes (easy, weight-only int8/int4) vs GPTQ/AWQ (calibrated, faster) vs GGUF (CPU/edge) vs TensorRT (max GPU throughput) — pick one for a given deployment and justify it.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can derive scale and zero_point for "map [xmin, xmax] onto [qmin, qmax]" and say why symmetric forces zero_point = 0.
• You can explain the scale/2 round-trip bound, and why one outlier wrecks a per-tensor scale.
• You can explain why per-channel beats per-tensor, and where per-group sits between them.
• You can explain the salient-channel problem and how GPTQ, AWQ, and SmoothQuant each attack it (weight-side error feedback, weight scaling, difficulty migration).
• You can place int8 / int4 / NF4 on a quality-vs-size curve and name the cliff.

Key takeaways

• Quantization is a memory and a bandwidth win. Fewer bytes per weight means more weights per GPU and more tokens per second on bandwidth-bound decode. It is the highest-leverage knob in inference, which is why it's a whole phase.
• The bits aren't uniformly useful — outliers are. A few salient channels carry most of the signal; the entire post-LLM.int8() research arc (GPTQ, AWQ, SmoothQuant) is about protecting them.
• Granularity is the cheapest lever. Per-tensor → per-channel → per-group trades a little metadata for a lot of accuracy; per-channel weight quant is the free default.
• PTQ is usually enough; QAT is the fallback. Post-training quantization with good calibration (GPTQ/AWQ) gets int4 working without retraining; you reach for quantization-aware training only when PTQ leaves quality on the table.
• int8 is nearly free, int4 has a cliff, NF4 softens it. Know where your model falls off, and measure it — never quote a precision without an eval number.

Next: Phase 07 — Alignment: RLHF, DPO & Preference Optimization.