Warmup Guide — Quantization & Model Compression

Zero-to-expert primer for Phase 03. Builds quantization from "what is a number format" through PTQ, QAT, and GPTQ — assuming only basic Python and the Phase 02 architecture vocabulary.

Table of Contents

Chapter 1: Number Formats — What Precision Actually Is

Zero background: computers store real numbers in fixed-width binary. A floating-point number is sign × mantissa × 2^exponent: the exponent gives range, the mantissa gives precision within that range. Key formats:

FormatBitsExponentMantissaRangePrecision
FP3232823±3.4e38~7 decimal digits
FP1616510±65504~3 digits — overflows easily
BF161687FP32's range~2 digits — never overflows where FP32 doesn't
INT88——[-128, 127]uniform steps
INT44——[-8, 7]16 values total

The crucial difference: float spacing is relative (numbers near 0 are dense, large numbers sparse); integer spacing is absolute (uniform grid). Quantization is the act of mapping the float world onto an integer grid — and choosing where that grid sits is the entire game.

Production reality: INT8/INT4 are not just smaller — NPU/GPU integer pipelines have 2–8× the throughput of float pipelines, and (more important on edge) memory traffic drops proportionally with bit-width. FP8 (E4M3/E5M2) is the newest middle ground on H100-class and recent NPU hardware.

Chapter 2: Why Quantize — The Memory Wall

A 7B-parameter model in FP16 is 14 GB of weights. To generate one token you must read every weight once — at batch 1, LLM decoding is memory-bandwidth bound, not compute-bound (Phase 07 formalizes this with roofline analysis). So:

$$\text{tokens/sec} \approx \frac{\text{memory bandwidth}}{\text{bytes of weights}}$$

Halve the bytes → roughly double the decode speed, independent of compute. INT4 cuts a 7B model to 3.5 GB — the difference between "doesn't fit on the phone" and "fits with room for the KV cache." That is why quantization is the edge-deployment skill, and why this phase exists.

The price: every weight moves to its nearest grid point. The question of this whole phase is: how much accuracy does that cost, and how do you minimize it? Naive INT8 can take a 7B model's MMLU from 72% to 55%; well-engineered INT4 can hold it within a point. The technique gap, not the bit-width, determines the outcome.

Chapter 3: Uniform Quantization — The Core Math

The affine map. Choose a scale $s$ and zero-point $z$:

$$q = \text{clamp}!\left(\text{round}!\left(\frac{x}{s}\right) + z,; q_{min},; q_{max}\right), \qquad \hat{x} = s,(q - z)$$

  • $s$ = step size of the integer grid in real units: $s = (x_{max} - x_{min})/(2^b - 1)$.
  • $z$ = which integer represents real 0. Symmetric quantization fixes $z = 0$ (grid centered on zero, signed ints) — simpler hardware, no zero-point multiplies in the matmul inner loop. Asymmetric allows $z \neq 0$ — fits one-sided distributions (post-ReLU activations live in $[0, +)$; symmetric wastes half the grid on negatives).
  • Convention: weights symmetric per-channel, activations asymmetric per-tensor is the standard hardware-friendly starting point.

The two error sources:

  1. Rounding error: uniform in $[-s/2, s/2]$, variance $s^2/12$. Shrinks as range shrinks.
  2. Clipping error: values outside $[x_{min}, x_{max}]$ saturate. Shrinks as range grows.

These trade against each other through $s$ — that tension is calibration (Chapter 5). One outlier weight of 50 in a tensor whose other values live in $[-1, 1]$ forces $s$ 50× larger, multiplying everyone else's rounding error by 50. Outliers are the villain of this entire phase.

Quantized matmul: $Y = XW \approx s_x s_w (Q_x Q_w)$ — the inner product runs in integer arithmetic (accumulating in INT32), with scales applied once at the end. This is what NPU MAC arrays actually execute.

Chapter 4: Granularity — Per-Tensor, Per-Channel, Per-Group

How many values share one $(s, z)$ pair?

  • Per-tensor: one scale for the whole weight matrix. Cheapest; one outlier row ruins every row.
  • Per-channel: one scale per output row of $W$. Each row's range fits its own distribution — the single biggest free win for weights; supported by essentially all inference hardware.
  • Per-group: one scale per contiguous group of (typically 128) weights within a row — the GPTQ/AWQ INT4 standard. Finer adaptation; cost = storing one FP16 scale per 128 INT4 values (≈ +3% memory).

Rule of thumb: INT8 weights → per-channel suffices. INT4 weights → per-group is required. Activations → per-tensor (dynamic per-token helps but costs runtime calibration).

Chapter 5: Calibration — Choosing the Range

Run a few hundred representative samples through the model, observe activation distributions, choose $[x_{min}, x_{max}]$ per quantizer:

  • MinMax: exact observed extremes. Zero clipping, maximal rounding error; one outlier ruins it. Right for weights (no sampling noise — you can see all of them).
  • Percentile (99.9th): clip the tail outliers deliberately. Usually beats MinMax for activations.
  • Entropy / KL-divergence (TensorRT's classic): choose the clip threshold whose quantized distribution diverges least from the original — principled rounding-vs- clipping balance.
  • MSE: directly minimize $|x - \hat{x}|^2$ over candidate clip values.

Engineering reality: calibration-set choice matters as much as method — it must cover the deployment input distribution. A model calibrated on English prose mis-ranges on code tokens. Lab 01 makes you implement three methods and measure the differences instead of trusting folklore.

Chapter 6: PTQ — The Baseline Pipeline

Post-Training Quantization = freeze the trained model, calibrate, quantize, ship. No gradients, minutes not days. The pipeline you build in Lab 01:

  1. Wrap target layers (QuantizedLinear replacing nn.Linear).
  2. Attach observers; run calibration batches; collect ranges.
  3. Compute $(s, z)$ per quantizer from the chosen calibration rule.
  4. Quantize weights offline; activations quantize at runtime (or simulate with QDQ — quantize-dequantize — nodes for accuracy evaluation in float).
  5. Measure: perplexity and task accuracy vs the FP16 baseline, per configuration.

When PTQ suffices: INT8 per-channel on most transformer weights — typically <0.5 PPL degradation. When it fails: INT4 without grouping; activation quantization in the presence of outliers; small models (less redundancy to absorb error); anything where the calibration set misses the deployment distribution.

Chapter 7: QAT and the Straight-Through Estimator

The idea: if quantization will distort the network, let training see the distortion and route around it. Insert fake-quantize nodes — $\hat{x} = s(\text{round}(x/s) - z)$ computed in float — into the forward pass and fine-tune.

The problem: $\text{round}(\cdot)$ has derivative 0 almost everywhere (and undefined at half-integers). Backprop through it kills all gradients.

The Straight-Through Estimator: in the backward pass, pretend round is the identity:

$$\frac{\partial,\text{round}(x)}{\partial x} \overset{\text{STE}}{=} 1 \quad (\text{within the clip range; 0 outside})$$

Mathematically false; empirically it works because round is "identity plus bounded noise" ($|round(x) - x| \le 0.5$), so the identity's gradient points in approximately the right direction, and SGD's stochasticity absorbs the bias. The clip-range masking matters: saturated values get zero gradient (they genuinely can't move the output), which also nudges weights back inside the representable range.

What QAT buys: weights migrate toward grid points and away from clip boundaries during fine-tuning — typically +1–3% accuracy over PTQ at INT8, and it's the main tool that makes INT4/INT2 activations viable. Cost: a training loop, data, and time — which is why PTQ-family methods (GPTQ/AWQ) dominate for LLMs where fine-tuning is expensive.

Lab 02 implements FakeQuant as a torch.autograd.Function with the STE backward and an observer that tracks running ranges — the same architecture as torch.ao.quantization's internals.

Chapter 8: GPTQ — Second-Order Quantization

The question GPTQ answers: when you round one weight, can you adjust the other weights to compensate? Yes — optimally, using curvature information.

Setup: quantize layer-by-layer, minimizing layer output error $|XW - X\hat{W}|^2$ over the calibration activations $X$. The curvature of this objective w.r.t. the weights of one output row is the Hessian $H = 2XX^\top$ (+ small damping $\lambda I$ for stability) — note it depends only on inputs, shared across rows.

The OBQ step, for one weight $w_q$ (from the Optimal Brain Surgeon lineage): quantize it, then update all not-yet-quantized weights in the row by

$$\delta = -\frac{w_q - \text{quant}(w_q)}{[H^{-1}]{qq}} , [H^{-1}]{:,q}$$

— spread the rounding error onto correlated weights, weighted by inverse curvature (move most where the loss is flattest).

GPTQ's algorithmic contributions that make this feasible at LLM scale:

  1. Quantize columns in a fixed order for all rows (vectorizes across the matrix).
  2. Maintain $H^{-1}$ updates via Cholesky factorization (numerically stable; the naive rank-1 inverse updates accumulate error and produce garbage at 175B scale).
  3. Lazy-batch the updates for memory locality.

Result: a 175B model quantized to INT4/3-bit in hours on one GPU, at near-FP16 perplexity. Lab 03 implements the per-layer algorithm: Hessian from calibration hooks, column-wise quantize-and-compensate, and the comparison vs round-to-nearest that shows why the compensation matters (typically 2–5× lower reconstruction error).

Chapter 9: The Activation Outlier Problem — AWQ and SmoothQuant in Brief

Transformer activations past ~6B parameters develop systematic outlier channels — specific hidden dimensions with values 10–100× the median, consistently across tokens (an emergent property tied to attention "sink" behavior). Weights quantize easily; these activations don't.

  • SmoothQuant: migrate difficulty from activations to weights with a per-channel rescale: $Y = (X,\text{diag}(s)^{-1})(\text{diag}(s),W)$ — mathematically identical, but $X' = X/s$ has tamed ranges while $W' = sW$ absorbs them (weights have headroom). Choose $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$, α ≈ 0.5.
  • AWQ: observe that ~1% of weight channels are salient (they multiply the high-magnitude activations); protect them by scaling up before quantization (equivalent trick in the other direction). No Hessian, no backprop — hardware-friendly and fast. You implement AWQ fully in Phase 10 Lab 01.

These two plus GPTQ form the LLM INT4 toolbox; Chapter 8's deep dive file (DEEP-DIVE-QAT-GPTQ.md) covers the math at full depth.

Lab Walkthrough Guidance

Order: Lab 01 (PTQ) → Lab 02 (QAT) → Lab 03 (GPTQ) — each builds on the previous's vocabulary.

  • Lab 01: implement quantize/dequantize and verify the round-trip error bound ($\le s/2$ per element) before touching calibration. Then observers, then the three calibration strategies, then the accuracy table. Resist evaluating before all three are in — the comparison is the lab.
  • Lab 02: write the STE autograd.Function first and gradcheck your own understanding: the gradient should be 1 inside the clip range, 0 outside. Then the observer, then training. Verify QAT > PTQ on the final table.
  • Lab 03: build compute_hessian and check positive-definiteness (damping!). Then single-column quantize-with-compensation on a toy 4×4 case you can verify by hand. Then the full loop, then compare_reconstruction_error — GPTQ must beat round-to-nearest clearly.

Success Criteria

You are ready for Phase 04 when you can, from memory:

  1. Write the affine quantization map and derive $s$ from a range and bit-width; state the rounding-vs-clipping tension in one sentence.
  2. Explain why weights are symmetric-per-channel and activations asymmetric-per-tensor in standard deployments.
  3. State the STE, why it's wrong, and why it works anyway.
  4. Write GPTQ's compensation update and explain each factor ($[H^{-1}]_{qq}$ in the denominator = move most where curvature is flattest).
  5. Explain the activation-outlier phenomenon and the SmoothQuant rescaling identity.
  6. Given "MMLU dropped 5% at INT4", produce the ordered debugging tree: granularity → calibration → SmoothQuant/AWQ → GPTQ → QAT.

Interview Q&A

Q: Why does per-channel weight quantization dominate per-tensor? Output channels of a trained layer have wildly different weight ranges (often 10–50× spread). Per-tensor forces one scale to cover the largest channel, inflating everyone else's step size. Per-channel scales are free at inference (folded into the output rescale) — better accuracy at zero runtime cost.

Q: Decode throughput is memory-bound. Walk me through the INT4 speedup math. Decode reads all weights per token: $t \approx \text{bytes}/\text{BW}$. 7B FP16 = 14 GB; at 100 GB/s that's 140 ms/token ≈ 7 tok/s. INT4 (+group scales ≈ 3.6 GB) → ~28 tok/s. Compute barely changes; we cut bytes 4×. This is also why INT4 weights with FP16 compute (dequantize-on-load) still wins on bandwidth-bound hardware.

Q: When does QAT fail to help? At extreme bit-widths (INT2) where the function space is too coarse for STE's gradient fiction; when the fine-tuning data distribution drifts from deployment; and when the bottleneck is activation outliers (a range problem QAT's weight adaptation can't fix — you need SmoothQuant-style migration first).

Q: Why does GPTQ use the Hessian rather than weight magnitude? Magnitude says how big a weight is, not how much the output changes when you perturb it — that's curvature. $H = 2XX^\top$ measures exactly the output sensitivity under the calibration distribution, so error is spread onto directions the layer is least sensitive to. Magnitude-based rounding is precisely the "naive" baseline GPTQ beats.

Q: NPU memory layout implication of group quantization? Each group's FP16 scale must be co-resident with its INT4 weights during the MAC — layouts interleave scales with weight blocks so a tile fetch brings both; misalignment doubles memory transactions. group_size is chosen to match the NPU's native tile width (often 64 or 128) for exactly this reason.

References

  • Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017) — arXiv:1712.05877
  • Nagel et al., A White Paper on Neural Network Quantization (2021) — arXiv:2106.08295 — the single best survey
  • Bengio et al., Estimating Gradients Through Stochastic Neurons (STE) (2013) — arXiv:1308.3432
  • Frantar et al., GPTQ: Accurate Post-Training Quantization for GPT (2022) — arXiv:2210.17323
  • Hassibi & Stork, Optimal Brain Surgeon (1992) — the lineage behind GPTQ
  • Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication and Emergent Outliers (2022) — arXiv:2208.07339
  • Xiao et al., SmoothQuant (2022) — arXiv:2211.10438
  • Lin et al., AWQ: Activation-aware Weight Quantization (2023) — arXiv:2306.00978
  • PyTorch quantization docs
  • DEEP-DIVE-QAT-GPTQ.md — this phase's companion deep dive