Knowledge 02 — Quantization

Why this is half the job: JD2 lists "Quantization techniques (INT8 / mixed precision)" as a core duty. Quantization is the single highest-leverage inference optimization: it cuts memory 2–4×, raises throughput 2–4× (decode is memory-bound, so fewer bytes = directly faster), and lets a model fit on fewer/cheaper accelerators. The art is doing it without breaking accuracy — and being able to prove you didn't. This module takes you from "what is a number format" to "I can explain why SmoothQuant works and when AWQ beats GPTQ."

Table of Contents

• 1. The core idea: representing floats with integers
• 2. The quantization equation (do this by hand once)
• 3. Granularity: per-tensor, per-channel, per-group
• 4. Symmetric vs asymmetric; static vs dynamic
• 5. What to quantize: weights, activations, KV cache
• 6. PTQ vs QAT
• 7. Calibration
• 8. The outlier problem — why naive INT8 breaks LLMs
• 9. The named algorithms: LLM.int8, SmoothQuant, GPTQ, AWQ, FP8, INT4
• 10. Mixed precision and sensitivity analysis
• 11. Measuring accuracy loss honestly
• 12. The decision framework (what a principal recommends)
• 13. References

1. The core idea: representing floats with integers

A trained model stores weights as 16- or 32-bit floats. Most of those bits are wasted: the weights in any given layer cluster in a narrow range, and the model is robust to small perturbations. Quantization maps a range of floating-point values onto a small set of integers (e.g., 256 values for INT8, 16 for INT4), storing the integer plus a scale factor to get back to the float.

The payoff is mechanical and follows straight from Knowledge 00:

• Memory: INT8 weights are ¼ the size of FP32, ½ of FP16. The model fits in less HBM, so fewer GPUs.
• Bandwidth: decode reads weights from HBM every token and is bandwidth-bound → half the bytes ≈ ~2× decode throughput.
• Compute: tensor cores run INT8/FP8 at ~2× the FP16 rate and INT4 faster still → helps compute-bound prefill too.

The cost is precision: you've thrown away bits, so values are approximated. Done carelessly this wrecks accuracy; done well it's nearly free. The rest of this module is "done well."

2. The quantization equation (do this by hand once)

Linear (affine) quantization maps a real value r to an integer q:

q = round(r / s) + z         (quantize)
r ≈ s · (q − z)              (dequantize)

• s (scale): a positive float, the size of one integer step. s = (r_max − r_min) / (q_max − q_min).
• z (zero-point): the integer that represents real 0. For symmetric quantization z = 0.
• For INT8 symmetric: q ∈ [−127, 127], s = max(|r|)/127, z = 0.

Worked example. A weight column has values in [−0.6, 0.4], INT8 symmetric.

• max(|r|) = 0.6 → s = 0.6/127 ≈ 0.004724.
• Quantize r = 0.31: q = round(0.31/0.004724) = round(65.6) = 66.
• Dequantize: 0.004724·66 = 0.3118. Error = 0.0018 — the rounding error, bounded by s/2. That bounded error, summed over a matmul, is the entire accuracy story: keep s small (tight range) and errors stay small.

Why a matmul still works in INT8: Σ (s_w·q_w)(s_a·q_a) = s_w·s_a·Σ q_w·q_a. The integer products Σ q_w·q_a accumulate in INT32 (no overflow), then you multiply by the scales once at the end. The expensive inner loop is pure integer MAC; the float scales are applied at the margins. This is why INT8 matmul is fast and representable on integer hardware (and why Qualcomm AI100-class INT8 cores are efficient).

If you can derive the line above on a whiteboard, you understand quantization better than most people who use it. Practice it once by hand.

3. Granularity: per-tensor, per-channel, per-group

The scale s covers some set of values. Bigger sets = fewer scales to store, but one outlier stretches the range and coarsens everyone's steps. The granularity choice is the central accuracy/overhead trade-off.

Rule of thumb: weights → per-channel (free accuracy, negligible overhead) and per-group (groups of 64–128) for INT4. Activations → per-tensor or per-token (they're computed at runtime, so per-channel is awkward). Knowing which axis to put the scale on is most of the practical skill.

The principle: put the scale boundary where the outliers are so they don't pollute the well-behaved values. In LLMs, outliers live in specific channels (§8) — hence per-channel weights and the SmoothQuant/AWQ tricks that target channels.

4. Symmetric vs asymmetric; static vs dynamic

Symmetric (z=0): range centered on 0, simplest, INT8 weights almost always symmetric. Asymmetric (z≠0): range can be offset (e.g., post-ReLU activations that are all ≥0) — better range utilization for skewed distributions, slightly more compute.

Static quantization: scales for activations are fixed ahead of time using a calibration pass (§7). Fast at runtime (no per-batch scale computation), but scales must generalize. Dynamic quantization: activation scales computed on the fly per batch/token from the actual values. More accurate (adapts to each input), slightly slower (compute max each step). Weights are always static (they don't change); the static/dynamic choice is about activations.

Practical default: weight-only INT8/INT4 with dynamic or per-token activation handling is the easiest big win for LLM decode (it attacks the memory-bound weight reads with minimal accuracy risk). Full static INT8 weight+activation (W8A8) is more aggressive — needed for max throughput and for integer-only accelerators — and needs calibration + outlier handling (§8–9).

5. What to quantize: weights, activations, KV cache

Three distinct targets, each with its own difficulty:

1. Weights — easiest and highest value. Static, known offline, well-behaved distributions. Weight-only INT8/INT4 (W8A16/W4A16) attacks decode's memory bottleneck directly. GPTQ/AWQ live here.
2. Activations — harder. Computed at runtime, input-dependent, and full of outliers in LLMs (§8). Needed for W8A8 (integer matmul on both operands → max throughput, mandatory for integer-only accelerators like AI100). SmoothQuant and FP8 live here.
3. KV cache — high leverage for long context / high concurrency. Quantizing cached K/V to INT8/FP8 halves the KV-cache memory → 2× concurrency or context. Small quality cost; increasingly standard.

Notation you'll see: W8A8 = 8-bit weights, 8-bit activations. W4A16 = 4-bit weights, 16-bit activations (weight-only). W8A8-KV8 adds INT8 KV. Read these fluently.

6. PTQ vs QAT

PTQ (Post-Training Quantization): take a trained FP16 model and quantize it without retraining, using a small calibration set to set scales. Fast (minutes–hours), no training pipeline, no labeled data. This is 90% of what you'll do — GPTQ, AWQ, SmoothQuant, FP8 are all PTQ. Risk: accuracy drop on aggressive settings (INT4, W8A8 with outliers).

QAT (Quantization-Aware Training): simulate quantization during training/fine-tuning (insert "fake quant" ops so the model learns weights robust to quantization). Recovers more accuracy at low bit-widths (INT4/INT2) but needs the training pipeline, data, compute, and time. Use when PTQ can't hit the accuracy floor — typically aggressive quantization or accuracy-critical deployments.

Decision rule: Start with PTQ. Escalate to QAT only when PTQ misses the accuracy bar and the deployment justifies the cost. Qualcomm's AIMET supports both; NVIDIA's TensorRT Model Optimizer, AutoGPTQ, llm-compressor (vLLM), bitsandbytes are the common toolchains.

7. Calibration

Static quantization needs to know the range of activations to set scales — but activations depend on input. Calibration runs a small, representative dataset (typically 128–1024 samples) through the FP16 model and records activation statistics to compute scales.

What can go wrong (and what you'll debug):

• Unrepresentative calibration data → scales don't generalize → accuracy collapses on real traffic. Use in-domain data (a customer's real prompts beat generic text).
• Range estimation method: min/max (sensitive to outliers), percentile (clip the top 0.1%), entropy/KL (TensorRT's default — minimizes information loss), MSE. Percentile/entropy usually beat naive min/max because they clip the rare extreme outliers rather than letting them stretch the scale.
• Too few samples → noisy scales. Too many → diminishing returns. 256–512 is a common sweet spot.

Customer-facing tell: "What does your real traffic look like?" isn't small talk — calibrating INT8 on the customer's actual prompt distribution is the difference between 0.3% and 5% accuracy loss. This is where domain access becomes technical leverage.

8. The outlier problem — why naive INT8 breaks LLMs

This is the single most important insight in LLM quantization, and a favorite interview probe.

Empirically (Dettmers et al., 2022), transformer activations contain rare but massive-magnitude outlier features — a handful of channels whose values are 10–100× larger than the rest, and they're systematic (same channels across tokens) and important (they carry real signal). With per-tensor INT8, one channel at magnitude 60 while everyone else is ~1 forces s = 60/127 → every normal value quantizes to 0, 1, or 2 → catastrophic precision loss → the model degrades sharply above ~6.7B params where these outliers emerge.

The fixes are all about isolating or taming the outliers:

• Keep outliers in higher precision (LLM.int8(): do the outlier columns in FP16, the rest in INT8 — "mixed-precision decomposition").
• Move the difficulty from activations to weights (SmoothQuant: scale outlier channels down in activations and up in weights, since weights quantize more easily — "smooth" the difficulty across the matmul).
• Protect the salient weights (AWQ: detect the ~1% of weight channels that matter most for output and scale them to preserve precision).

Whiteboard answer to "why can't you just round LLM weights to INT8?": "Weights are fine — they're well-behaved, per-channel INT8 is nearly lossless. The problem is activations: LLMs develop a few systematic outlier channels 10–100× larger than the rest, and per-tensor scaling lets those outliers crush the resolution of everything else. So either keep the outliers in FP16 (LLM.int8), migrate the difficulty into the weights (SmoothQuant), or protect the salient channels (AWQ)." That paragraph is a hire signal.

9. The named algorithms: LLM.int8, SmoothQuant, GPTQ, AWQ, FP8, INT4

Know each by mechanism, target, and when to use. This table is interview gold.

The two families to never confuse:

• Weight-only (W4A16: GPTQ, AWQ, GGUF) — quantize weights, compute in FP16. Attacks the memory/bandwidth bottleneck (great for decode). Easy, robust, popular. Doesn't speed up compute-bound prefill much.
• Weight+activation (W8A8: SmoothQuant, FP8, LLM.int8) — quantize both, integer/FP8 matmul. Attacks compute too (great for prefill, mandatory for integer-only accelerators) but harder (activation outliers).

Qualcomm AI100 note: integer-first accelerators want W8A8/W4A8 (both operands low-bit) to use their integer matmul units, so SmoothQuant-style activation handling + AIMET calibration is the relevant path — not just weight-only. Knowing that the hardware dictates the quant scheme is exactly the model↔hardware bridge JD2 is about.

10. Mixed precision and sensitivity analysis

"Mixed precision" (a JD phrase) means not every layer gets the same precision. Some layers are sensitive (quantizing them hurts a lot); some are robust. A principal does sensitivity analysis:

1. Quantize one layer at a time, measure accuracy delta on an eval set.
2. Rank layers by sensitivity.
3. Keep sensitive layers (often the first/last layers, embeddings, and the LM head) in higher precision (FP16/INT8); push robust middle layers to INT4.
4. Result: a per-layer bit-width assignment on the accuracy/size Pareto frontier.

Common findings: the LM head and embeddings are sensitive (keep ≥INT8), attention is often more robust than you'd guess, and the first couple and last couple of transformer blocks matter most. Tools (AIMET, TensorRT Model Optimizer, llm-compressor) can automate the sweep.

This is the literal meaning of "trade-off analysis between accuracy vs compatibility, precision vs efficiency" in the JD. You produce a curve, not a yes/no.

11. Measuring accuracy loss honestly

Quantization is only "free" if you prove the accuracy held. Three layers of evidence, weakest to strongest:

1. Perplexity (on WikiText/C4): cheap, catches gross breakage, but weakly correlated with task quality. A 0.1 perplexity rise can hide a real task regression. Necessary, not sufficient.
2. Task benchmarks (MMLU, GSM8K, HumanEval, the customer's domain eval): the real signal. Run the quantized vs FP16 model on the actual use case. A "2% MMLU drop" may be fine for chat, fatal for medical triage.
3. Side-by-side on the customer's traffic + human/LLM-judge eval (the original folder's Lab 5): the deployment-grade proof. Catches subtle degradation (formatting, refusals, edge cases) that aggregate metrics miss.

Honest reporting (a principal habit): report the eval set, the metric, the FP16 baseline, the quantized number, and the delta with confidence — never "INT8 works fine." And watch for task-specific cliffs: quantization often hurts reasoning/math/code (long dependency chains amplify small errors) far more than it hurts chat or summarization. Always eval the hardest task the customer runs.

The trap to avoid: declaring victory on perplexity. The number of teams that shipped a "lossless" INT4 model that quietly lost 8 points of GSM8K is large. Your credibility as an advisor is built on catching that before the customer does.

12. The decision framework (what a principal recommends)

A compact algorithm you can run in a customer meeting:

1. What's the bottleneck?  (from Knowledge 00/01)
     decode/memory-bound  → weight-only INT4/INT8 (AWQ/GPTQ) + INT8 KV
     prefill/compute-bound → W8A8 / FP8 (use the faster tensor-core math)
     integer-only accelerator (AI100, Inferentia) → W8A8/W4A8 (SmoothQuant + calibration)
2. What's the accuracy floor?  (define it as a number on a real eval)
     loose (chat, summarize) → INT4 weight-only is usually fine
     tight (math, code, agents, safety) → FP8 or INT8, sensitivity-analyzed mixed precision
3. Have a training pipeline + budget?  → QAT if PTQ misses the floor; else PTQ.
4. What hardware?  FP8 needs Hopper+; INT8 is universal; INT4 needs kernel support (Marlin, etc.).
5. PROVE IT.  Eval on the customer's hardest task, report the delta honestly, side-by-side on real traffic.

The output of this framework is the JD's deliverable: "Here's the quantization scheme, here's the throughput/memory win, here's the measured accuracy cost on your workload, and here's why."

Frontier (2025–2026): FP4 block-scaled formats (MXFP4/NVFP4) and rotation-based outlier removal (QuaRot/SpinQuant) push this module below 8 bits — see Knowledge 09 §6.

13. References

• Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022) — the outlier discovery.
• Xiao et al., "SmoothQuant" (2022). Frantar et al., "GPTQ" (2022). Lin et al., "AWQ" (2023).
• NVIDIA, FP8 Formats for Deep Learning (Micikevicius et al., 2022) and TensorRT Model Optimizer docs.
• Nagel et al., "A White Paper on Neural Network Quantization" (Qualcomm AI Research, 2021) — the best single reference on PTQ/QAT theory; pairs directly with AIMET.
• Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (Google, 2018) — the foundational affine-quant + integer-matmul paper (§2).
• llm-compressor (vLLM), AutoGPTQ, bitsandbytes, llama.cpp k-quant docs.

➡ Next: Knowledge 03 — Model Conversion & Compilers — how to get the (now-quantized) model onto the accelerator at all.