Phase 03 — Quantization & Model Compression
Difficulty: ⭐⭐⭐⭐⭐
Estimated Time: 4 weeks (90–110 hours)
Roles Supported: Core differentiating skill — every NPU deployment requires quantization; accuracy recovery under aggressive quantization is what Senior Staff engineers own
Why This Phase Exists
This is the skill that distinguishes Qualcomm ML engineers from generic ML engineers. A Qualcomm NPU (HTP) achieves peak performance with INT8/INT4 activations and weights. Moving from FP32 → INT8 gives ~4x memory reduction and ~4–8x compute speedup, but at what accuracy cost? The answer is: it depends entirely on how you quantize.
Naive quantization (round FP32 to INT8 with global min/max clipping) easily degrades MMLU from 72% to 55% for a 7B model. State-of-the-art quantization (GPTQ + activation-aware calibration) can maintain 70.8% — barely above FP16 baseline. The difference between these outcomes is what this phase teaches.
The JD requires "expertise in model training and inference workflows" with "graph optimizations for performance and accuracy." Quantization is the single most important accuracy-vs-performance tradeoff in the NPU deployment pipeline.
Concepts
- Uniform quantization: $Q(x) = \text{round}(x / s) + z$, where scale $s = (x_{max} - x_{min}) / (2^b - 1)$ and zero-point $z$ is the integer offset
- Per-tensor vs per-channel vs per-group quantization: coarser granularity → higher error; finer granularity → more overhead; group quantization (AWQ, GPTQ) balances both
- Symmetric vs asymmetric: symmetric ($z=0$, signed ints) simpler for hardware; asymmetric handles non-zero activation distributions better (important for ReLU outputs)
- Post-Training Quantization (PTQ): quantize after training using a calibration dataset to determine scales; fast but suboptimal
- Calibration strategies: MinMax, Percentile (discard outliers), Entropy/KL-divergence (minimize information loss), MSE minimization
- Quantization-Aware Training (QAT): insert fake-quantization nodes during training; gradients flow through straight-through estimator (STE); best accuracy
- Straight-Through Estimator (STE): $\frac{\partial \text{round}(x)}{\partial x} \approx 1$ in backward pass — enables gradients to flow through the non-differentiable round operation
- GPTQ (Generative Pre-trained Transformer Quantization): layer-wise quantization using second-order (Hessian) information; solves an optimal quantization problem per-layer; 4-bit models at near-FP16 accuracy
- AWQ (Activation-Weighted Quantization): identifies salient weights (high activation magnitude) and applies per-group scaling to protect them; simpler than GPTQ, hardware-friendly
- SmoothQuant: migrates quantization difficulty from activations to weights by multiplying activations by $1/s$ and weights by $s$; activations become easier to quantize
- FP8 quantization: used in H100/H200 and Qualcomm's latest chips; E4M3 or E5M2 format; lower quantization error than INT8 for many distributions
- Knowledge Distillation: train a student to match teacher logits/features; use for accuracy recovery after aggressive quantization
- Magnitude pruning: zero out weights below threshold; reduces model size but needs sparse hardware support for speedup
- Structured pruning: remove entire channels/heads/layers; immediately reduces compute; can be combined with quantization
Labs
Lab 01 — PTQ Pipeline with Accuracy Benchmarking
| Field | Value |
|---|---|
| Goal | Build a complete Post-Training Quantization pipeline for a small transformer (TinyLlama or OPT-125M) that: quantizes all linear layers to INT8/INT4, supports multiple calibration strategies, and reports perplexity/task accuracy before and after quantization |
| Concepts | Min/max calibration, percentile calibration, per-tensor vs per-channel, quantize/dequantize (QDQ) nodes, accuracy regression |
| Steps | 1. Implement QuantizedLinear module with per-channel INT8 quantization of weights; 2. Implement calibration hooks (capture activation statistics over 512 sample forward passes); 3. Implement MinMax, Percentile (99.9th), and KL-divergence calibration for activations; 4. Replace all nn.Linear in target model with QuantizedLinear; 5. Run calibration on WikiText-2 train set; 6. Evaluate perplexity on WikiText-2 test + MMLU 5-shot accuracy for each calibration strategy |
| Stack | PyTorch 2.3+, transformers, datasets, lm_eval |
| Datasets | WikiText-2, MMLU (via lm_eval) |
| Output | Accuracy comparison table: FP16 vs INT8 per-tensor vs INT8 per-channel vs INT4 per-channel across 3 calibration strategies; calibrated .pt model |
| How to Test | pytest test_lab.py — checks: quantized values fit in [-128,127] (INT8) or [-8,7] (INT4), QDQ round-trip error < 1% of weight range, perplexity degradation < 0.5 ppl for INT8 per-channel |
| Talking Points | Why is per-channel better than per-tensor? What is the KL-divergence calibration optimizing? When would you use percentile over min-max? |
| Resume Bullet | Built PTQ pipeline for OPT-125M with 4 calibration strategies; achieved 0.3 PPL degradation under INT8 per-channel vs FP16 baseline, enabling 3.2× memory reduction for NPU deployment |
| Extensions | Add rotation-based outlier suppression (QuaRot); implement per-group quantization (group_size=128); add accuracy CI comparison with statistical significance |
Lab 02 — QAT from Scratch
| Field | Value |
|---|---|
| Goal | Implement Quantization-Aware Training from scratch using fake-quantization nodes with Straight-Through Estimator; train a ResNet-18 on CIFAR-10 from QAT initialization (vs PTQ baseline) and demonstrate superior accuracy |
| Concepts | Fake quantization, STE, torch.quantization.FakeQuantize, ObserverBase, HistogramObserver, QAT fine-tuning schedule, fold-BN-into-conv before deployment |
| Steps | 1. Implement FakeQuant function with STE backward using torch.autograd.Function; 2. Implement MinMaxObserver that tracks running min/max and updates quantization parameters; 3. Implement QuantizationConfig with per-layer precision specification; 4. Insert fake-quant nodes after every Conv2d and before every activation; 5. Fine-tune ResNet-18 with QAT for 30 epochs on CIFAR-10; 6. Compare: FP32 (93.1%) → PTQ INT8 → QAT INT8; 7. Export to torch.jit.script with quantized ops folded |
| Stack | PyTorch 2.3+, torchvision |
| Datasets | CIFAR-10 |
| Output | Trained QAT model; accuracy table FP32 vs PTQ vs QAT (INT8 weights + INT8 activations); size comparison |
| How to Test | pytest test_lab.py — checks: STE gradient passes through round operation, observer statistics are monotonically updated, QAT model accuracy ≥ PTQ+1% on CIFAR-10 test set |
| Talking Points | Why does STE work despite being mathematically incorrect? When does QAT fail to help (hint: very low bits like INT2)? How do you schedule learning rates differently during QAT? |
| Resume Bullet | Implemented QAT from scratch with STE-based fake quantization; improved INT8 ResNet-18 accuracy by 2.4% over PTQ baseline (91.8% vs 89.4%) on CIFAR-10 |
| Extensions | Implement mixed-precision QAT (different layers at INT8/INT4); add learned step size quantization (LSQ); implement channel-wise quantization in QAT |
Lab 03 — GPTQ: Hessian-Based Optimal Quantization
The shipped lab (lab-03-gptq/) implements GPTQ end-to-end: Hessian computation, column-wise OBQ with Cholesky updates, error propagation, and a full model pipeline with calibration hooks. AWQ is implemented in Phase 10 — Lab 01. The AWQ/SmoothQuant/Pareto sweep described below is the extension target once both labs are done.
| Field | Value |
|---|---|
| Goal | Implement simplified versions of GPTQ, AWQ, and SmoothQuant for LLaMA-style models; run all three on LLaMA-3.2-1B at INT4 and INT8; plot accuracy-latency Pareto frontier |
| Concepts | Hessian-based optimal quantization (GPTQ), activation magnitude analysis (AWQ), scale migration (SmoothQuant), group quantization, accuracy vs throughput Pareto curve |
| Steps | 1. Implement SmoothQuant: collect activation max per-channel over calibration set; compute per-channel scale $s = \max(|X|) / \max(|W|)$; apply to W and X; 2. Implement AWQ: find top-1% salient weight channels (max activation × max weight); scale those channels; 3. Implement GPTQ: layer-by-layer; compute pseudo-Hessian from activations; update remaining unquantized weights to compensate for quantization error; 4. Evaluate all methods on WikiText-2 PPL + MMLU at INT4 and INT8; 5. Plot 2D Pareto curves (accuracy vs latency, accuracy vs bits, latency vs model size) |
| Stack | PyTorch 2.3+, transformers, lm_eval, matplotlib |
| Datasets | WikiText-2, MMLU, C4 (for calibration) |
| Output | 3-method comparison table at INT4/INT8; Pareto curve visualization; quantized model weights; end-to-end demo: python demo.py --model llama-3.2-1b --method gptq --bits 4 |
| How to Test | pytest test_lab.py — checks: SmoothQuant scales are per-channel not per-tensor, AWQ identifies correct salient channels (high act × high weight), GPTQ error decreases per layer, PPL ranking: FP16 < AWQ < GPTQ < SmoothQuant < PTQ at INT4 |
| Talking Points | Why does GPTQ use Hessian information? What does "activation-aware" mean in AWQ? Why does SmoothQuant help hardware but not always help accuracy? When would you choose AWQ over GPTQ? |
| Resume Bullet | Implemented GPTQ/AWQ/SmoothQuant for LLaMA-3.2-1B; achieved 6.1 PPL at INT4 (vs 5.8 FP16 baseline) with GPTQ — 4× memory reduction for NPU deployment with <5% accuracy loss |
| Extensions | Add SpQR (sparse + quantized) combination; implement GGML-style quantization (llama.cpp format); add FP8 E4M3 quantization path; run on Qualcomm AI Hub |
Deliverables Checklist
- PTQ pipeline with calibration comparison table on OPT-125M / TinyLlama
- QAT implementation with STE showing accuracy > PTQ on CIFAR-10
- GPTQ + AWQ + SmoothQuant comparison with Pareto curves on LLaMA-3.2-1B
-
All
test_lab.pysuites pass - Pareto curve plots saved as PNG artifacts
Guides in This Phase
- HITCHHIKERS-GUIDE.md — quantization math, GPTQ, AWQ, SmoothQuant, decision tree, key numbers
- DEEP-DIVE-QAT-GPTQ.md — PTQ, QAT/STE, and GPTQ from first principles (supports Labs 02–03)
Interview Relevance
- "Explain the Straight-Through Estimator and why it works." — after Lab 02, you can draw the backward graph and explain the approximation
- "How does GPTQ minimize quantization error?" — describe the Cholesky-based optimal weight update, the per-column sequential quantization, and the Hessian estimation
- "Given a model that loses 5% accuracy under INT4, what would you try first?" — you have a decision tree: SmoothQuant (free, helps activation outliers), AWQ (salient channels), GPTQ (optimal per-layer), QAT (expensive but best)
- "What is the memory layout consideration for per-group quantization on an NPU?" — group scales must be loaded alongside weights, understand the bandwidth implications