Lab 03 — GPTQ: Optimal Brain Quantization

Phase: 03 — Quantization & Model Compression | Difficulty: ⭐⭐⭐⭐⭐ | Time: 4–5 hours

GPTQ (Frantar et al., 2022) quantizes LLMs to INT4 with minimal accuracy loss using second-order weight optimization.

What you build

  • compute_hessian — H = 2·X·X^T + damping·I from calibration activations
  • symmetric_quantize — INT4/INT8 symmetric per-tensor quantization
  • gptq_quantize_layer — column-wise OBQ with Cholesky-based Hessian inverse update
  • compare_reconstruction_error — GPTQ vs naive round-to-nearest
  • GPTQLayer — inference layer with W_q as buffer (no grad)
  • gptq_quantize_model — full pipeline with forward hooks for calibration

Key concepts

ConceptWhat to understand
OBQOptimal Brain Quantizer: minimize per-column reconstruction error
HessianH = 2XX^T captures sensitivity of loss to weight perturbations
Cholesky updateNumerically stable Hessian inverse for sequential column updates
Error propagationQuantizing column j updates remaining columns to compensate
GPTQ vs AWQGPTQ: Hessian-based; AWQ: activation scaling — both INT4 SOTA

Files

FilePurpose
lab.pyStubs with TODO markers
solution.pyComplete working implementation
test_lab.pypytest suite
requirements.txtDependencies

Run

pip install -r requirements.txt
python lab.py
pytest test_lab.py -v