Hitchhiker's Guide — Quantization & Model Compression

"Quantization is not rounding. It is finding the best integer world to map the floating-point world into."

1. Quantization Math — Precise Derivations

1.1 Affine (Asymmetric) Quantization

$$Q(x) = \text{clamp}\left(\text{round}\left(\frac{x}{s}\right) + z, ; q_{\min}, q_{\max}\right)$$

$$x_{\text{dequant}} = s \cdot (Q(x) - z)$$

Scale and zero-point derivation from observed range $[x_{\min}, x_{\max}]$:

$$s = \frac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}}, \qquad z = q_{\min} - \text{round}\left(\frac{x_{\min}}{s}\right)$$

For INT8: $q_{\min} = -128, q_{\max} = 127$ (signed) or $0, 255$ (unsigned).

Symmetric quantization ($z = 0$): $s = \frac{\max(|x|)}{q_{\max}}$. Simpler hardware: multiplication by scale instead of multiply + add.

1.2 Quantization Error

The quantization error (rounding error) for value $x$ is bounded by:

$$|Q(x) - x| \leq \frac{s}{2}$$

Therefore, smaller scale → smaller error. This is why per-channel quantization (separate scale per output channel) outperforms per-tensor (single scale): each channel can have its own optimal scale.

Signal-to-Quantization-Noise Ratio (SQNR):

$$\text{SQNR} \approx 6.02 b + 4.77 \text{ dB}$$ (for Gaussian-distributed values, $b$ = number of bits)

INT8 ≈ 52.9 dB SQNR; INT4 ≈ 28.9 dB SQNR. This is a 24 dB drop — a factor of ~250 in noise power.

1.3 Per-Group Quantization

In group quantization (used by GPTQ, AWQ, llama.cpp), weights are quantized in groups of $G$ elements sharing one (scale, zero-point):

$$Q(W_{i, j:j+G}) = \text{round}(W_{i, j:j+G} / s_{i,j})$$

Trade-off: $G=128$ → 1 extra FP16 scale per 128 INT4 weights = 3.1% overhead, but ~2-3 dB SQNR improvement over per-tensor.

Hardware consideration for Qualcomm HTP: HTP natively supports INT4 with per-group scales at G=32, G=64, G=128. G=32 is most accurate but requires more scale storage; G=128 is most hardware-friendly.

2. GPTQ — Hessian-Based Optimal Quantization

2.1 The Core Idea

GPTQ frames weight quantization as an optimization problem. For a layer with weight $W$ and input activations $X$:

$$\min_{\hat{W}} |WX - \hat{W}X|_F^2$$

subject to $\hat{W}$ being quantized.

The optimal update when quantizing weight $w_q$ to $\hat{w}_q$ is:

$$\delta W = -\frac{w_q - \hat{w}q}{[H^{-1}]{qq}} \cdot H^{-1}_{:,q}$$

where $H = 2X X^T$ is the (approximated) Hessian of the squared error.

Key insight: after quantizing one weight, update all remaining unquantized weights to compensate for the introduced error. This is the same idea as "least squares with constraints" — each quantization decision corrupts the loss function, and GPTQ corrects for it greedily.

2.2 Cholesky Implementation

Direct Hessian inversion is $O(d^3)$ — expensive for large layers. GPTQ uses the Cholesky decomposition trick: process columns in blocked order using triangular solve, achieving $O(d^2)$ amortized.

def gptq_quantize_layer(W, X, bits=4, group_size=128, block_size=128):
    """
    W: (out_features, in_features)
    X: (in_features, n_samples) — calibration activations
    Returns: W_quantized, scales, zeros
    """
    d = W.shape[1]
    H = 2 * (X @ X.T)  # (in_features, in_features)
    
    # Add damping for numerical stability
    H += 0.01 * torch.diag(H).mean() * torch.eye(d, device=H.device)
    
    # Cholesky decomposition for stable inversion
    L = torch.linalg.cholesky(H)
    H_inv = torch.cholesky_inverse(L)
    
    W_q = W.clone()
    scales = []
    zeros = []
    
    for i in range(0, d, block_size):
        block_end = min(i + block_size, d)
        for j in range(i, block_end):
            # Find quantization params for this column group
            g_start = (j // group_size) * group_size
            w_col = W_q[:, j]
            scale, zero = compute_qparams(w_col, bits)
            
            # Quantize this column
            w_q = quantize(w_col, scale, zero, bits)
            
            # Compute error
            err = (w_col - w_q) / H_inv[j, j]
            
            # Update remaining columns to compensate
            W_q[:, j+1:block_end] -= err.unsqueeze(1) * H_inv[j, j+1:block_end].unsqueeze(0)
            W_q[:, j] = w_q
    
    return W_q, scales, zeros

3. AWQ — Activation-Weighted Quantization

3.1 Salient Weight Detection

AWQ's key observation: not all weights are equally important. Weights corresponding to input channels with large activation magnitudes are disproportionately impactful on model accuracy.

$$\text{importance}(j) = \mathbb{E}{x \sim D}[|x_j|] \cdot |W{:,j}|_2$$

For the top-1% most important channels, quantization error multiplies by the (large) activation magnitude. For the bottom 99%, quantization error is scaled down.

3.2 The Scaling Trick

For input channel $j$ with scale $s_j$:

$$y = xW = (x \cdot s) \cdot (W / s) = x' \cdot W'$$

By choosing $s_j > 1$ for salient channels: $W'{:,j} = W{:,j} / s_j$ → smaller magnitude → less quantization error (fits better in the quantization grid). $x'_j = x_j \cdot s_j$ → but this is a multiplication of activations, which can be absorbed into the previous layer's weights (no extra compute).

Critical: the scale $s_j$ is chosen to minimize quantization error of $W'_{:,j}$ on the quantization grid, not based on a fixed formula. AWQ uses grid search: $s_j \in {1, 2, 4, 8, \ldots}$.

4. SmoothQuant — Migrating Difficulty from Activations to Weights

4.1 The Outlier Problem

LLMs have severe activation outliers: certain token positions and embedding dimensions have values 100× larger than typical. These outliers destroy per-tensor activation quantization (the scale becomes too large, compressing most values to a small range of integers).

Example: Activation channel 1053 in OPT-13B might have max value 3847 while 99% of channels are below 4. Per-tensor scale = 3847/127 = 30.3. Channel 0 with value 1.2 → quantizes to round(1.2/30.3) = 0. Massive quantization error.

4.2 The Per-Channel Scale Migration

For matrix multiply $y = xW$, introduce per-input-channel scales $s_j$:

$$y = x \cdot W = \underbrace{(x / s)}{\text{easy to quantize}} \cdot \underbrace{(W \cdot s)}{\text{absorb into weights}}$$

Scale is chosen as:

$$s_j = \frac{\max(|X_{:,j}|)^\alpha}{\max(|W_{j,:}|)^{1-\alpha}}, \quad \alpha = 0.5 \text{ (default)}$$

This migrates the quantization challenge from activations (hard to quantize, dynamic) to weights (easy to quantize, static, absorb offline).

Result: activation quantization improves dramatically; weight quantization barely worsens (weights are already well-distributed).

5. QAT and the Straight-Through Estimator

5.1 Why the Round Function Breaks Gradients

forward:  x → round(x/s) * s → quantized_x
backward: d_loss/d_x = d_loss/d_quantized_x × d_quantized_x/d_x
                                                = 0 (everywhere)

round() has derivative 0 almost everywhere (flat function with unit jumps). Gradient is zero → no learning signal. QAT is stuck.

5.2 STE: The Principled Hack

The Straight-Through Estimator approximates:

$$\frac{\partial \text{round}(x)}{\partial x} \approx \mathbf{1}{q{\min} \leq x \leq q_{\max}}$$

In practice, this means: just pass the gradient through as if quantization didn't happen (but zero it outside the quantization range).

class FakeQuantize(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, scale, zero_point, quant_min, quant_max):
        # Actual quantization (for forward accuracy)
        x_int = torch.round(x / scale) + zero_point
        x_int = torch.clamp(x_int, quant_min, quant_max)
        x_dequant = (x_int - zero_point) * scale
        # Save mask for backward (STE clips gradient outside range)
        ctx.save_for_backward(x, torch.tensor([quant_min * scale]),
                              torch.tensor([quant_max * scale]))
        return x_dequant
    
    @staticmethod
    def backward(ctx, grad_output):
        x, x_min, x_max = ctx.saved_tensors
        # STE: pass gradient through, clip outside quantization range
        mask = (x >= x_min) & (x <= x_max)
        grad_input = grad_output * mask.float()
        return grad_input, None, None, None, None

Why does it work despite being wrong? The approximation error in the gradient is small relative to the overall optimization landscape. The model learns to arrange its weights such that most values are in the quantization-friendly zone — which is exactly the goal.

6. Quantization Decision Tree (Interview Cheatsheet)

Given: model M, target device D, accuracy budget A, latency budget L

1. Can you afford QAT? (requires retraining time + labeled data)
   YES → QAT (best accuracy, especially for INT4)
   NO → go to 2

2. Are activation outliers severe? (check: max(|X|) / median(|X|) > 100)
   YES → Apply SmoothQuant first (free, offline)
   Then → go to 3

3. Target bitwidth?
   INT8 → PTQ with per-channel KL-divergence calibration (usually <0.5 PPL loss)
   INT4 → AWQ if latency matters, GPTQ if accuracy matters, both if you have time

4. Accuracy still insufficient after step 3?
   → Add Knowledge Distillation: quantized model learns from FP16 teacher
   → Try mixed-precision: INT8 for sensitive layers (first/last, embedding), INT4 for others
   → Increase group size (G=32 vs G=128)

5. Deploy to Qualcomm HTP?
   → Use QNN SDK quantization workflow on top of your quantized model
   → Enable HTP-specific optimizations (HTA = Hexagon Tensor Accelerator weight layout)

7. Key Accuracy Numbers to Know

8. Resources

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., 2023)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (Xiao et al., 2022)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022) — the mixed-precision INT8 approach used by bitsandbytes
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (Jacob et al., 2018) — the Google QAT paper
• PyTorch Quantization Documentation
• Qualcomm AI Hub — Quantization Guide