Deep Dive — PTQ, QAT & GPTQ from First Principles (Phase 03)

"PTQ gets you to INT8 for free. QAT gets you to INT4 with effort. GPTQ gets you to INT4 with math."

Section 1: Why Quantization Exists — The Memory Wall

A 7B parameter model in FP32 = 7 × 10⁹ × 4 bytes = 28 GB VRAM. A single A100 has 80 GB; an iPhone 15 Pro has 8 GB. Without quantization, large models simply don't fit on the hardware where they need to run.

Quantization converts floating-point weights (and optionally activations) to lower-bit integers:

The difference between "0.5%" and "15%" INT4 accuracy loss is entirely in the algorithm. That's why QAT and GPTQ exist.

Section 2: Post-Training Quantization (PTQ) — The Baseline

2.1 Symmetric Quantization Math

Symmetric per-tensor (the simplest form):

Given weights $W \in \mathbb{R}^{m \times n}$ and target bit width $b$:

$$\text{max_int} = 2^{b-1} - 1 \quad (\text{e.g., 127 for INT8, 7 for INT4})$$

$$\text{scale} = \frac{\max(|W|)}{\text{max_int}}$$

$$W_{\text{int}} = \text{round}\left(\frac{W}{\text{scale}}\right) \text{ clamped to } [-\text{max_int}, \text{max_int}]$$

$$W_{\text{deq}} = W_{\text{int}} \times \text{scale}$$

Example with INT4 (max_int = 7):

W = tensor([[0.23, -1.45, 0.07, 2.10],
            [-0.89,  0.51, 1.33, -0.42]])

abs_max = 2.10
scale   = 2.10 / 7 = 0.300

W_int   = round(W / 0.300) = [[1, -5,  0,  7],
                                [-3,  2,  4, -1]]  # clamped to [-7, 7]

W_deq   = W_int * 0.300    = [[0.30, -1.50, 0.00, 2.10],
                                [-0.90,  0.60, 1.20, -0.30]]

Error   = W - W_deq         = [[-0.07, 0.05, 0.07, 0.00],
                                 [0.01, -0.09, 0.13, -0.12]]

The maximum error is bounded by scale/2 = 0.15 (half the quantization step).

2.2 Asymmetric Quantization

Symmetric works when weights are centered around zero. Activations (after ReLU, GELU) are often non-negative — half the INT range would be wasted.

Asymmetric per-tensor:

$$\text{n_levels} = 2^b \quad (\text{e.g., 256 for INT8})$$

$$\text{scale} = \frac{\max(W) - \min(W)}{\text{n_levels} - 1}$$

$$\text{zero_point} = \text{round}\left(\frac{-\min(W)}{\text{scale}}\right) \text{ clamped to } [0, \text{n_levels}-1]$$

$$W_{\text{int}} = \text{round}\left(\frac{W}{\text{scale}} + \text{zero_point}\right) \text{ clamped to } [0, \text{n_levels}-1]$$

$$W_{\text{deq}} = (W_{\text{int}} - \text{zero_point}) \times \text{scale}$$

Example with INT8 and non-negative activations:

Activations after ReLU: min=0.0, max=3.2
scale = (3.2 - 0.0) / 255 = 0.01255
zero_point = round(-0.0 / 0.01255) = 0

For value 1.6:
  W_int = round(1.6 / 0.01255 + 0) = round(127.5) = 128
  W_deq = (128 - 0) * 0.01255 = 1.606
  Error = 0.006  — very small

With symmetric: min=0 would map to -127, wasting half the range.

2.3 Per-Tensor vs Per-Channel

Per-tensor: one scale for entire weight matrix. Simple but inaccurate for weights with very different magnitudes across output channels.

Per-channel: one scale per output channel (row of weight matrix). Much better for weights:

# Weight matrix W: [out_features, in_features]
# Per-channel: compute max abs per row
max_per_channel = W.abs().max(dim=1).values  # [out_features]
scale = max_per_channel / 127.0              # [out_features]

# Quantize row-by-row (broadcast)
W_int = (W / scale.unsqueeze(1)).round().clamp(-127, 127)
W_deq = W_int * scale.unsqueeze(1)

Why this matters: A transformer's linear layers often have output channels spanning 0.01–10.0 in weight magnitude. Per-tensor scale = 10.0/127 = 0.079, so small-magnitude channels are quantized with massive relative error. Per-channel fixes this.

Section 3: Quantization-Aware Training (QAT)

3.1 The Core Problem with PTQ

PTQ quantizes after training is complete. The model was never trained to be robust to quantization noise. This is fine for INT8 but catastrophic for INT4:

INT8 quantization noise ~ 0.5% of weight range → small perturbation
INT4 quantization noise ~ 7%  of weight range → very large perturbation

At INT4, each weight is rounded to one of only 16 values. A model trained in FP32 never experienced this level of discretization and its outputs change dramatically.

Solution: Train the model while simulating quantization. The model learns to be robust.

3.2 The Straight-Through Estimator (STE)

The quantization function $Q(x) = \text{round}(x/s) \times s$ has zero gradient almost everywhere and undefined gradient at step discontinuities. We can't backprop through it.

STE trick (Hinton, Bengio et al.): approximate $\partial Q(x)/\partial x \approx 1$ (pass gradient straight through):

$$\text{FakeQuantize}(x) = x + \underbrace{(Q(x) - x)}{\text{quantization error}} \cdot \underbrace{\mathbf{1}}{\text{stop-gradient}}$$

In PyTorch:

def fake_quantize(x, scale, zero_point, n_levels):
    """Forward: quantize + dequantize. Backward: identity gradient."""
    # Quantize
    x_int = (x / scale + zero_point).round().clamp(0, n_levels - 1)
    # Dequantize
    x_dq  = (x_int - zero_point) * scale
    
    # STE: x_out = x + (x_dq - x).detach()
    # Forward:  x_out = x_dq  (quantized values)
    # Backward: ∂x_out/∂x = ∂x/∂x = 1  (gradient flows through unchanged)
    return x + (x_dq - x).detach()

Why this works: During forward pass, the model sees quantized values — it experiences the quantization error and must produce correct output despite it. During backward pass, the gradient flows as if quantization didn't happen — weights still get useful gradient signal to improve.

Concrete gradient walkthrough:

import torch

x = torch.tensor([1.7], requires_grad=True)
scale = 0.5
zero_point = 0
n_levels = 256

# Forward
x_int = (x / scale + zero_point).round().clamp(0, 255)  # = round(3.4) = 3
x_dq  = (x_int - zero_point) * scale                    # = 3 * 0.5 = 1.5
x_out = x + (x_dq - x).detach()                         # = x + (1.5 - 1.7).detach()
                                                          # = x + (-0.2)  [no grad to -0.2]
# x_out = 1.5 (numerically)

loss = x_out ** 2                                        # = 2.25
loss.backward()

print(x.grad)  # = 2 * x_out * dx_out/dx
               # = 2 * 1.5 * 1   ← STE: dx_out/dx = 1
               # = 3.0
               # NOT: 2 * 1.5 * 0 = 0 (what you'd get from round)

Without STE: x.grad = 0.0 → no learning signal. With STE: x.grad = 3.0 → normal gradient.

3.3 QAT Training Loop

def train_qat(model, train_data, n_epochs=3, lr=1e-3, bits=8):
    """
    Full QAT training loop.
    
    The key: wrap model BEFORE creating optimizer.
    Then train normally — FakeQuantize handles the rest.
    """
    # Step 1: Replace all Linear layers with QATLinear (FakeQuantize on weights)
    qat_model = wrap_model_for_qat(model, bits=bits)
    
    # Step 2: Create optimizer — ALL parameters trainable
    # (the base weight learns to minimize error despite quantization)
    optimizer = torch.optim.AdamW(qat_model.parameters(), lr=lr)
    
    losses = []
    for epoch in range(n_epochs):
        epoch_loss = 0.0
        for x, y_true in train_data:
            optimizer.zero_grad()
            
            # Forward: weights go through FakeQuantize
            #   → model sees INT8-like values
            #   → must produce accurate output
            y_pred = qat_model(x)
            loss   = F.mse_loss(y_pred, y_true)
            
            # Backward: STE allows gradients to flow
            #   → weights update to reduce quantization sensitivity
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
        
        losses.append(epoch_loss / len(train_data))
    
    return qat_model, losses

What the model learns during QAT:

1. Weight smoothing: weights tend toward values that quantize well (multiples of the scale)
2. Reduced sensitivity: activations become less sensitive to weight perturbations (the model learns to be more robust)
3. Calibrated ranges: the FakeQuantize scale adapts to actual weight distribution

3.4 Converting QAT Model to INT8

After QAT training, the model's weights are FP32 but trained to be robust to INT8 quantization. To actually save memory, we need to extract the integer weights:

def convert_to_int8(qat_model):
    """
    After QAT training:
    1. For each QATLinear layer:
       a. Read current weight (FP32, but quantization-friendly)
       b. Compute scale/zero_point from actual min/max
       c. Store W_int (INT8 tensor)
    2. Replace QATLinear → nn.Linear with dequantized weights
       (in real deployment: use true INT8 GEMM kernel)
    """
    converted = copy.deepcopy(qat_model)
    quant_info = {}
    
    for name, module in list(converted.named_modules()):
        if isinstance(module, QATLinear):
            W = module.weight.data
            fq = module.fake_q
            
            # Compute quantization parameters
            min_val, max_val = W.min(), W.max()
            scale = (max_val - min_val) / (fq.n_levels - 1)
            zp    = (-min_val / scale).round().clamp(0, fq.n_levels-1).int()
            
            # Quantize to integers
            W_int = (W / scale + zp).round().clamp(0, fq.n_levels-1).to(torch.int8)
            
            # Dequantize (for our CPU reference implementation)
            W_deq = (W_int.float() - zp) * scale
            
            # Save quantization metadata
            quant_info[name] = QuantizedWeight(W_int, scale.item(), zp.item(), fq.bits)
            
            # Replace module with standard Linear using dequantized weights
            # (in production: use torch.ao.nn.quantized.Linear with INT8 GEMM)
            new_linear = nn.Linear(W.shape[1], W.shape[0], bias=module.bias is not None)
            new_linear.weight = nn.Parameter(W_deq)
            # ... set on parent module
    
    return converted, quant_info

Memory layout in true INT8 deployment:

FP32 Linear weight: [4096, 4096] × 4 bytes = 64 MB
INT8 Linear weight: [4096, 4096] × 1 byte  = 16 MB  ← 4× smaller
Scale: [4096] × 4 bytes = 16 KB (per-channel scale, negligible)

During inference:
  x: FP16  → INT8 quantize
  W: INT8  (stored)
  GEMM:    INT8 × INT8 → INT32 accumulator  (fast on Qualcomm HTP/CUDA INT8 cores)
  Output:  INT32 → dequantize → FP16

3.5 QAT vs PTQ: When to Use Each

Section 4: GPTQ — Second-Order Quantization

4.1 The Core Insight

PTQ quantizes each weight independently: $W_q[i,j] = \text{round}(W[i,j] / s) \times s$.

This ignores correlations. But the reconstruction objective is:

$$\min_{W_q} |W \cdot X - W_q \cdot X|_F^2$$

We want the output of the layer to be close, not the weights themselves. Some weight perturbations cancel out, others amplify. GPTQ uses the Hessian of this objective to make smarter quantization decisions.

4.2 The Hessian of Layer Reconstruction

For the layer output $WX$ (W is the weight matrix, X is the input data), the reconstruction loss is:

$$L(W_q) = |WX - W_q X|_F^2$$

The gradient: $\nabla_{W_q} L = -2(W - W_q)X^T$
The Hessian: $H = 2XX^T$

This Hessian captures how much the output changes when we perturb each weight. Large eigenvalues = that direction of weight space strongly affects output. Small eigenvalues = we can change those weights without hurting accuracy.

def compute_hessian(X):
    """
    X: [N, d_in]  — N examples of layer inputs
    Returns H: [d_in, d_in]
    """
    # H = 2 * X^T @ X  (in the form X is [d_in, N] convention)
    Xt = X.T.float()       # [d_in, N]
    H  = 2.0 * Xt @ Xt.T  # [d_in, d_in]
    
    # Numerical stabilization: add damping
    # dampening = λ_mean * 0.01 prevents near-singular H
    damp = 0.01 * H.diag().mean()
    H += damp * torch.eye(H.shape[0])
    
    return H

# Example:
# Network input x with batch=100 samples, d_in=32
X = torch.randn(100, 32)
H = compute_hessian(X)
# H is 32×32 positive definite symmetric matrix
# H[i,i] ≈ 2 * sum_k x_k[i]^2 = variance of input dimension i

Intuition: $H_{ii}$ tells us how sensitive the reconstruction loss is to weight column $i$. If the $i$-th input feature is always near-zero (small $H_{ii}$), quantizing that column's weights doesn't matter much. If $H_{ii}$ is large (that feature is active and important), quantization error there hurts a lot.

4.3 Optimal Brain Quantizer (OBQ) — The Algorithm

Problem: Given weight matrix $W \in \mathbb{R}^{d_{out} \times d_{in}}$ and Hessian $H$, find quantized $W_q$ minimizing $|WX - W_q X|_F^2$.

OBQ idea (one column at a time):

For column $j$ (processing in order $j = 0, 1, \ldots, d_{in}-1$):

1. Quantize column $j$: $w_q^{(j)} = \text{round}(W[:, j] / s) \times s$
2. Compute quantization error: $\delta^{(j)} = W[:, j] - w_q^{(j)}$
3. Update remaining unquantized columns to compensate: $$W[:, j'] \leftarrow W[:, j'] - \delta^{(j)} \cdot \frac{H_{j, j'}^{-1}}{H_{jj}^{-1}}$$

The update step propagates the error to remaining columns in a way that minimizes the total reconstruction loss. This is the "brain surgery" analogy from OBS (Optimal Brain Surgeon): removing (quantizing) one weight, then adjusting others to compensate.

4.4 GPTQ: OBQ Made Practical

OBQ processes columns in arbitrary order (choosing the column with minimum quantization error each time) — $O(d_{in}^2)$ work per column, $O(d_{in}^3)$ total. Too slow for LLMs.

GPTQ simplification:

1. Process columns left-to-right (no selection overhead)
2. Use Cholesky decomposition of $H^{-1}$ for numerically stable updates

def gptq_quantize_layer(W, H, bits=4):
    """
    W: [d_out, d_in]  weight matrix
    H: [d_in, d_in]   Hessian from calibration data
    """
    W = W.clone().float()
    d_out, d_in = W.shape
    
    # Step 1: Compute H^{-1} via Cholesky decomposition
    # torch.linalg.cholesky gives L such that H = L @ L^T (lower triangular)
    try:
        L = torch.linalg.cholesky(H)
        I = torch.eye(d_in, device=H.device)
        H_inv = torch.cholesky_solve(I, L)  # H^{-1} = L^{-T} L^{-1}
    except:
        H_inv = torch.linalg.inv(H)  # fallback
    
    W_q = W.clone()
    
    # Step 2: Process columns left-to-right
    for j in range(d_in):
        # Quantize column j
        w_col   = W[:, j]                    # [d_out]
        w_q_col = quantize_symmetric(w_col, bits)
        W_q[:, j] = w_q_col
        
        error   = w_col - w_q_col            # [d_out]  quantization error
        h_inv_j = H_inv[j, j+1:]             # [d_in - j - 1]  off-diagonal
        h_inv_jj = H_inv[j, j].clamp(min=1e-8)  # diagonal element
        
        # Propagate error to remaining columns:
        # W[:, j+1:] -= error.outer(h_inv_j) / h_inv_jj
        if j + 1 < d_in:
            W[:, j+1:] -= (error.unsqueeze(1) * h_inv_j.unsqueeze(0)) / h_inv_jj
    
    return W_q

Step-by-step walkthrough with 1D output (d_out=1) and 4 input features:

W     = [0.5, 1.2, -0.3, 0.8]        # original weights
H_inv = [[0.4, -0.1,  0.0,  0.2],    # inverse Hessian
          [-0.1, 0.3,  0.1, -0.1],
          [0.0,  0.1,  0.5,  0.1],
          [0.2, -0.1,  0.1,  0.6]]

bits=4, scale = max(|W|)/7 = 1.2/7 ≈ 0.171

Step j=0:
  w[0] = 0.5
  w_q[0] = round(0.5/0.171)*0.171 = round(2.92)*0.171 = 3*0.171 = 0.513
  error = 0.5 - 0.513 = -0.013
  H_inv[0, 1:] = [-0.1, 0.0, 0.2],  H_inv[0,0] = 0.4
  Update: W[1:] -= (-0.013) * [-0.1, 0.0, 0.2] / 0.4
        W[1]  += (-0.013) * (-0.1) / 0.4 = +0.00325  → 1.203
        W[2]  += (-0.013) * (0.0)  / 0.4 = 0         → -0.300
        W[3]  += (-0.013) * (0.2)  / 0.4 = -0.0065   → 0.7935

Step j=1:
  w[1] = 1.203 (already adjusted!)
  w_q[1] = round(1.203/0.171)*0.171 = 7*0.171 = 1.197  (now 1.197 ≈ 1.203)
  error = 1.203 - 1.197 = 0.006  ← much smaller error after adjustment!
  ... (continue)

The key: after compensating, the next column's quantization error is smaller — errors are propagated forward and partially cancelled.

4.5 Why Cholesky?

Direct matrix inversion is numerically unstable and $O(n^3)$ every time. Cholesky gives:

1. Numerical stability: Cholesky only works for positive-definite matrices (which $H$ is, after damping), and the triangular solve is well-conditioned
2. Efficiency: compute $H = LL^T$ once ($O(n^3)$), then triangular solves are $O(n^2)$
3. Block processing: GPTQ processes weights in blocks of 128 columns, recomputing a local Cholesky for each block — this reduces numerical drift

# Why this works:
# H @ H_inv = I
# cholesky(H) = L  (lower triangular, L @ L.T = H)
# cholesky_solve(I, L) solves:  H @ X = I  →  X = H^{-1}
# 
# This is more stable than: H_inv = torch.linalg.inv(H)
# because inv() suffers from floating-point catastrophic cancellation
# for near-singular H

L = torch.linalg.cholesky(H)
H_inv = torch.cholesky_solve(torch.eye(n), L)

4.6 GPTQ vs AWQ vs Naive Quantization: Benchmark

For Llama-2-7B at INT4, on WikiText-2 perplexity:

GPTQ's advantage: purely post-training, needs only ~128 calibration samples, takes ~4 hours for 7B model on a single A100. No gradient computation needed.

4.7 Full Pipeline: Calibration + GPTQ + Deployment

import torch, copy
from transformers import AutoModelForCausalLM, AutoTokenizer

# 1. Load model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# 2. Collect calibration data
def get_calibration_data(tokenizer, n_samples=128, seq_len=512):
    """Use a diverse sample of real text for calibration."""
    # In practice: C4, WikiText, or domain-specific data
    texts = ["The quick brown fox..."] * n_samples
    inputs = tokenizer(texts, return_tensors="pt", max_length=seq_len,
                      truncation=True, padding=True)
    return inputs["input_ids"]

cal_data = get_calibration_data(tokenizer)

# 3. Run GPTQ layer-by-layer
def quantize_model_gptq(model, cal_data, bits=4):
    """Collect inputs via hooks, then quantize each layer."""
    quantized_model = copy.deepcopy(model)
    
    for name, module in quantized_model.named_modules():
        if isinstance(module, nn.Linear):
            # Register hook to capture inputs during calibration pass
            inputs_collected = []
            hook = module.register_forward_hook(
                lambda m, inp, out: inputs_collected.append(inp[0].detach()))
            
            # Run calibration
            with torch.no_grad():
                model(cal_data)
            hook.remove()
            
            # Stack calibration inputs and compute Hessian
            X_cal = torch.cat([x.reshape(-1, x.shape[-1]) for x in inputs_collected])
            H = compute_hessian(X_cal)
            
            # Quantize this layer's weights
            W_q = gptq_quantize_layer(module.weight.data, H, bits=bits)
            module.weight.data = W_q
    
    return quantized_model

quantized_model = quantize_model_gptq(model, cal_data, bits=4)

# 4. Evaluate
def evaluate_perplexity(model, tokenizer, text):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        loss = model(**inputs, labels=inputs["input_ids"]).loss
    return torch.exp(loss).item()

4.8 Interview Pitfalls

Q: "Explain STE without jargon to a PM."

"We train the model as if the weights were quantized to 8-bit integers — every forward pass uses the rounded values so the model learns to deal with it. But for the backward pass (updating weights), we pretend the quantization step wasn't there and use the normal gradient as if the weights were still floating-point. This trick — called Straight-Through Estimator — lets us train the model to be robust to quantization noise even though we can't actually differentiate through the rounding operation."

Q: "Why does GPTQ need calibration data? Can't you just quantize without it?"

"GPTQ uses the Hessian $H = 2XX^T$ to determine which weights are sensitive to quantization (large $H_{ii}$ = that input dimension is highly active, so quantizing that column's weights hurts more). Without calibration data, we don't know which directions in weight space are important. With 128 random samples, we get a good approximation of the typical input distribution. Without it, we'd have to treat all weights equally — that's naive round-to-nearest, which loses 5–15% accuracy at INT4."

Q: "QAT vs GPTQ — when would you choose each at Qualcomm?"

"For a new on-device model being trained from scratch or fine-tuned: QAT, because you have training data and compute budget, and it gives the best accuracy. For quantizing a large pretrained model (Llama-3, Gemma-2) for NPU deployment without access to the training data or compute: GPTQ (or AWQ), because you only need 128 calibration samples and a few GPU-hours. GPTQ is the practical choice for the productionization of public models."

Q: "What is the relationship between PTQ, QAT, and GPTQ?"

"PTQ: quantize post-hoc with no training. Cheap, works for INT8, fails at INT4.
QAT: simulate quantization during training with STE. Best accuracy, expensive.
GPTQ: second-order PTQ — use Hessian to propagate quantization errors across columns, compensating for each quantized column before moving to the next. Near-QAT accuracy, no training needed."

Section 5: Quick Setup

pip install torch>=2.1.0
# For running GPTQ on real LLMs:
pip install auto-gptq transformers

QAT quick test:

import torch, torch.nn as nn
from phase_03_fundamentals.lab_02_qat.solution import FakeQuantize, wrap_model_for_qat

model = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 16))

# Check STE works
fq = FakeQuantize(bits=8)
x = torch.randn(4, 64, requires_grad=True)
y = fq(x)
y.sum().backward()
print(f"STE gradient norm: {x.grad.norm():.4f}")  # Should be non-zero!

# QAT training
qat_model = wrap_model_for_qat(model, bits=8)
print("QAT layers:", [name for name, m in qat_model.named_modules() if 'QAT' in type(m).__name__])

GPTQ quick test:

from phase_03_fundamentals.lab_03_gptq.solution import compute_hessian, gptq_quantize_layer, compare_reconstruction_error

d_out, d_in, N = 32, 64, 128
X = torch.randn(N, d_in)        # calibration inputs
W = torch.randn(d_out, d_in)    # pretrained weight

H = compute_hessian(X)
result = compare_reconstruction_error(W, X, bits=4)
print(f"GPTQ error:  {result['gptq_error']:.4f}")
print(f"Naive error: {result['naive_error']:.4f}")
print(f"Improvement: {result['improvement_ratio']:.1f}x")  # GPTQ should be better