Quantization Guide

Phase 6 · Document 06 · Local Inference Prev: 05 — MLX and Apple Silicon · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Quantization is the single technique that makes local inference practical. A 70B model is ~140 GB in FP16 — impossible on consumer hardware; at 4-bit it's ~40 GB and runs on a 48 GB Mac or two consumer GPUs. Quantization is also what cloud providers sometimes apply silently to cut their costs, which is why "the same model ID" can be worse on a cheap endpoint (Phase 5.10). Every GGUF variant (03), every MLX 4-bit model (05), and every "will it fit?" calculation (02) is downstream of understanding quantization. This doc explains, from zero, what it is, why it works, how it's done, and how to choose a level without wrecking quality.

2. Core Concept

Plain-English primer: trading precision for size

A model's weights are billions of numbers. By default each is stored in 16 bits (FP16/BF16 — 2 bytes). Quantization stores each number in fewer bits — 8, 4, even 2 — so the model takes less memory and is faster to read (and decode is bandwidth-bound, so smaller = faster, 01).

The catch: fewer bits means less precision — you can represent fewer distinct values, so each weight is rounded to the nearest representable value. That rounding error can degrade quality. The art of quantization is shrinking the model while keeping the rounding error small enough that quality barely moves.

Why it works at all: trained networks are redundant and noise-tolerant — no single weight is precisely critical, and the network already learned to be robust to small perturbations. Rounding weights a little is a small perturbation. So at 8-bit and good 4-bit schemes, quality loss is often negligible; only at 3-bit and below does it usually become visible.

Precision formats (the number types)

FP16 vs BF16: same size, but BF16 keeps FP32's exponent range (fewer overflow issues) at the cost of mantissa precision — that's why it's the modern default (Phase 1.02).

How quantization is actually done

The core operation: take a weight w, store it as an integer q plus a scale s (and sometimes a zero-point), so w ≈ q × s. Computed per block/group of weights (each block gets its own scale — keeping local error small).

# Symmetric per-group quantization to b bits (the essence)
import numpy as np
def quantize_group(w, bits=4):
    qmax = 2**(bits-1) - 1                 # e.g. 7 for 4-bit signed
    s = np.max(np.abs(w)) / qmax           # one scale per group
    q = np.round(w / s).clip(-qmax-1, qmax)
    return q.astype(np.int8), s            # store small ints + one float scale
def dequantize(q, s):
    return q * s                           # reconstruct at compute time

Methods differ in how cleverly they pick scales and which weights get more bits:

• RTN (round-to-nearest): the naive version above. Fast, no data needed; lowest quality at 4-bit.
• GPTQ: uses a small calibration dataset and second-order (Hessian) info to choose quantization that minimizes output error, not just weight error. Strong 4-bit quality; common for GPU/vLLM (safetensors).
• AWQ (Activation-aware): observes which weights see the largest activations and protects those (keeps them higher precision). Excellent 4-bit quality; popular for GPU serving.
• k-quants (llama.cpp / GGUF): block-wise mixed-precision (Q4_K_M, etc.) that gives more bits to important tensors (attention, some FFN). The S/M/L suffix = how aggressive. The Q*_K family is the GGUF default (03).
• imatrix (importance matrix): llama.cpp calibration that weights blocks by importance using sample data — improves low-bit GGUF quality (the GGUF analog of GPTQ/AWQ's calibration idea).
• QAT (Quantization-Aware Training): quantization is simulated during training/fine-tuning so the model learns to compensate. Best low-bit quality, but requires the model maker to do it (e.g., Gemma QAT checkpoints) — it's not something you apply post-hoc.

PTQ vs QAT: everything except QAT is Post-Training Quantization (applied to a finished model — cheap, no training). QAT bakes robustness in during training — more expensive, best quality, only available when the lab ships a QAT checkpoint.

Two other dimensions you'll meet

• Weight-only vs weight+activation: most local quant is weight-only (weights small, math still in FP16). FP8/INT8 activation quant (e.g., on H100) also quantizes the activations for more speed.
• KV-cache quantization: quantize the KV cache (not the weights) to int8/int4 to fit more context/concurrency (02) — orthogonal to weight quant.
• Dynamic quantization: quantize on the fly / per-tensor at load or runtime (e.g., bitsandbytes 8-bit/4-bit nf4 in Transformers) — convenient, no separate file.

The trade-off, concretely

bits ↓  →  size ↓ (fits more)  ·  speed ↑ (less to read)  ·  quality ↓ (more rounding)

Empirically (good schemes): 8-bit ≈ lossless; 4-bit (Q4_K_M / AWQ / GPTQ) near-lossless on most tasks, small drops on the hardest reasoning/coding; 3-bit noticeable; 2-bit usually only for "fit at any cost." Reasoning and precise coding degrade first and most — measure on your task (Phase 1.07).

3. Mental Model

   WEIGHT 16-bit  ──quantize──►  small int q  +  scale s   (w ≈ q×s, per group)
        2 bytes                    0.5 byte (4-bit) + tiny shared scale

   bits:  16 ───── 8 ───── 6 ───── 5 ───── 4 ───── 3 ───── 2
   size:  full ──────────────── ¼ at 4-bit ────────────── ⅛
   qual:  ████████ lossless ██ near-lossless ██ ▓▓ visible ░ poor
                   └ Q8_0/INT8 ┘ └ Q4_K_M/AWQ/GPTQ ┘  └ emergency only ┘

   HOW SMART:  RTN < imatrix/k-quant < GPTQ ≈ AWQ < QAT   (quality at same bits)
   reasoning/coding lose quality FIRST → always eval YOUR task

Mnemonic: store q × s in fewer bits; smarter scale-picking (AWQ/GPTQ/QAT) buys quality at the same bit-width; 4-bit is the sweet spot, measure the hard tasks.

4. Hitchhiker's Guide

What to look for first: the bit-width (8/4/etc.) and the method (Q*_K / AWQ / GPTQ / QAT). Together they predict size and quality. Default to 4-bit with a good method.

What to ignore at first: the zoo of exotic variants (IQ-quants, 2-bit, per-channel vs per-group nuance). Start at Q4_K_M (GGUF) or AWQ/GPTQ 4-bit (GPU).

What misleads beginners:

• "Lower bits = proportionally worse." It's flat then a cliff — 8 and good-4 are fine; 3 and 2 fall off.
• "All 4-bit is equal." RTN-4bit ≪ AWQ/GPTQ/QAT-4bit at the same size — method matters as much as bits.
• "Benchmarks transfer." A quant that's fine for chat may break structured output or hard reasoning — eval your task.
• "Quantization changes the API model I call." It can — providers may serve a quantized variant (Phase 5.10).

How experts reason: pick the highest bit-width that fits with headroom (02); prefer calibrated/learned methods (AWQ/GPTQ/QAT/imatrix) over RTN; and measure degradation on the actual task (coding pass-rate, reasoning accuracy, JSON validity) rather than trusting "4-bit is fine."

What matters in production: consistent quant across deploys (so behavior doesn't drift), KV-cache quant for concurrency, and a quality gate that fails a build if a quant regresses your eval.

How to debug/verify: run your eval set at FP16 vs each quant; watch for specific failures (math, long-context, structured output) rather than average score; compare size/tok/s to confirm the speed/fit win.

Questions to ask vendors: What precision do you serve this model at? Is it the lab's QAT checkpoint or a community PTQ? Is the KV cache quantized? (These are the serving-fidelity questions.)

What silently gets expensive/unreliable: over-aggressive quant that quietly tanks the hardest 10% of tasks; mixing quant levels across replicas (drift); assuming a screenshot's quality holds on your prompts.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Quantization stores w ≈ q × s — small integers + per-group scales — at 8/4/3/2 bits.
• It works because trained networks are redundant/noise-tolerant — small rounding ≈ small perturbation.
• 8-bit ≈ lossless; good 4-bit (Q4_K_M/AWQ/GPTQ) near-lossless on most tasks; ≤3-bit degrades visibly.
• Method matters as much as bits: RTN < imatrix/k-quant < GPTQ ≈ AWQ < QAT at the same bit-width.
• QAT is train-time (best, lab-provided); everything else is PTQ (post-hoc, cheap).
• GGUF → k-quants (CPU/Mac/llama.cpp); AWQ/GPTQ → safetensors (GPU/vLLM); bitsandbytes → dynamic in Transformers.
• Reasoning, math, and precise coding degrade first — eval those specifically (Phase 1.07).
• KV-cache quantization is separate from weight quant and helps context/concurrency (02).
• Providers may serve quantized variants of "the same" model — a serving-fidelity risk.

9. Observations from Real Systems

• GGUF release pages (Unsloth/TheBloke-style) publish a full ladder of k-quants with size/quality notes — the practical menu (03).
• vLLM/TGI serve AWQ/GPTQ safetensors on GPUs; many production open-weight deployments run 4-bit AWQ (Phase 7).
• Gemma QAT checkpoints demonstrate QAT's edge: 4-bit quality close to FP16 because the model trained for it.
• bitsandbytes load_in_4bit (nf4) is the one-line way to quantize an HF model dynamically for fine-tuning (QLoRA, Phase 13).
• Cheap aggregator endpoints sometimes quantize harder than first-party APIs — the measurable root of provider variance.

10. Common Misconceptions

11. Engineering Decision Framework

CHOOSE A QUANT for model M on device D, task T:
 1. Compute fit at 8/5/4-bit: weights(params,bits)+KV+overhead ≤ D × 0.8.   [02]
 2. Start at the HIGHEST bit-width that fits with headroom.
 3. Pick method by stack:
       GGUF/CPU/Mac  → Q*_K (default Q4_K_M; imatrix for low-bit)   [03]
       GPU/vLLM      → AWQ or GPTQ 4-bit (or FP8 on H100)           [Phase 7]
       fine-tuning   → bitsandbytes nf4 (QLoRA)                     [Phase 13]
       lab offers QAT → use the QAT checkpoint
 4. EVAL on T: FP16 vs quant on coding/reasoning/JSON — not just avg score.  [1.07]
 5. Regression? → go up a bit-width or switch method. Fits but quality holds → ship.
 6. Tight on ctx/concurrency? → add KV-cache quant (orthogonal).             [02]

12. Hands-On Lab

Goal

Quantify the size / speed / quality trade-off by running one model at several precisions on your eval set.

Prerequisites

• A local engine (03/04) and a small golden set (10–20 task items with known answers — Phase 1.07). Include some reasoning/coding/JSON items.

Setup

# Pull the same model at multiple GGUF quants (example with llama.cpp/HF)
for Q in Q3_K_M Q4_K_M Q5_K_M Q8_0; do
  huggingface-cli download <repo>-GGUF "<model>-$Q.gguf" --local-dir ./models || true
done

Steps

1. Record size of each file (ls -lh).
2. Measure speed: run a fixed decode-heavy prompt at each quant; record tok/s and peak memory (01).
3. Measure quality: run your golden set at each quant; score correctness, and separately track reasoning, coding pass-rate, and JSON validity.
4. Tabulate: quant → size, tok/s, memory, overall score, and per-category scores.
5. Find the knee: identify the lowest bit-width where per-category scores are still acceptable — that's your pick.
6. (Optional) Method compare: if available, compare RTN vs imatrix (GGUF) or GPTQ vs AWQ (GPU) at the same 4-bit to see method effect.

Expected output

A trade-off table where size/speed improve monotonically as bits drop while quality is flat then falls — with the knee marked and the first category to degrade identified.

Debugging tips

• Quality cliff far higher than expected → RTN/no-calibration quant; try a k-quant/AWQ/GPTQ build.
• Speed doesn't improve with lower bits → you're prefill/CPU-bound, not bandwidth-bound (01).

Extension task

Add KV-cache quantization at a fixed weight quant and show how many more concurrent/long sequences fit (02).

Production extension

Wire the eval as a quality gate: CI fails if a candidate quant regresses any category beyond a threshold.

What to measure

Size, tok/s, peak memory, overall + per-category quality; the knee; method effect at equal bits.

Deliverables

• A quant trade-off report (size/speed/quality, per-category).
• A recommended quant with justification + the first-to-degrade category.
• (Optional) a method comparison at equal bit-width.

13. Verification Questions

Basic

1. What does quantization store instead of the raw 16-bit weight?
2. Why does quantization barely hurt quality at 8-bit?
3. What's the difference between PTQ and QAT?

Applied 4. Rank RTN, AWQ, GPTQ, QAT by expected 4-bit quality and explain why. 5. Pick a quant + method for: (a) a 24 GB GPU serving a 32B, (b) a Mac running a 70B, (c) QLoRA fine-tuning.

Debugging 6. A 4-bit model is fine for chat but fails structured output. Likely cause and fix. 7. Two "4-bit" builds differ in quality. Why?

System design 8. Design a quantization quality gate that prevents shipping a quant that regresses reasoning.

Startup / product 9. How does serving a quantized open-weight model change your unit economics and your quality-assurance obligations?

14. Takeaways

1. Quantization stores q × s in fewer bits — it's what makes local inference fit and go fast.
2. 8-bit ≈ lossless; good 4-bit near-lossless; ≤3-bit degrades (reasoning/coding/JSON first).
3. Method matters as much as bit-width: AWQ/GPTQ/QAT/imatrix ≫ RTN.
4. GGUF k-quants (local) · AWQ/GPTQ (GPU) · bitsandbytes (dynamic) · QAT (lab-provided).
5. Always eval your task, and remember KV-cache quant is a separate, additive lever.

15. Artifact Checklist

• A size/speed/quality trade-off table across ≥4 precisions.
• Per-category quality (reasoning/coding/JSON), not just average.
• The recommended quant + first-to-degrade category.
• (Optional) a method comparison at equal bit-width.
• A reusable quality-gate eval script.

Up: Phase 6 Index · Next: 07 — MTP and Speculative Decoding