Warmup Guide — Quantization & Model Compression

Zero-to-senior primer for Phase 06. We start from "what is a number in a computer, and why does a model store billions of them in a format that wastes most of its bits" and end with the full modern quantization stack: the affine/symmetric map onto an integer grid, the round-and-clamp kernel, per-tensor vs per-channel vs per-group granularity, calibration, PTQ vs QAT, the outlier/salient-channel problem, and the three families that solve it — GPTQ (second-order weight error feedback), AWQ (activation-aware weight scaling), and SmoothQuant (difficulty migration) — plus int8/int4/NF4 cliffs, KV-cache quantization, and the runtimes that ship it. Every claim ties back to Phase 00's memory and bandwidth arithmetic.

Table of Contents

• Chapter 1: Why Quantize — Memory and the Bandwidth Wall
• Chapter 2: Fixed-Point — Mapping Reals onto an Integer Grid
• Chapter 3: Affine vs Symmetric, Scale, Zero-Point, Rounding & Clamping
• Chapter 4: Granularity — Per-Tensor, Per-Channel, Per-Group
• Chapter 5: Calibration, PTQ vs QAT
• Chapter 6: The Outlier Problem — Why Activations Are Harder Than Weights
• Chapter 7: GPTQ — Second-Order, Layer-Wise Error Minimization
• Chapter 8: AWQ & SmoothQuant — Moving the Difficulty Around
• Chapter 9: int8 vs int4 vs NF4 — The Quality Cliffs
• Chapter 10: KV-Cache Quantization & The Deployment Runtimes
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Quantize — Memory and the Bandwidth Wall

From zero. A trained model is, physically, a long list of N numbers — the weights. By default each is stored in fp16 or bf16: 2 bytes, 16 bits. Quantization is the act of storing each number in fewer bits — int8 (1 byte), int4 (half a byte) — and reconstructing an approximation of the original when you need it. It is a lossy compression of the weights.

Why bother — the two budgets from Phase 00. Recall the two physical ceilings a model lives under:

• Memory. Weights take N × bytes_per_param. A 70B model is 140 GB in fp16 (two 80 GB GPUs), 70 GB in int8 (still two, barely), 35 GB in int4 (one GPU, with headroom for the KV-cache). Quantization changes how many GPUs you need — a deployment-architecture decision.
• Bandwidth. Decode is memory-bandwidth-bound: each generated token re-reads every weight from HBM, so single-stream throughput is ≈ bandwidth / weight_bytes. Halve the bytes per weight and you roughly double the tokens per second. Quantization is a latency lever, not just a memory one.

$$ \text{tok/s}{\text{decode}} ;\approx; \frac{\text{HBM bandwidth}}{\text{bytes}{\text{weights}}} \quad\Longrightarrow\quad \text{fewer bytes} \Rightarrow \text{more tokens/s.} $$

The senior's reflex. "Can we serve this?" is not a vendor question. It's: weights at the candidate precision, plus the KV-cache at peak concurrency × context (Phase 00), versus HBM. If it doesn't fit or it's too slow, the first lever is precision, because it's the only one that attacks both budgets at once.

Common misconception. "Quantization is a quality hack that makes the model worse." int8 weight quantization is, for most LLMs, nearly lossless — the model has far more precision than it uses. The interesting engineering only starts at int4 and below, and even there the modern calibrated methods (GPTQ/AWQ) keep the loss small. The default assumption should be "quantize unless proven harmful," not the reverse.

Chapter 2: Fixed-Point — Mapping Reals onto an Integer Grid

What an integer grid is. An n-bit integer can represent 2^n distinct values — int8 gives 256 levels, int4 gives 16. Quantization picks 2^n evenly-spaced points on the real line and snaps every weight to the nearest one. Storing a weight becomes storing which grid point (a small integer code) instead of the full float.

real:    -1.0        -0.5         0.0         0.5         1.0
           |           |           |           |           |
grid:   ●---●---●---●---●---●---●---●---●---●---●---●---●---●---●   (16 levels = int4)
codes:  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14
                          ▲
                  step = "scale" (reals per integer)

The two numbers that define the grid. A uniform affine grid is fully described by:

• scale s — the spacing between grid points, in real units per integer step.
• zero-point z — the integer code that maps to the real value 0.0 (so the grid can be off-centre, e.g. for an all-positive activation).

Reconstruction is then a single affine formula:

$$ \hat{x} = (q - z)\cdot s, \qquad q \in [q_{\min}, q_{\max}]. $$

Why fixed-point and not just "smaller float." fp16 spends bits on a floating exponent so it can represent both 1e-8 and 1e8. A layer's weights live in a narrow, known range, so a fixed-point grid spends all its bits on resolution inside that range — which is exactly why a 4-bit fixed-point quant of a weight matrix beats a hypothetical 4-bit float. The model section that needs huge dynamic range (the activations, Chapter 6) is precisely where fixed-point struggles.

Common misconception. "More bits is the only way to lower error." More bits helps (Chapter 9), but on a fixed bit budget the error is set by how well you place the grid — the scale, the granularity (Chapter 4), and which values you protect (Chapters 7–8). Placement, not just bit count, is where the field lives.

Chapter 3: Affine vs Symmetric, Scale, Zero-Point, Rounding & Clamping

The affine (asymmetric) map. Given a real range [xmin, xmax] and an integer grid [qmin, qmax], fit the grid to cover exactly that range:

$$ s = \frac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}}, \qquad z = \mathrm{round}!\left(q_{\min} - \frac{x_{\min}}{s}\right). $$

The zero-point z is "the code that lands on 0.0." For an all-positive range like a post-ReLU activation [0, 8], z = qmin so the whole grid sits above zero — no levels wasted on negatives that never occur. This asymmetry is why activations usually use affine.

The symmetric map. Force z = 0 and centre the grid on zero, using the larger magnitude:

$$ s = \frac{\max(|x_{\min}|, |x_{\max}|)}{\max(|q_{\min}|, |q_{\max}|)}, \qquad z = 0. $$

Now the integer 0 is the real 0.0. Symmetric is the default for weights, which are roughly zero-centred, and it makes the matmul cheaper (no zero-point cross-terms to subtract). The cost is a wasted half-grid if the data is lopsided — a fine trade for weights.

Rounding. q = round(x / s) + z. Round-to-nearest is the standard; the residual it leaves (x − x̂) is at most half a step:

$$ |x - \hat{x}| \le \frac{s}{2} \quad \text{for any in-range } x. $$

That s/2 bound is the single most useful fact about uniform quantization — it's why smaller s (finer grid, more bits, or per-channel scales) means less error, and it's a test in the lab.

Clamping — the part that bites. After rounding you clamp into [qmin, qmax]:

q = clamp(round(x/s) + z, qmin, qmax)

Without the clamp, an out-of-range value would overflow the integer and wrap (a huge positive weight becomes a huge negative one — catastrophic). With the clamp it saturates at the grid edge. But saturation is itself a problem: if you set s from the true min/max, a single outlier stretches the range and makes s coarse for everyone else. This tension — clamp outliers (lose them) vs widen the range (lose precision on the bulk) — is the seed of the entire salient-channel story in Chapter 6.

Common misconception. "Symmetric and affine are interchangeable, pick either." They encode different priors: symmetric assumes zero-centred (weights), affine assumes a possibly-shifted range (activations). Using symmetric on a [0, 8] activation throws away half your levels.

Chapter 4: Granularity — Per-Tensor, Per-Channel, Per-Group

The question. How many (scale, zero_point) pairs do you keep for one weight matrix? More pairs = better fit = more metadata to store. Three standard answers:

• Per-tensor — one (s, z) for the whole matrix. Cheapest metadata. Fails when channels have different magnitudes, because the single scale is dictated by the loudest channel and is far too coarse for the quiet ones.
• Per-channel — one (s, z) per output channel (per row). Each row gets a grid matched to its own range. This is the free default for weights — the metadata (one float per row) is negligible next to the matrix itself, and the accuracy win is large.
• Per-group — one (s, z) per group of g weights within a row (typical g = 64 or 128). Finer than per-channel, more metadata; this is what GPTQ's group_size=128 and GGUF's Q4_K blocks actually use to make int4 viable.

per-tensor          per-channel              per-group (g=4)
┌──────────────┐    ┌──┐ each row            ┌──┬──┐ each block
│  one s,z for │    │s0│ its own s,z         │s │s │ of 4 its own s,z
│  everything  │    │s1│                     ├──┼──┤
│              │    │s2│                     │s │s │
└──────────────┘    └──┘                     └──┴──┘
 cheapest, worst    free win for weights     finest, what int4 needs

Why this is the cheapest lever. Going per-channel costs one extra float per row and buys you a grid matched to each channel. The lab's soul test makes this concrete: a tensor with a quiet row and a ~10× louder row has lower MSE per-channel than per-tensor, every time, because the quiet row stops sharing the loud row's coarse scale.

$$ \text{error}{\text{per-tensor}} ;\propto; s{\text{global}} = \frac{\max_i |x_i|}{\text{levels}}, \qquad \text{error}{\text{per-channel,row }r} ;\propto; s_r = \frac{\max{i\in r} |x_i|}{\text{levels}}. $$

When one row's max dwarfs another's, s_global ≫ s_r for the quiet row — pure waste that per-channel reclaims.

Common misconception. "Per-channel works for activations too, so just do that." Weights are quantized per output channel (the column dimension of the matmul output) — static, computed once. Activations are dynamic and laid out so that per-channel quant would break the matmul's reduction; the standard is per-tensor or per-token for activations, which is why activation outliers (Chapter 6) are so painful.

Chapter 5: Calibration, PTQ vs QAT

Calibration. To pick a scale you need the range [xmin, xmax]. For weights you just read them (static, data-free min/max — what the lab does). For activations you must observe them, so you run a small calibration set (a few hundred representative samples) through the model and record the activation ranges. Choices here matter: pure min/max is outlier-sensitive; percentile clipping (drop the top 0.1%) or MSE/entropy-minimizing range search (TensorRT's calibrator) trade a little clipping for a much better scale on the bulk.

PTQ — Post-Training Quantization. Take a finished fp16 model and quantize it without retraining, using calibration. Cheap (minutes to hours), no training data or gradients needed. Modern PTQ — GPTQ and AWQ — gets int4 working well, so PTQ is the default; you only escalate when it leaves quality on the table.

QAT — Quantization-Aware Training. Simulate quantization during training (or fine-tuning): insert "fake-quant" nodes that round-and-clamp in the forward pass but pass gradients through via the straight-through estimator (treat round as identity for the backward pass, since its true gradient is zero almost everywhere). The model learns weights that survive quantization. Higher quality at low bits, but you need the training pipeline and data. Reach for QAT when PTQ's int4 gap is unacceptable and you can afford to train.

PTQ:   fp16 model ──calibrate──▶ quantized model        (cheap, no gradients)
QAT:   fp16 model ──train with fake-quant in the loop──▶ quantized model   (better, costly)
                       ▲ straight-through estimator passes gradients past round()

Common misconception. "We're quantizing, so we need QAT." Almost never to start. int8 is fine PTQ; int4 with GPTQ/AWQ is fine PTQ for most models. QAT is the specialist tool for squeezing the last accuracy out of aggressive (sub-4-bit) regimes — measure the PTQ gap before you pay for it.

Chapter 6: The Outlier Problem — Why Activations Are Harder Than Weights

The empirical fact. In large transformers, a tiny number of feature dimensions develop massive-magnitude activations — outliers 10–100× larger than the rest — and they appear in a consistent set of channels across tokens. Dettmers et al. (LLM.int8()) showed these emerge past ~6.7B parameters and that they are functionally critical: zero them out and the model's accuracy collapses. The signal lives in the outliers.

Why this kills naive quantization. Recall the s = max/levels scale. One outlier of magnitude 100 in a channel whose other values are ~1 forces s ≈ 100/levels. With int8 that's s ≈ 0.8, so every non-outlier value (the bulk of the signal) rounds to one of a handful of grid points — effectively quantized to ~2–3 bits. The outlier you kept precise; everything else you destroyed.

without outlier:  values in [-1,1], s small, 256 levels well used
with one 100×:    s = 100/127 ≈ 0.8  →  the [-1,1] bulk gets ~3 usable levels
                  ↑ one number wrecks the grid for everyone else

Why activations are harder than weights. Weights are static and zero-centred, quantize per-channel cheaply, and have no extreme outliers — int8 weights are nearly free. Activations are (a) dynamic (range varies per input), (b) quantized per-tensor/per-token (the matmul layout forbids cheap per-channel), and (c) full of these outlier channels. So the hard problem is "int8 the activations too" (needed for true integer matmul speedups), and that's what SmoothQuant and the LLM.int8() mixed-precision decomposition exist for.

The three responses (Chapters 7–8 detail them).

• Keep outliers in higher precision — LLM.int8() runs the outlier dimensions in fp16 and the rest in int8, then recombines. Simple, robust, but the mixed path costs speed.
• Protect salient weights — AWQ scales the weight columns that multiply large activations up so they keep their bits.
• Move the difficulty — SmoothQuant migrates the activation outlier magnitude into the weights (which quantize easily) by a per-channel rescale.

Common misconception. "Outliers are noise — clip them." They are the opposite of noise; they carry critical signal. Clip them and you get exactly the accuracy collapse LLM.int8() warned about. The art is keeping them while still using few bits for the rest.

Chapter 7: GPTQ — Second-Order, Layer-Wise Error Minimization

The objective. Forget min/max. For one linear layer, GPTQ asks the right question directly: choose the quantized weights Ŵ that minimize the output error on calibration data:

$$ \arg\min_{\hat{W}} ; \big| WX - \hat{W}X \big|_2^2, $$

where X is the calibration activations. This is a least-squares problem, and its curvature is the Hessian H = XXᵀ — which encodes how much each weight's rounding error actually matters for the output (a weight that multiplies large/correlated activations is "expensive" to round badly).

The mechanism — Optimal Brain Quantization (OBQ) made fast. GPTQ quantizes the weights in a layer one column at a time, and after fixing each column to its grid value, it pushes the rounding error into the not-yet-quantized columns, using the Hessian to compute the optimal compensating update:

for each column j (left → right):
    round W[:,j] to the grid                 # commit this column
    err = W[:,j] - Ŵ[:,j]                     # the rounding residual
    W[:, j+1:] -= err · (H⁻¹ row for j)       # spread it onto remaining columns (OBQ update)

So later columns absorb the damage earlier columns took — error feedback, guided by second-order (Hessian) information. GPTQ's contribution was the algorithmic trickery (lazy batched updates, a Cholesky reformulation) that makes this tractable for billion-parameter layers in a few GPU-hours instead of geological time.

Why it works at int4. Pure round-to-nearest treats every weight independently and ignores that errors correlate through X. GPTQ's error feedback means the aggregate output error is minimized, not the per-weight error — which is exactly what you care about. It's why int4 GPTQ models are usable where naive int4 falls apart.

Common misconception. "GPTQ retrains the model." It does not touch the training loss or use labels — it's PTQ. It only solves a per-layer least-squares reconstruction with a calibration set. No backprop through the network, no optimizer over the task.

Chapter 8: AWQ & SmoothQuant — Moving the Difficulty Around

AWQ — Activation-aware Weight Quantization. AWQ's premise: weights are not equally important — the ones that multiply large activations dominate the output, so protect them. It does not keep mixed precision; instead it scales the salient weight columns up by a per-channel factor s_j before quantizing (and folds the inverse 1/s_j into the preceding layer / the activation), so the full-precision product is unchanged but the salient columns now occupy more of the integer grid:

$$ y = Wx = (W \cdot \mathrm{diag}(s)),(\mathrm{diag}(s)^{-1} x), \qquad s_j = (\text{act_scale}_j)^{\alpha}. $$

A scaled-up column spans a wider range, so after quantization its relative rounding error shrinks — and because it multiplies a big activation, that's exactly the error that mattered. AWQ searches the exponent α (and a clip) on a calibration set to minimize the activation-weighted output error ‖Wx − Ŵx‖. This is precisely the lab's AWQ soul test: a matrix with one salient column has far lower output error after scaling than naive int4, and the search picks a non-zero α.

naive int4:   salient column shares the grid, gets crushed → big output error
AWQ:          scale salient column by act**α  → it keeps its bits  → small output error
              (the matching activation is scaled down by 1/(act**α), so y is preserved)

AWQ is data-light (no backprop, no Hessian inverse — just a scale search), which makes it fast and popular for int4 deployment.

SmoothQuant — migrate difficulty from activations to weights. The problem (Chapter 6): you want int8 activations too, but activations have the outliers and quantize per-tensor. SmoothQuant's move is to rescale per channel so the activation gets smoother (outliers shrunk) and the weight gets harder (it absorbs the magnitude) — and weights quantize easily, so the trade is a win:

$$ y = (X \cdot \mathrm{diag}(s)^{-1}),(\mathrm{diag}(s),W), \qquad s_j = \frac{\max_i |X_{ij}|^{\alpha}}{\max_i |W_{ij}|^{1-\alpha}}. $$

The α (typically 0.5) balances how much difficulty you push from the activation side to the weight side. After smoothing, both activations and weights are int8-friendly, enabling true integer matmuls (W8A8) — the throughput win, not just memory.

The shared idea. AWQ and SmoothQuant are the same trick applied with different intent: a per-channel diag(s) rescale that is mathematically free (it cancels through the matmul) but moves the quantization difficulty to where it's cheapest to pay — AWQ protects salient weights, Smooth makes activations quantizable. GPTQ, by contrast, attacks the weight rounding directly with error feedback. Real int4 deployments often combine them.

Common misconception. "AWQ and GPTQ are competitors; pick the better one." They optimize different things (AWQ: which weights to protect via scaling; GPTQ: how to round given error feedback) and are often complementary. The practical choice is usually about runtime support and speed, not a universal accuracy winner.

Chapter 9: int8 vs int4 vs NF4 — The Quality Cliffs

int8. ~256 levels, 1 byte. For LLM weights, int8 is effectively lossless with per-channel scales — the model has far more precision than it uses. Use it freely; the only real work is int8 activations (Chapter 8). This is the "free" tier.

int4. 16 levels, half a byte. Naive int4 falls apart (outliers, Chapter 6). With per-group scales and a calibrated method (GPTQ/AWQ), int4 is the workhorse of consumer-grade LLM deployment — small, measurable quality loss. This is where the engineering is. The lab's "int8 beats int4" test is this cliff in miniature: same data, ~400× more MSE at 4 bits with a uniform grid.

NF4 — NormalFloat 4. The key realization (QLoRA, Dettmers et al.): weights are roughly normally distributed, so a uniform int4 grid wastes levels in the tails. NF4 places its 16 levels on the quantiles of a unit normal — information-theoretically optimal for normal data — so more levels land where the weights actually are. It beats uniform int4 at the same bit count. QLoRA pairs it with double quantization (quantize the per-block scales themselves to save a bit more) and uses it to fine-tune a 4-bit-frozen base with small fp16 LoRA adapters (Phase 05).

uniform int4:  ●--●--●--●--●--●--●--●   evenly spaced, wastes levels in sparse tails
NF4:           ●-●--●---●----●---●--●-●  spaced by normal quantiles → matches the data

The cliff, summarized.

Common misconception. "int4 is half of int8, so half the quality." Quality vs bits is a cliff, not a slope — int8 is nearly free, int4 needs calibration to be usable, and below 4 bits quality drops fast. Always quote a measured eval number for the precision you ship, never a vibe.

Chapter 10: KV-Cache Quantization & The Deployment Runtimes

Quantize the KV-cache too. Phase 00 showed the KV-cache, not the weights, is the inference memory wall — 2 · L · T · d · bytes · batch, growing with concurrency and context. Those bytes are a quantization target: storing K/V in int8 halves the cache (so you fit 2× the concurrency or context for free), int4 quarters it. The K/V tensors also have outliers (the key cache especially), so production KV quant uses per-token or per-channel scales and sometimes keeps the most recent tokens in higher precision. It trades a small perplexity hit for a large capacity win — frequently the right call at high concurrency.

The runtimes (where this ships). You rarely write the kernel; you pick a stack:

fp8 — the new tier. Hopper/Ada GPUs have native fp8 (E4M3/E5M2). Unlike int8 it keeps a floating exponent, so it handles outliers more gracefully and is becoming the default for both weights and activations on new hardware (W8A8 with less calibration pain). Know it exists and that it's an 8-bit float, not an integer scheme.

Common misconception. "Quantization is one technique." It's a stack: a grid (uniform/NF4/fp8), a granularity (tensor/channel/group), a calibration method (min-max/percentile/GPTQ/AWQ/Smooth), a target (weights / activations / KV-cache), and a runtime that fuses it into fast kernels. Senior answers name which choices and why, not just "we quantized."

Lab Walkthrough Guidance

The lab (lab-01-quantization-engine) turns these chapters into code. Suggested order (matches the file):

1. compute_scale_zeropoint — Chapter 3. Affine: s = range/steps, z = round(qmin − xmin/s). Symmetric: larger magnitude, z = 0. Guard the degenerate xmin == xmax range so you never divide by zero.
2. quantize_affine / dequantize_affine — Chapters 2–3. The whole game is clamp(round(x/s)+z) and (q−z)·s. The clamp is the part the tests probe.
3. _qrange + quantize_tensor / dequantize_tensor — Chapters 3–4. Symmetric grid is [-(2^(n-1)−1), 2^(n-1)−1]; per-channel = one (s,z) per row. The test_per_channel_beats_ per_tensor test is a soul of the lab.
4. quant_error_mse — the objective. Flatten, mean of squared differences.
5. AWQ (apply_awq_scaling, awq_reconstruct, awq_output_error, awq_scale_search) — Chapter 8. Scale columns by act**α, quantize, undo the scaling, and measure the activation-weighted output error ‖Wx − Ŵx‖ — not raw weight MSE. The search picking a non-zero α is the AWQ soul test.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can derive the affine s and z and the symmetric variant without notes, and say which one weights vs activations use and why.
• You can state and explain the s/2 round-trip bound, and why clamping (not wrapping) outliers is essential — and what that clamp costs.
• You can explain mechanically why per-channel beats per-tensor on mixed-magnitude channels, and where per-group fits.
• You can explain the salient/outlier problem and how GPTQ (Hessian error feedback), AWQ (scale salient columns), and SmoothQuant (migrate difficulty to weights) each attack it.
• You can place int8/int4/NF4 on a quality-vs-size curve, name the cliff, and say why NF4 beats uniform int4.

Interview Q&A

• "Walk me through quantizing a weight matrix to int4." Pick a range per channel (min/max or better), compute s = max/levels (symmetric, z=0), q = clamp(round(W/s)), store codes + the per-row scales; dequant is q·s. Mention per-group for int4 to make it actually work.
• "Affine vs symmetric — when each?" Symmetric (z=0) for weights (zero-centred, cheaper matmul); affine for activations (possibly shifted ranges, e.g. post-ReLU [0,8]) so you don't waste levels on negatives that never occur.
• "Why does per-channel beat per-tensor?" A single per-tensor scale is set by the loudest channel, so quiet channels get a grid far too coarse — effectively a few bits. Per-channel gives each channel a matched scale; the metadata is one float per row, negligible.
• "Why are activations harder to quantize than weights?" Activations are dynamic, quantized per-tensor/per-token (layout forbids cheap per-channel), and contain outlier channels 10–100× the rest that carry critical signal — one outlier wrecks the shared scale.
• "What problem do outliers cause and what's the evidence they matter?" They force a coarse scale that crushes the bulk. LLM.int8() showed they emerge past ~6.7B params and that zeroing them collapses accuracy — they're signal, not noise.
• "Explain GPTQ in one breath." Per-layer, minimize ‖WX − ŴX‖; quantize columns left-to-right and push each column's rounding error onto the remaining columns via the Hessian (OBQ error feedback). It's PTQ — no retraining, just a calibration set.
• "Explain AWQ." Salience is set by activations; scale the weight columns that multiply large activations up by act**α before quantizing (fold the inverse into activations), so salient columns keep their bits; search α to minimize activation-weighted output error.
• "Explain SmoothQuant and how it differs from AWQ." Same diag(s) rescale trick, but to make activations quantizable: migrate outlier magnitude from activations (hard) into weights (easy) so you can do int8 W8A8 matmuls. AWQ protects salient weights; Smooth enables int8 activations.
• "PTQ or QAT?" PTQ by default (int8 free; int4 fine with GPTQ/AWQ). QAT (fake-quant + straight-through estimator in training) only when PTQ's low-bit gap is unacceptable and you can train.
• "int8 vs int4 vs NF4?" int8 ≈ lossless for weights; int4 needs per-group + calibration to be usable (the cliff); NF4 places 16 levels on normal quantiles so it beats uniform int4 on normally-distributed weights (QLoRA).
• "Would you quantize the KV-cache?" Yes at high concurrency/long context — it's the memory wall (Phase 00). int8 K/V halves it; watch the key-cache outliers and consider keeping recent tokens higher-precision.
• "Name the runtimes and when you'd use each." bitsandbytes (easy, QLoRA), GPTQ/AWQ (quality int4 serving), GGUF/llama.cpp (CPU/edge), TensorRT-LLM (max GPU throughput, SmoothQuant). fp8 on new hardware for W8A8 with less calibration pain.

References

• Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022) — the outlier-feature discovery and the mixed-precision int8 path.
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022) — Hessian-guided, layer-wise error-feedback weight quantization (OBQ-fast).
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023) — protect salient weight channels by activation-aware scaling.
• Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022) — migrate quantization difficulty from activations to weights (W8A8).
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023) — NF4, double quantization, 4-bit base + LoRA adapters.
• Nagel et al., A White Paper on Neural Network Quantization (Qualcomm, 2021) — the clean reference for scale/zero-point, per-channel, PTQ vs QAT, and the straight-through estimator.
• NVIDIA TensorRT-LLM, llama.cpp/GGUF, bitsandbytes, AutoAWQ, and AutoGPTQ docs — the runtimes that turn these algorithms into shipped kernels.