Brother Talk — Quantization & Model Compression

Off the record. The stuff I'd tell you over a beer, not in a design review.

Quantization is the cheapest superpower in this whole curriculum. Most of the impressive phases — attention, RLHF, distributed training — take real work to show value. Quantization is two lines of math (x̂ = (q−z)·s) and it routinely turns "we need two GPUs" into "we need one" and "this is too slow" into "ship it." When you're the person who can shrink the model in an afternoon and prove the quality held, you become very hard to argue with on the deployment plan. Learn it cold; it pays out constantly.

The outlier thing is the whole game, and almost nobody internalizes it. Everyone can repeat "int4 is smaller." Very few can tell you why naive int4 is bad — that a handful of channels carry 10–100× the magnitude and the rest of the signal, and one of them drags the scale coarse for everything else. Once you really get that, GPTQ, AWQ, SmoothQuant, NF4, LLM.int8() all collapse into "different answers to where do I spend my bits." You'll understand a whole research field from one idea, and you'll sound like you've read every paper because you understood the one that matters.

int8 is free; stop overthinking it. There's a reflex to treat any quantization as a dangerous quality compromise that needs a committee. For LLM weights, int8 per-channel is effectively lossless — the model has way more precision than it uses. Just do it. The actual engineering, the part worth your attention and a real eval, starts at int4. Don't spend a week defending the free thing.

Measure, or you're guessing. The single biggest way people embarrass themselves here is quoting a precision with no number — "we'll use int4, it's fine." Quality vs bits is a cliff, not a slope, and where your specific model falls off depends on the model. Run the eval at the precision you'll ship. "int8 was free, int4 cost 0.6 perplexity which fails our bar, so we shipped int8" is a senior sentence. "int4 should be fine" is how you get paged.

The KV-cache is the trap, again. I'll keep saying it across phases because it keeps catching people: you'll quantize the weights, feel clever, and still OOM in production because the KV-cache was the wall the whole time (Phase 00 told you this). Quantizing the thing that's actually full beats quantizing the thing that was already fine. Always ask "what's actually full?" before you optimize.

Don't confuse memory wins with speed wins. Quantization saves memory unconditionally, but it only saves latency if the runtime fuses the dequant into the matmul kernel. I've watched teams quantize, save the GPUs, and get zero speedup because the kernel path was unfused — they moved fewer bytes but added overhead. If you care about tok/s, benchmark tok/s, and pick a runtime (AWQ/GPTQ kernels, TensorRT, llama.cpp) that's built for it.

What's actually worth caring about: the grid math (you'll use it forever), the outlier intuition (it's the whole field), per-channel as the free default, and measuring the quality at your shipping precision. What's not worth losing sleep over: memorizing GPTQ's Cholesky reformulation or AWQ's exact clip search — know what they optimize and why, and let the library do the rest. Nobody's going to ask you to re-derive the OBQ update at a whiteboard; they'll ask why int4 needs calibration, and that you should own.

The career framing. This is the phase that makes you the person who ships, not just the person who prototypes. Anyone can get a model running on a fat GPU; the senior gets it running on the hardware the company can afford, fast enough for the SLA, at the quality the eval demands — and quantization is the lever that makes all three true at once. Get this cold and you stop asking "what GPU do we need?" and start answering it. Now go make pytest green.