Hitchhiker's Guide — Foundations: Scaling Laws & the Cost Math

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember."

The 30-second mental model

A model is a pile of matmuls over N weights. Three resources govern everything: compute (FLOPs), memory (bytes), bandwidth (bytes/s). Training costs 6ND once; inference costs 2N per token forever. The weights are the number everyone quotes; the KV-cache is the number that OOMs you. Decode is memory-bandwidth-bound, so the levers are batch and quantize, not more FLOPs. There's no "best model," only "best for these priorities."

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "Can a 70B fit on one 80GB GPU?"
70e9 * 2 = 140 GB fp16  → no (need 2);  70e9 * 0.5 = 35 GB int4 → yes (+ KV headroom)

# "How fast can a 7B decode, single stream?"
2e12 (HBM B/s) / 14e9 (fp16 weights) ≈ 143 tok/s ceiling  → batch to go faster

# "KV-cache for 13B, 8k ctx, 40 layers, d=5120, 100 users, fp16?"
2 * 40 * 8192 * 5120 * 2 * 100 ≈ 670 GB  → KV-cache, not weights, sets the GPU count

# "Train 7B on 2T tokens — how big a job?"
6 * 7e9 * 2e12 = 8.4e22 FLOPs;  / (5e17 FLOP/s @ 50% MFU) ≈ 1.9 days

The framework one-liners (where these numbers live in real tools)

# Count params (the N in every formula)
sum(p.numel() for p in model.parameters())            # PyTorch
# Estimate FLOPs
from deepspeed.profiling.flops_profiler import FlopsProfiler
# Measured throughput / MFU
# vLLM logs tokens/s; nvidia-smi dmon for bandwidth; torch.profiler for kernels
# KV-cache config knobs that change the memory math:
#   num_key_value_heads (GQA), max_model_len (T), gpu_memory_utilization (vLLM)

War stories

• The KV-cache OOM at 3pm. Load tests passed at batch 8; production hit batch 60 at peak and OOM'd — because nobody multiplied the KV-cache by concurrency. The weights fit fine; the cache didn't. Capacity = weights + KV-cache at peak concurrency × context + activations. Always.
• The "100% GPU but slow" ticket. nvidia-smi showed 100%; latency was terrible. It was a memory-bound decode at ~4% MFU — the GPU was busy waiting on HBM, not computing. We fixed it with bigger batches (continuous batching) and KV quantization, not a faster GPU.
• The bigger-model regret. Team shipped a 70B because it scored 2 points higher on a benchmark. The inference bill was 5× and p99 latency missed SLO. A distilled 13B + RAG would have cleared the actual quality bar at a fifth of the cost. They optimized the benchmark, not the TCO.
• The under-trained giant. Pre-Chinchilla, a team scaled params and starved the model of data; a half-size model on 3× the tokens beat it at less compute and served cheaper.

Vocabulary (rapid-fire)

• FLOP — one multiply or add. MAC = multiply-accumulate ≈ 2 FLOPs.
• KV-cache — stored keys/values so decode is O(T) not O(T²); the inference memory wall.
• Roofline / ridge — min(peak, bw·I); ridge = peak/bw, the compute/memory crossover.
• Arithmetic intensity — FLOPs per byte moved.
• MFU — fraction of peak FLOPs actually used; honest efficiency metric.
• TTFT / TPOT — time to first token (prefill) / time per output token (decode).
• Chinchilla — compute-optimal D≈20N.
• TCO — total cost of ownership; inference cost × lifetime, the real objective.

Beginner mistakes

• Quoting weight memory and forgetting the KV-cache (and multiplying by concurrency).
• Trying to fix memory-bound decode with a faster-FLOPs GPU instead of batching/quantizing.
• Treating nvidia-smi utilization as efficiency.
• Picking the model by benchmark score with no cost/latency constraint.
• Confusing training memory (huge: grads + Adam) with inference memory (weights + KV).
• Forgetting that inference cost is paid per request, forever — the asymmetry that makes distillation/quantization/PEFT the default senior move.

The one thing to take away

Before you have an opinion about any model decision, compute FLOPs, weight memory, KV-cache, and $/1M — and name the deciding dial. If you can't, you're guessing; if you can, you're engineering.