Warmup Guide — Foundations: Scaling Laws, FLOPs/Memory Math & Tradeoffs

Zero-to-senior primer for Phase 00. We start from "what is a neural network actually doing in terms of arithmetic and memory" and end with the five numbers and the one artifact that turn architectural opinion into engineering: the compute budget (6ND / 2N), the memory budget (weights + the KV-cache), the bandwidth ceiling (the roofline, and why decode is memory-bound), the scaling laws (Chinchilla), the serving cost ($/1M tokens), and the weighted tradeoff decision. Every later phase plugs a real mechanism into a slot in the model built here.

Table of Contents

• Chapter 1: A Model Is a Physical System
• Chapter 2: FLOPs — The Compute Budget (6ND and 2N)
• Chapter 3: Memory I — The Weights
• Chapter 4: Memory II — The KV-Cache, the Real Inference Wall
• Chapter 5: The Roofline — Compute-Bound vs Memory-Bandwidth-Bound
• Chapter 6: Scaling Laws & Chinchilla — How Big, How Much Data
• Chapter 7: Throughput & Cost — $/1M Tokens
• Chapter 8: The Tradeoff — Quantifying "Best for These Priorities"
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: A Model Is a Physical System

From zero. A large language model, stripped of all the framing, is a function: it takes a sequence of token IDs and returns a probability distribution over the next token. That function is implemented as a long chain of matrix multiplications interleaved with a few cheap element-wise operations (normalization, activation, residual addition). That is almost the entire computational story — a transformer is, to first order, a stack of matmuls.

Why this framing matters. Once you see the model as "a pile of matmuls over some weights," three physical resources fall out immediately, and every engineering decision you will ever make is about one of them:

The senior's habit. When someone says "let's deploy a 34B model," the senior does not nod. They silently compute: 34e9 × 2 = 68 GB of weights (won't fit one 80 GB GPU once you add the KV-cache and activations), decode is bandwidth-bound so single-stream throughput is roughly HBM_bandwidth / 68 GB ≈ 30 tok/s (too slow without batching), and the $/1M tokens at the target QPS is X. Then they have an actual opinion. This chapter's goal is to make that reflex automatic.

Common misconception. "Bigger model = better, end of story." Bigger raises quality with diminishing returns (Chapter 6) and raises every inference cost linearly forever. The senior move is almost always to find the smallest model that clears the quality bar, then quantize and distill it — because you pay the inference cost on every single request for the life of the product.

Chapter 2: FLOPs — The Compute Budget (6ND and 2N)

What a FLOP is. A floating-point operation: one multiply or one add. A matrix multiply of an m×k by a k×n matrix does m·n·k multiplies and m·n·k adds — about 2·m·n·k FLOPs. The "2" (one multiply + one add per element, a multiply-accumulate or MAC) shows up everywhere.

The forward pass: ~2N per token. Every parameter in the model participates in roughly one MAC per token that flows through it. With N parameters and the factor-of-2 for the MAC, a forward pass over one token costs about

$$ \text{FLOPs}_{\text{fwd}} \approx 2N \quad \text{per token.} $$

This is astonishingly clean and it is the single most useful number in the field. A 7B model costs ~14 GFLOP to advance one token. Generating 500 tokens costs ~7 TFLOP. Memorize 2N.

Training: ~6ND. Training does a forward pass and a backward pass. The backward pass computes gradients with respect to both the inputs and the weights, which is about twice the work of the forward pass. So one training step over a token costs ~2N (fwd) + ~4N (bwd) = ~6N. Over a whole training run of D total tokens:

$$ \boxed{;\text{FLOPs}_{\text{train}} \approx 6 \cdot N \cdot D;} $$

Why you must know this cold. It is the back-of-envelope that sizes a training run, a cluster, and a budget. A 7B model on 1T tokens: 6 × 7e9 × 1e12 = 4.2e22 FLOPs. On a cluster doing 1e18 FLOP/s (an exaflop) at 50% utilization, that's 4.2e22 / 5e17 ≈ 84,000 seconds ≈ a day — and you can now reason about whether your idea is a weekend or a quarter.

Prefill vs decode (same FLOPs, different physics). Processing a 1000-token prompt ("prefill") costs ~2N × 1000 FLOPs, but you do it in one parallel pass over all positions — it's a big, efficient matmul (compute-bound). Generating the next 1000 tokens ("decode") costs the same ~2N × 1000 FLOPs, but you must do it one token at a time, each step a skinny matrix-times-vector that barely uses the GPU's math units (memory-bound — Chapter 5). Same arithmetic, completely different engineering. This distinction drives all of P09.

Common misconception. "Attention dominates the FLOPs." For typical sequence lengths the feed-forward (MLP) layers dominate — they hold ~2/3 of the parameters. Attention's O(T²) cost only overtakes the MLP at long context (the reason FlashAttention and long-context tricks exist). For the 2N estimate, attention's quadratic term is a correction, not the main term, until T gets large.

Chapter 3: Memory I — The Weights

What it is. The weights are N numbers, each stored in some precision. Memory for weights is simply

$$ \text{bytes}_{\text{weights}} = N \times \text{bytes_per_param}. $$

Why precision is a first-class lever. A 70B model in fp16 (140 GB) needs two 80 GB GPUs just for the weights; in int4 (~35 GB) it fits on one with room to spare. Quantization is not a micro-optimization — it changes how many GPUs you need, which changes the cost and the architecture. That is why P06 is a whole phase.

Training memory is much larger than the weights. During training you also hold:

• Gradients: one per parameter (N values).
• Optimizer state: Adam keeps a 1st- and 2nd-moment estimate — 2N values — usually in fp32, plus an fp32 master copy of the weights.
• Activations: the forward-pass intermediates saved for the backward pass (scales with batch and sequence length; activation checkpointing trades compute to shrink this).

A rough fp16-training rule: ~16 bytes/param (2 weights + 2 grad + 4+4 Adam moments + 4 master), so training a 7B model needs ~112 GB before activations. This 2N optimizer-state fact is exactly why ZeRO/FSDP shard it across GPUs (P10).

Common misconception. "We quantized to int4 so training will fit." Quantization is an inference/serving technique for the frozen weights; training still needs full-precision gradients and optimizer state (QLoRA, P05, is the clever exception — it keeps the base frozen and quantized and only trains small adapters in higher precision).

Chapter 4: Memory II — The KV-Cache, the Real Inference Wall

The problem it solves. Autoregressive decoding generates token t+1 by attending over tokens 1..t. Naively, every new token recomputes the keys and values for all previous tokens — O(T²) work to produce a sequence. The KV-cache stores the key and value vectors of every past token so each new step only computes K/V for the one new token and reads the rest from cache. It turns decode from O(T²) into O(T).

The cost. For every layer and every past token you store a key vector and a value vector, each of width d_model (for full multi-head attention):

$$ \text{bytes}{\text{KV}} = 2 \times n{\text{layers}} \times T \times d_{\text{model}} \times \text{bytes_per_elem} \times \text{batch}. $$

The leading 2 is "K and V." Worked example — Llama-2-7B (32 layers, d_model 4096), 4096 context, fp16, one sequence:

$$ 2 \times 32 \times 4096 \times 4096 \times 2 = 2.15\ \text{GB}. $$

Per sequence. Now serve 50 concurrent users and that's ~107 GB of KV-cache — more than the 14 GB of weights, and the actual reason you OOM. At scale, you provision GPUs for the KV-cache, not the weights. Internalize this; it is the most common thing junior candidates miss in a serving design.

The fixes (previewed; built in P02/P09):

• GQA / MQA — share K/V across query heads so the stored width is d_model · n_kv/n_head instead of d_model. With 8 KV heads out of 32, the cache is 4× smaller (0.54 GB above). This is why every modern model uses GQA.
• PagedAttention (P09) — store the cache in fixed blocks like OS pages so you don't waste memory on fragmentation and over-allocation.
• KV quantization — store the cache in int8/int4.
• Sliding-window / shorter context — cap T.

Common misconception. "Long context is just an attention-cost problem." The O(T²) attention compute matters, but the linear-in-T KV-cache memory is usually the binding constraint in production, and it grows with every concurrent request.

Chapter 5: The Roofline — Compute-Bound vs Memory-Bandwidth-Bound

The question. A GPU has two peak rates: how fast it can do math (peak FLOP/s) and how fast it can move data between HBM and the compute units (memory bandwidth, bytes/s). For any operation, which one is the bottleneck? The roofline model answers it with one number.

Arithmetic intensity. Define

$$ I = \frac{\text{FLOPs performed}}{\text{bytes moved}} \quad (\text{FLOP/byte}). $$

The roofline. The attainable performance is

$$ \text{attainable FLOP/s} = \min(\text{peak FLOP/s},; \text{bandwidth} \times I). $$

Plot it and you get a "roof": a slanted line (bandwidth × I) that rises until it hits a flat ceiling (peak FLOP/s). The corner — the ridge point — is at

$$ I_{\text{ridge}} = \frac{\text{peak FLOP/s}}{\text{bandwidth}}. $$

 FLOP/s
   peak ┤        ________________   ← compute-bound (flat roof)
        │       /
        │      /  slope = bandwidth
        │     /
        │    /          ridge point = peak/bandwidth
        └───┴──────────────────────► arithmetic intensity (FLOP/byte)
          memory-bound      compute-bound

• Left of the ridge (I < I_ridge): you finish the math before the data arrives — you are memory-bandwidth-bound. More FLOP/s does nothing; move fewer bytes.
• Right of the ridge (I > I_ridge): you are saturating the math units — compute-bound. Faster memory does nothing; do less math or use better units (tensor cores).

The punchline for LLMs. An A100 has ~312 TFLOP/s (bf16) and ~2 TB/s HBM, so I_ridge ≈ 156 FLOP/byte. Now consider decode: to produce one token you read every weight (~2 bytes) and do ~2 FLOPs per weight → arithmetic intensity ≈ 1 FLOP/byte. That is deeply to the left of 156 — autoregressive decode is hopelessly memory-bandwidth-bound. This is the most important single fact in inference engineering, and it explains:

• why batching helps (you read the weights once and apply them to many sequences, raising I),
• why quantization helps (fewer bytes per weight → more tokens per second),
• why PagedAttention/continuous batching exist (keep I high by keeping the batch full),
• why throwing a faster-FLOPs GPU at a decode workload is often money wasted.

Prefill and training, by contrast, are big dense matmuls with high I → compute-bound → they do benefit from more FLOP/s and tensor cores.

Common misconception. "Our GPU is at 100% utilization, so we're optimal." nvidia-smi utilization means "a kernel was running," not "the math units were busy." A memory-bound decode kernel shows 100% utilization while doing <5% of peak FLOPs. The roofline, not the utilization meter, tells you the truth (P17 goes deeper).

Chapter 6: Scaling Laws & Chinchilla — How Big, How Much Data

Scaling laws. Empirically, test loss falls as a power law in model size, dataset size, and compute, over many orders of magnitude:

$$ L(N) \approx L_\infty + \left(\frac{N_c}{N}\right)^{\alpha_N}. $$

In words: doubling the model gives a predictable, diminishing drop in loss. This is why "just make it bigger" worked for years — and why it eventually stops being worth it.

The compute-optimal question. You have a fixed compute budget C ≈ 6ND. Should you spend it on a bigger model (N) or more data (D)? The Chinchilla result (Hoffmann et al., 2022) found that for compute-optimal training, N and D should scale together, roughly:

$$ D \approx 20 \times N \quad (\text{~20 tokens per parameter}). $$

Why it mattered. Many earlier "large" models were massively under-trained — too big for the data they saw. Chinchilla (70B on 1.4T tokens) beat the 175B GPT-3 (300B tokens) at less compute. The senior lesson: a smaller model trained on more data is often better and much cheaper to serve (smaller N = less inference cost forever).

The inference caveat (post-Chinchilla practice). Chinchilla optimizes training compute. But if you'll serve the model billions of times, you should happily train a smaller model on more than 20×N tokens (past the compute-optimal point) because the extra training cost is paid once and the smaller N saves inference cost forever. This is why models like Llama are trained far past their Chinchilla point (e.g. 7B on ~2T tokens, ~285 tokens/param). Optimize for total cost of ownership, not training compute alone.

Common misconception. "Chinchilla says 20 tokens/param is optimal, so train to exactly there." 20×N is the training-compute-optimal floor; for a model you'll deploy at scale, over-training a small model is the right move. Know both the rule and when to break it.

Chapter 7: Throughput & Cost — $/1M Tokens

Decode throughput ceiling. From the roofline (Chapter 5), single-stream decode is bandwidth-bound: you re-read all the weights every token, so

$$ \text{tok/s}{\text{single}} \approx \frac{\text{bandwidth}}{\text{bytes}{\text{weights}}}. $$

A 7B fp16 model (14 GB) on 2 TB/s HBM: 2e12 / 1.4e10 ≈ 143 tok/s — a ceiling, real systems hit less. Batching reads the weights once for B sequences, so throughput scales ~linearly with batch (until you become compute-bound or run out of KV-cache room):

$$ \text{tok/s}{\text{batch}} \approx B \times \frac{\text{bandwidth}}{\text{bytes}{\text{weights}}}. $$

This is the whole economic argument for batched serving: at batch 32 our 7B model approaches ~4,500 tok/s on the same GPU.

Cost per million tokens. Given a GPU hourly price and a sustained throughput:

$$ \frac{$}{1\text{M tokens}} = \frac{\text{gpu_$/hour}}{\text{tok/s} \times 3600} \times 10^6. $$

At $2/hour and 4,500 tok/s, that's about $0.12 / 1M tokens — and now you can compare an open model on your own GPU against an API's published price with an apples-to-apples number, which is a real decision seniors make weekly.

MFU (Model FLOPs Utilization). achieved_FLOP/s ÷ peak_FLOP/s. Good training runs hit 40–55%; memory-bound decode is far lower (because it's not FLOP-limited at all). MFU is the honest efficiency metric — it tells you how much of the hardware you actually paid for is doing useful model work.

Common misconception. "The API is more expensive than self-hosting." Sometimes — but only once you account for batching (utilization), engineering time, idle GPUs at low traffic, and the quality gap. Compute the $/1M at your real utilization, not at theoretical peak, before you make the build-vs-buy call.

Chapter 8: The Tradeoff — Quantifying "Best for These Priorities"

The core idea. There is no "best model" or "best architecture." There is only best for a given set of priorities. A regulated medical assistant weights quality and safety far above cost; an internal batch summarizer weights cost and throughput above latency. The same two candidate designs can have opposite winners under those two weightings — and proving that to yourself is the entire point of the lab's tradeoff resolver.

The mechanism. Score each candidate 0..1 on each dial (here: quality, latency, cost, memory, safety, where 1 = best). Pick weights for the workload. The candidate score is the weight-normalized sum:

$$ \text{score}(c) = \sum_{d \in \text{dials}} \text{score}_c[d] \cdot \frac{w_d}{\sum_j w_j}. $$

The deciding dial is the one whose weighted (winner − loser) gap is largest — that single sentence is what you write in the decision record: "We chose the local int4-7B because the workload weights cost 4× and it wins decisively on cost, accepting a measured 0.25 quality gap."

Why "deciding dial" beats a raw score. A score of 0.71 vs 0.68 tells a reviewer nothing. "We traded ~0.25 of quality to win 4× on cost, and here's the eval showing the quality gap is acceptable for this use case" is an argument. Seniors win design reviews with the deciding dial and the number, not the aggregate.

The decision record (ADR-lite). For any model/serving decision, write three lines:

• Context — the workload facts and the weights (what we optimized for).
• Decision — the candidate chosen and the deciding dial.
• Consequence — the dial(s) we traded away, and the number that makes the trade acceptable.

Common misconception. "We'll just pick the model with the best benchmark score." Benchmarks measure quality on their distribution at unbounded cost/latency. Production is a constrained optimization; the resolver makes the constraints explicit so the decision is defensible six months later when someone asks "why didn't we use the bigger model?"

Lab Walkthrough Guidance

The lab (lab-01-transformer-cost-modeler) turns this chapter into code. Suggested order (matches the file):

1. FLOPs (training_flops, forward_flops_per_token, inference_flops) — Chapter 2. The only trick is validation; the formulas are 6ND and 2N.
2. Memory (weight_memory_bytes, kv_cache_bytes) — Chapters 3–4. The KV-cache GQA branch is the interesting one: scale d_kv by n_kv/n_head, and require both head counts or neither.
3. Scaling (chinchilla_optimal_tokens/params) — Chapter 6. A round-trip: tokens↔params at ratio 20.
4. Roofline (arithmetic_intensity, roofline_ridge_intensity, roofline_attainable_flops, is_memory_bound) — Chapter 5. The ridge is peak/bandwidth; attainable is the min.
5. Throughput & cost (decode_tokens_per_sec, cost_per_million_tokens, mfu) — Chapter 7. Note batch multiplies throughput.
6. Tradeoff (Candidate.__post_init__, resolve_tradeoff) — Chapter 8. Validate the dials, weight-normalize, find the deciding dial. The test_weights_flip_the_winner test is the soul of the lab.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can derive 6ND (MAC × 3 passes × N × D) and 2N/token without notes.
• You can compute the KV-cache size for (L=32, T=4096, d=4096, fp16) ≈ 2 GB in your head and explain why GQA with 8/32 KV heads makes it 0.54 GB.
• You can state the roofline ridge for an A100 (~156 FLOP/byte) and explain why decode (I≈1) is far left of it and therefore memory-bound.
• You can take any "deploy model X" prompt and produce, in two minutes on paper: training FLOPs, weight + KV-cache memory, a decode $/1M, and a named tradeoff with the deciding dial.

Interview Q&A

• "How many FLOPs to train a 13B model on 2T tokens?" — 6 × 13e9 × 2e12 = 1.56e23. Then reason about cluster-days at a given FLOP/s and MFU.
• "Serve a 70B model — how many 80 GB GPUs, and what limits it?" — fp16 weights = 140 GB → 2 GPUs minimum for weights alone; then add the KV-cache at your concurrency/context (often the binding constraint) and activations; int4 (~35 GB) changes the answer to one GPU. Lead with the KV-cache.
• "Is decoding compute- or memory-bound? Prove it." — memory-bound: I≈1 FLOP/byte vs an A100 ridge of ~156; you re-read all weights per token. Therefore batch and quantize, don't buy more FLOPs.
• "Bigger model or fine-tune a smaller one?" — inference cost is 2N/token forever; prefer the smallest model that clears the eval bar, then quantize/distill — unless measured quality and volume justify the perpetual bill. Cite Chinchilla and TCO, not training compute alone.
• "What did Chinchilla change and when do you ignore it?" — D≈20N is training-optimal; for a model you'll serve at scale, over-train a smaller model past that point because inference cost dominates TCO.
• "Why is nvidia-smi at 100% not the same as efficient?" — utilization = "a kernel ran," not "FLOPs were busy"; memory-bound decode shows 100% util at low MFU. Use the roofline.

References

• Kaplan et al., Scaling Laws for Neural Language Models (2020) — the original power laws.
• Hoffmann et al., Training Compute-Optimal Large Language Models ("Chinchilla", 2022).
• Williams, Waterman, Patterson, Roofline: An Insightful Visual Performance Model (2009).
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM, 2023).
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023).
• "Transformer Inference Arithmetic" (kipp.ly) and "LLM inference speed of light" (the cleanest write-ups of the 2N / KV-cache / bandwidth-bound math).
• NVIDIA A100/H100 datasheets — peak FLOP/s and HBM bandwidth numbers for your roofline.