Phase 00 — Foundations: Scaling Laws, FLOPs/Memory Math & Tradeoffs

The phase that builds the mental model and the arithmetic every later phase plugs into. Before you implement attention, fine-tune a model, or design a serving stack, you have to be able to answer — on a whiteboard, in two minutes — how much compute does this take, how much memory, how fast can it possibly go, what does it cost, and what am I trading? This is the round-1 filter of every senior AI interview and the first slide of every design review.

Why this phase exists

The difference between an engineer who "uses LLMs" and a Senior AI Engineer is that the senior treats a model as a physical system with a compute budget, a memory budget, and a bandwidth ceiling, not as a magic API. Almost every important decision in this curriculum — quantize or not, fine-tune or RAG, single GPU or sharded, bigger model or distilled smaller one, what batch size, what context length — is downstream of five numbers you can compute on paper:

1. FLOPs — 6ND to train, ~2N per token to run. Compute is the training bill and part of the inference bill.
2. Weight memory — N × bytes. The thing everyone quotes.
3. KV-cache memory — the thing everyone forgets, and the thing that actually OOMs your server at scale.
4. The roofline — whether you are compute-bound or memory-bandwidth-bound, which decides your entire optimization strategy. (Spoiler: decode is bandwidth-bound, and that one fact explains PagedAttention, batching, and quantization.)
5. Cost & the tradeoff — $/1M tokens, and a weighted resolver that proves "best model" is a category error — there is only "best for these priorities."

Get fluent here and the rest of the track is filling in mechanisms behind numbers you already respect.

Concept map

                        ┌─────────────────────────────────────┐
                        │   A model is a physical system       │
                        └─────────────────────────────────────┘
                          │            │             │
              ┌───────────┘     ┌──────┘      ┌──────┘
              ▼                  ▼             ▼
        COMPUTE (FLOPs)    MEMORY (bytes)   BANDWIDTH (bytes/s)
        6ND train          weights N·b       roofline ridge
        2N / token         KV-cache 2·L·T·d  decode is BW-bound
              │                  │             │
              └────────┬─────────┴──────┬──────┘
                       ▼                ▼
                  $ / 1M tokens    Chinchilla (D≈20N)
                       │                │
                       └───────┬────────┘
                               ▼
                    TRADEOFF RESOLVER
            quality · latency · cost · memory · safety
        (same two designs flip winners when weights change)

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Size a serving deployment: given a 13B model, 1k concurrent users, 8k context — compute weights, KV-cache at peak, and the GPU count. Show that the KV-cache, not the weights, sets the number.
• Defend "distill, don't scale": compute the perpetual inference bill of a 70B vs a distilled 13B at your QPS; show the breakeven volume.
• Pick compute-optimal: given a fixed FLOP budget, use Chinchilla to choose N and D; show why the bigger-but-under-trained option is both worse and more expensive to serve.
• Explain the roofline to a new hire: why throwing more FLOPs at decode does nothing, and why batching + quantization do.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can derive 6ND and 2N/token from first principles (MACs × passes).
• You can compute a KV-cache size in your head for a given (L, T, d_model) and explain why GQA shrinks it.
• You can state whether prefill and decode are compute- or memory-bound, and why.
• You can turn any "deploy model X" prompt into FLOPs + memory + $/1M + a named tradeoff.

Key takeaways

• Training is a one-time 6ND; inference is 2N per token, forever. At scale the bill, the latency, and the carbon live on the inference side — which is why this curriculum spends whole phases on quantization, serving, and the KV-cache.
• The KV-cache is the inference memory wall. Weights are fixed; the cache grows with batch × context and is what actually OOMs you. GQA/MQA and quantized KV exist to fight it.
• Decode is memory-bandwidth bound. You re-read every weight per token, so the lever is bytes moved (quantize, batch to amortize), not more FLOPs. This single fact explains half of inference engineering.
• "Best model" is meaningless; "best for these priorities" is engineering. The tradeoff resolver makes that concrete — and the deciding dial is the one sentence you put in the ADR.

Next: Phase 01 — Tokenization & the Multimodal Input Pipeline.