Phase 2 — Transformer Foundations
The "under the hood" phase. Every serving optimization, cost behavior, and model-page number you'll meet later is a consequence of what's in this phase. You don't implement transformers here — you learn them deeply enough to reason about memory, latency, and cost on sight.
Why this phase matters
Phase 1 gave you the vocabulary; Phase 2 gives you the mechanisms behind it. After this, KV cache OOMs, prefill/decode latency, MoE memory surprises, and reasoning-model cost are all predictable rather than mysterious.
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | The Transformer Big Picture | Map any cost/perf question to a component |
| 01 | Token Embeddings | Reason about embeddings, RoPE, and context extension |
| 02 | Attention and Self-Attention | Explain O(n²) cost, GQA, FlashAttention, sliding windows |
| 03 | Feed-Forward Layers | See where params/knowledge live; understand MoE's target |
| 04 | LayerNorm and Residuals | Use the residual-stream model; know RMSNorm/pre-norm |
| 05 | Autoregressive Generation | Reason about serial decode, latency, and speedups |
| 06 | KV Cache | Compute KV memory; understand PagedAttention/prefix caching |
| 07 | Prefill vs Decode | Attribute latency to the right phase and fix it |
| 08 | MoE and Dense Models | Decode 26B-A4B; predict memory vs speed |
| 09 | Reasoning Models | Route by difficulty; control reasoning cost/latency |
How to work through it
Read in order — each builds on the last (embeddings → attention → FFN → norm/residual → generation → KV cache → prefill/decode → MoE → reasoning). Do the labs: they produce a memory/KV calculator and latency-measurement tools you'll reuse in Phases 6–8.
Phase 2 artifacts
- An architecture-tracing script + memory/KV calculator (GQA- and MoE-aware)
- From-scratch attention, FFN (+SwiGLU), RMSNorm/residual block, and an autoregressive generator
- Latency benchmarks: prefill-vs-decode, TTFT/TPOT, and a reasoning on/off + difficulty router
- Notes/diagrams: the symptom→component map, residual stream, and the total-vs-active param rule