Brother Talk — The Transformer From Scratch

Off the record. The stuff I'd tell you over a beer, not in a design review.

This is the phase that separates "I use transformers" from "I understand them," and the gap is enormous. Almost everyone in this field can call model.generate(). A shocking number of people with "ML" or "AI" in their title cannot write attention on a whiteboard, cannot tell you what √d_k is for, and have never once thought about what the causal mask is actually doing. The day you can build a GPT forward pass from nothing — embeddings, attention, the mask, RoPE, the KV-cache — you stop being a user of the magic and become someone who can fix it. That's the whole game.

Write it by hand, even though it's "just calling PyTorch" in real life. I know it feels pointless to implement matmul with a triple loop when torch has a fused kernel. Do it anyway. The reason seniors can debug a broken model at 2 a.m. is that they once built the thing with their bare hands and know in their body what each line does. When the loss is suspiciously low, the senior thinks "mask leak" before the junior has finished re-reading the stack trace. You earn that reflex here, once, and keep it forever.

The causal mask will save your job someday. I'm serious. At some point you'll see a model with a gorgeous training curve that generates garbage in production, and a room full of people will be baffled because "the loss went down, it must be working." You'll ask "are we sure the mask isn't leaking the future?" — and you'll be right, because a model that can peek at the answer during training learns nothing useful for inference. Tattoo this: low train loss + broken generation = suspect the mask. That one diagnosis is a story you'll tell for years.

The KV-cache equality is the most beautiful thing in this lab — don't skip the soul test. It's easy to treat the cache as a performance hack and move on. But sit with why it's correct: the last query, under a causal mask, attends to exactly the tokens already in the cache. That's not an approximation or an engineering shortcut — it's an identity. Once you really feel that, the entire serving stack (PagedAttention, continuous batching, all of P09) makes sense, because it's all built on that one guaranteed-equal trick. Make test_kv_cache_equals_full_recompute pass and understand it cold.

Don't get hypnotized by attention. Attention gets all the press — the paper is literally called "Attention Is All You Need." But ~2/3 of the parameters live in the boring feed-forward, and most of the FLOPs at normal context lengths are there too. The model stores what it knows in the MLP and moves information around with attention. Knowing that the FFN is where the mass is will make you right in a dozen conversations about quantization, MoE, and pruning where everyone else is staring at the attention.

You will be tempted to memorize the equations. Don't — derive them. Anyone can recite softmax(QKᵀ/√d_k)·V. The senior can tell you why the √d_k is there (the dot-product variance grows with dimension and saturates softmax) and why RoPE encodes relative position (rotation, and the dot product of two rotated vectors depends only on the angle difference). Interviewers don't want the formula; they want to watch you reconstruct it. Practice explaining it out loud to a wall.

What's actually worth caring about: the attention equation and what each term means, the causal mask, why √d_k, where the parameters live (~2/3 FFN), and the KV-cache identity. What's not worth losing sleep over: memorizing the exact GELU constant, whether your toy uses pre- or post-norm, the precise SwiGLU d_ff ratio. Get the load-bearing ideas into your bones; look the constants up.

The career framing. This phase is where the curriculum stops being about numbers around the model (Phase 00) and becomes about the model itself. Everything ahead — autograd differentiates this forward pass, fine-tuning patches these linear layers, quantization shrinks these weights, serving caches these keys and values — is a modification of the machine you build right here. Own this one cold and every later phase is "a change to a thing I already understand," not new magic. That's the difference between learning ten disconnected tools and understanding one system deeply. Now go make pytest green — and don't skip the soul test.