The Transformer — Big Picture

Phase 2 · Document 00 · Transformer Foundations Prev: Phase 2 Index · Next: 01 — Token Embeddings

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Every LLM you will ever serve, route, or fine-tune is a transformer. You don't need to implement one, but you must understand it at the behavioral and cost level — because KV cache, quantization, PagedAttention, prefill/decode, and MoE all exist as direct consequences of this architecture. Without the big picture you cannot reason about why a model needs 140 GB, why generation slows as context grows, why long-context serving is expensive, or why some optimizations work and others don't. This document is the map for the rest of Phase 2, which zooms into each component.

2. Core Concept

Plain-English math primer (read first — everything in Phase 2 is built from these five ideas)

You do not need a math degree to understand transformers. You need five small ideas, each of which is just "organized arithmetic." We build them from zero, with examples and code. If you know these, the rest of Phase 2 is readable.

1. A vector = a list of numbers (a point in space). A token is represented inside the model as a vector — say [0.2, -1.3, 0.8, ...]. The number of entries is the hidden dimension (e.g. 4096). Purpose: a vector lets the model place each word in a "meaning space" where distance/direction encode relationships.

cat   = [0.9, 0.1, 0.3]
kitten= [0.8, 0.2, 0.3]   # close to "cat" → similar meaning
car   = [-0.5, 0.9, 0.0]  # far from "cat" → different meaning

2. The dot product = a similarity/alignment score between two vectors. Multiply matching entries and add them up. Big positive = pointing the same way (similar); near zero = unrelated. This single operation is the heart of attention ("which earlier tokens are relevant to me?").

def dot(a, b): return sum(x*y for x, y in zip(a, b))
dot([0.9,0.1,0.3], [0.8,0.2,0.3])  # = 0.83  (cat·kitten, high → similar)
dot([0.9,0.1,0.3], [-0.5,0.9,0.0]) # = -0.36 (cat·car, low → unrelated)

3. A matrix = a table of numbers that transforms vectors (a "linear layer"). "Multiplying a vector by a matrix" (x @ W) turns one vector into another — it's how the model reshapes information at every step. A matrix with shape (in, out) takes an in-sized vector and returns an out-sized one. Purpose: the model's learned weights are mostly these matrices; "running the model" is mostly multiplying vectors by matrices.

# a 3→2 linear layer: turns a 3-number vector into a 2-number vector
x = [1.0, 2.0, 3.0]
W = [[0.1, 0.4],
     [0.2, 0.5],
     [0.3, 0.6]]            # shape (3, 2)
out = [sum(x[i]*W[i][j] for i in range(3)) for j in range(2)]
# out = [1.4, 3.2]

GPUs are fast at exactly this (huge parallel matrix multiplies) — which is why LLMs run on GPUs.

4. Softmax = turn a list of raw scores into probabilities that sum to 1. The model's final scores over the vocabulary are called logits (raw, unbounded numbers). Softmax exponentiates and normalizes them into a probability distribution, so we can "pick the next token." Bigger logit → bigger probability; the gap gets amplified.

import math
def softmax(scores):
    m = max(scores)
    exps = [math.exp(s - m) for s in scores]    # subtract max for numerical stability
    total = sum(exps)
    return [e/total for e in exps]
softmax([2.0, 1.0, 0.1])  # → [0.66, 0.24, 0.10]  (sums to 1.0)

Use case: this is the last step before choosing a token, and the dial temperature (Phase 1.03) simply divides the logits before softmax to make the distribution sharper or flatter.

5. "Dimension" and why shapes matter. Every vector has a length (its dimension); every matrix has a shape (rows, cols). The whole model is a pipeline of shape-compatible multiplications: tokens → vectors of size hidden → through matrices → back to a vector of size vocab (one logit per possible next token). When we say a model is "4096-dimensional with a 128K vocab," those are the sizes flowing through this pipe.

Mantra for all of Phase 2: vectors carry meaning; matrices transform it; the dot product measures relevance; softmax turns scores into a choice. Everything below — attention, FFN, normalization — is a specific arrangement of these four moves.

Plain English

A transformer processes a sequence of tokens by letting every token look at (attend to) every earlier token, then transform what it learned. Stack that operation N times and you get a network that, given some tokens, produces a probability distribution over the next token. Run it in a loop and it writes text.

Technical depth — the decoder-only stack

Modern LLMs (GPT, Llama, Claude, Gemini, Qwen) are decoder-only transformers trained to predict the next token given all previous tokens. The data path:

1. Token IDs — text → integers via the tokenizer ("Hello world" → [9906, 1917]). (Phase 1.01)
2. Embedding layer — each ID indexes a learned matrix (vocab_size, hidden_dim), producing a vector. (01)
3. Positional information — order is injected, today usually via RoPE (rotary position embedding), which encodes relative position inside attention. (01)
4. N transformer layers, each with:
   • Multi-head self-attention — each token forms a Query, compares to all Keys, and mixes Values: softmax(QKᵀ/√d_k)·V. (02)
   • Feed-forward network (FFN) — a per-token 2-layer MLP, ~4× wider than hidden; where most "knowledge" lives. (03)
   • Residual connections + normalization around both sublayers (stable gradients, trainable depth). (04)
5. Output projection — final hidden state → hidden_dim → vocab_size logits; softmax → probabilities; sampler picks a token. (05)

It became dominant because attention captures long-range dependencies, parallelizes on GPUs (unlike serial RNNs), and scales predictably with data and parameters.

Why these internals show up in your job

• KV cache (06) — caches K/V for prior tokens so decode doesn't recompute them; grows with layers × heads × context × batch.
• Prefill vs decode (07) — parallel prompt processing (compute-bound, → TTFT) vs serial generation (bandwidth-bound, → TPOT).
• Quantization (Phase 1.06) — weights dominate memory; fewer bits = smaller, faster, slight quality loss.
• MoE (08) — replace one FFN with many experts, activate a few per token: huge total params, modest active compute.

3. Mental Model

Tokens (text)
   ↓ Tokenizer
Token IDs
   ↓ Embedding table  (+ positional info, e.g. RoPE)
Token Vectors  ── one per token, hidden_dim wide
   ↓ Layer 1 ┌── Self-Attention: each token looks at all previous tokens
   │         ├── + Residual / Norm
   │         ├── FFN: independent per-token transform (where knowledge lives)
   │         └── + Residual / Norm
   ↓ Layer 2 … Layer N   (the assembly line repeats)
Final Hidden State (last position)
   ↓ Linear projection → Logits (one score per vocab token)
   ↓ Softmax + Sampler
Next Token ID → detokenize → append → REPEAT

Mantra: attention mixes information across tokens; the FFN transforms each token; residuals/norm make depth trainable; the loop makes it generate.

4. Hitchhiker's Guide

What to understand first: the flow tokens → attention → generation, and which component drives which cost (attention/KV → memory & decode; FFN → compute; weights → memory). Skip the math on the first pass.

What to ignore at first: exact attention derivations, CUDA kernels, gradient flow. Learn behavior and cost; the math is easy later once it's motivated.

What misleads beginners:

• "The model reads the prompt, then thinks." No — it processes the whole prompt in parallel (prefill), then emits one token at a time (decode).
• "More layers = smarter." Data, post-training, and calibration matter as much as depth.
• "Attention sees everything, so long context is fine." It can attend to everything, but quality degrades over long context ("lost in the middle").

How experts reason: they map any performance/cost question to a component — "that's a KV-cache memory issue," "that's FFN compute," "that's a decode-bandwidth limit" — and reach for the matching optimization.

What matters in production: layers × heads × context × batch sets KV memory; weights set base memory; FFN dominates FLOPs (and is what MoE makes sparse).

How to verify: load a small model and print n_layer, n_embd, n_head, vocab_size; count params; estimate memory and KV size with the formulas below and check against observed usage.

Questions to ask: Is it dense or MoE? How many layers/heads (KV cost)? What position encoding (long-context behavior)? What precision are the weights?

What silently gets expensive: long context × high concurrency (KV blow-up), and assuming "fits in memory" from weights alone while ignoring KV cache.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Modern LLMs are decoder-only transformers trained on next-token prediction.
• Weights dominate base memory: FP16 ≈ params × 2 bytes (70B ≈ 140 GB); 4-bit ≈ params × 0.5 byte.
• KV cache per token ≈ 2 × layers × heads × head_dim × bytes — it grows with context × batch.
• Attention mixes across tokens; the FFN transforms each token independently.
• FFN holds most parameters/compute; MoE makes it sparse (few experts per token).
• Prefill is compute-bound (→TTFT); decode is bandwidth-bound (→TPOT).
• RoPE is the common modern position encoding; it shapes long-context behavior.
• "Can attend to N tokens" ≠ "reasons well over N tokens" — long context degrades.

9. Observations from Real Systems

• vLLM's PagedAttention exists because KV cache (a direct architectural artifact) is the serving bottleneck — pure cause-and-effect from this diagram.
• Hugging Face config.json exposes num_hidden_layers, hidden_size, num_attention_heads, vocab_size — the exact knobs in Section 7, readable for any open model.
• Mixtral/Qwen MoE swap the FFN for expert FFNs, showing the architecture's modularity (huge total params, modest active compute).
• llama.cpp/Ollama let you watch memory = weights + KV grow as you raise --ctx-size, making the KV formula tangible.
• Long-context model cards report context windows far beyond where recall stays reliable — the architecture accepts more than it reasons over well.

10. Common Misconceptions

11. Engineering Decision Framework

Map a symptom to a component, then to a fix:

OOM under long context / concurrency → KV cache (layers×heads×ctx×batch)
   → shorter ctx, fewer concurrent seqs, PagedAttention, more memory (Phase 5/6/7).

Base memory too high to load        → weights
   → quantize (4-bit), smaller model, tensor parallelism (Phase 6/7).

Slow first token (TTFT)             → prefill (compute)
   → prompt/prefix caching, shorter prompt (Phase 5/7).

Slow per token (TPOT)               → decode (bandwidth)
   → smaller model, speculative decoding/MTP (Phase 6/7).

Want big-model quality at small-model speed → MoE (sparse FFN)
   → choose an MoE variant IF total-param memory fits (Phase 1.02/2.08).

12. Hands-On Lab

Goal

Trace text through a real small transformer: inspect its architecture, count params, estimate memory/KV, and read next-token probabilities.

Prerequisites

• Python 3.10+, ~2 GB free; CPU is fine (GPT-2 is tiny).

Setup

pip install transformers torch

Steps

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "openai-community/gpt2"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

inputs = tok("The transformer architecture", return_tensors="pt")
print("IDs:", inputs["input_ids"])
print("Tokens:", [tok.decode([t]) for t in inputs["input_ids"][0]])

with torch.no_grad():
    out = model(**inputs, output_hidden_states=True)

c = model.config
print(f"vocab={c.vocab_size} hidden={c.n_embd} layers={c.n_layer} heads={c.n_head}")
print("hidden-state shapes:", [tuple(h.shape) for h in out.hidden_states[:3]], "...")

probs = torch.softmax(out.logits[0, -1], dim=-1)
top = torch.topk(probs, 5)
print("Top-5 next tokens:", [(tok.decode([i]), round(p.item(),4)) for p,i in zip(top.values, top.indices)])

n = sum(p.numel() for p in model.parameters())
print(f"params={n/1e6:.1f}M  FP16 weight mem≈{n*2/1e9:.3f} GB")
# KV per token (rough): 2 * layers * heads * head_dim * 2 bytes
head_dim = c.n_embd // c.n_head
print(f"KV/token≈{2*c.n_layer*c.n_head*head_dim*2/1e3:.1f} KB")

Expected output

• Architecture numbers (GPT-2: 12 layers, 768 hidden, 12 heads, 50257 vocab).
• Hidden states shaped (1, seq_len, hidden).
• Sensible top-5 next tokens; a param count (~124M) and memory estimate.

Debugging tips

• Download blocked? Use any small public causal LM you have cached.
• Shapes confusing? (batch, seq_len, hidden) — one vector per token per layer.

Extension task

Change the input; watch top-5 shift. Add tokens and watch seq_len (and thus KV) grow.

Production extension

Estimate weights + KV for a 7B model at FP16 and 4-bit for a 4K and 32K context; compare to a 16 GB and 24 GB GPU (precursor to the Phase 6 calculator).

What to measure

Architecture dims, param count, FP16 vs 4-bit weight memory, KV/token, and how top-5 changes with input.

Deliverables

• An architecture table for GPT-2.
• Param count + memory + KV estimates.
• Top-5 predictions for 3 inputs with notes.

13. Verification Questions

Basic

1. What is a decoder-only transformer trained to do?
2. What does self-attention do vs what the FFN does?
3. Why do residual connections matter?

Applied 4. Why does a 70B model need ~140 GB in FP16, and ~35 GB at 4-bit? 5. Compute KV/token for a model with 32 layers, 32 heads, head_dim 128, FP16.

Debugging 6. Generation slows as the conversation grows. Which component explains it? 7. A model loads fine but OOMs under concurrent long-context traffic. What's the cause?

System design 8. You need big-model quality but tight latency on fixed hardware. Which architectural choice helps, and what's the catch?

Startup / product 9. Explain to a non-technical cofounder, using this architecture, why long-context features raise both cost and latency — and one mitigation.

14. Takeaways

1. LLMs are decoder-only transformers: attention mixes across tokens, the FFN transforms each token, residuals/norm make depth trainable.
2. Weights dominate base memory; KV cache dominates the concurrency limit.
3. Prefill (compute, →TTFT) and decode (bandwidth, →TPOT) are distinct.
4. FFN holds most compute/knowledge; MoE sparsifies it.
5. Every Phase-2 optimization (KV cache, quantization, MoE, speculative decoding) is a consequence of this architecture.
6. "Can attend to N tokens" ≠ "reasons well over N tokens."

15. Artifact Checklist

• Code: the architecture-tracing script.
• Architecture table for one model (layers/hidden/heads/vocab).
• Memory estimate: weights + KV for a 7B at two precisions/contexts.
• Diagram: redraw the assembly-line stack from memory.
• Notes: which component drives which cost (the symptom→component map).
• Top-5 prediction log for 3 inputs.

Next: 01 — Token Embeddings