Warmup Guide — RNNs & Language Modeling

Zero-to-expert primer for Phase 03: what a language model is (the probability framing that survives every architecture change), and recurrent networks — the architecture whose failures explain why transformers look the way they do.

Table of Contents

• Chapter 1: Language Modeling — The Definition That Never Changes
• Chapter 2: Cross-Entropy and Perplexity
• Chapter 3: The Recurrent Idea
• Chapter 4: Backpropagation Through Time and the Vanishing Gradient
• Chapter 5: LSTM and GRU — Gating as Gradient Plumbing
• Chapter 6: Sampling from a Language Model
• Chapter 7: Why RNNs Lost — and Where They Won
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Language Modeling — The Definition That Never Changes

A language model assigns probability to sequences. By the chain rule, exactly:

$$P(w_1, \ldots, w_n) = \prod_{i=1}^{n} P(w_i \mid w_1, \ldots, w_{i-1})$$

So the entire field reduces to one learnable function: given a prefix, output a probability distribution over the next token. Every architecture in this curriculum — n-grams, the RNN you build here, the transformer in Phase 04, GPT-4 — is a different parameterization of $P(w_i \mid w_{<i})$. Generation is just repeated sampling from it. Internalizing this framing is the phase's real product: when later phases discuss KV caches or speculative decoding, they're discussing engineering of this conditional distribution's evaluation, nothing more.

The n-gram baseline (know it; it calibrates everything): approximate the condition by the last $k{-}1$ words and estimate by counting. Fails by sparsity — most 5-grams never occur even in huge corpora (smoothing/backoff is the classic patchwork) — and by having no notion of similarity: counts for cat sat teach nothing about kitten sat. Neural LMs fix both at once: embeddings give similarity (Phase 02), and a parametric function generalizes across contexts.

Chapter 2: Cross-Entropy and Perplexity

Training minimizes cross-entropy: average negative log-probability the model assigns to the actual next token:

$$\mathcal{L} = -\frac{1}{N}\sum_{i} \log P_\theta(w_i \mid w_{<i})$$

— equivalently, maximum likelihood. Perplexity is its exponential, $\text{PPL} = e^{\mathcal{L}}$: the effective branching factor ("as uncertain as a fair choice among PPL options"). Calibration numbers worth carrying: a uniform model over vocab V has PPL = V; a character-level model on English text reaching PPL ~3–4 (≈1.6–2.0 bits/char) is learning real structure; word/subword PPL of strong LLMs on WikiText runs single digits. (The full measurement discipline — tokenizer dependence, sliding windows — is in the model-accuracy track's Phase 09; here you just need the loss-curve intuition: watch bits-per-character fall during training and know what a given level "feels like" in samples. The lab makes you do exactly that.)

Chapter 3: The Recurrent Idea

Feed-forward nets take fixed-size inputs; text is variable-length. The recurrent answer: process one token at a time, carrying a fixed-size hidden state $h_t$ as a running summary of everything seen:

$$h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b), \qquad y_t = W_{hy} h_t$$

Three properties define the design:

• Parameter sharing across time — the same $W$ at every step (like convolution shares across space): generalization across positions, and any-length sequences.
• O(1) state: memory of the whole past is compressed into $h$ — exactly the property that makes RNN inference cheap (foreshadowing: this is what Mamba resurrects, model-accuracy Phase 02 Ch. 8).
• Inherently sequential: $h_t$ needs $h_{t-1}$ — training cannot parallelize across time. Hold that thought for Chapter 7.

Chapter 4: Backpropagation Through Time and the Vanishing Gradient

Training unrolls the recurrence into a deep computation graph (one "layer" per time step) and backpropagates — BPTT. The gradient from a loss at step $t$ to the state at step $k$ passes through a product of Jacobians:

$$\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i} ,\text{diag}(\tanh') , W_{hh}^\top$$

A product of $t-k$ matrices: if their spectral norms sit below 1, the product decays exponentially — gradients from distant errors vanish, and the network simply cannot learn long-range dependencies (it could represent them; it can't be taught them). Norms above 1 explode instead — sudden loss spikes, NaNs.

The standard mitigations (all in your lab): gradient clipping for explosion (cap the global norm — crude, universal, still used in every LLM training run today), truncated BPTT (backprop only K steps — bounds cost, also bounds learnable dependency length), careful init (orthogonal $W_{hh}$), and — the real fix — architectural: Chapter 5.

Chapter 5: LSTM and GRU — Gating as Gradient Plumbing

The LSTM's move: add a cell state $c_t$ — a conveyor belt modified only by element-wise, gated operations:

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$

with forget gate $f_t$, input gate $i_t$, candidate $\tilde{c}t$, and output gate $o_t$ producing $h_t = o_t \odot \tanh(c_t)$ — each gate a small sigmoid network of $(x_t, h{t-1})$.

Why this fixes vanishing: the gradient path along $c$ is multiplication by $f_t$ per step — no repeated $W_{hh}$ matrix product, no tanh-derivative shrinkage. With $f_t \approx 1$, gradients flow back nearly unattenuated for as long as the forget gate chooses. The gates make memory learned and content-dependent: keep this, overwrite that. (Squint and you can see both the residual stream of transformers and Mamba's selective state — gating-as-gradient-highway is one of deep learning's most recycled ideas.) Practical detail the lab uses: initialize forget-gate bias positive (~1.0) so training starts in "remember" mode.

GRU: the 2014 simplification — merges cell and hidden state, two gates (update/reset), ~25% fewer parameters, usually within noise of LSTM quality. The default when you want recurrence cheap.

Chapter 6: Sampling from a Language Model

The model outputs logits → softmax → a distribution. How you pick from it shapes everything users see (this section is permanent knowledge — identical for your char-RNN and for GPT-4):

• Greedy (argmax): deterministic, repetitive, gets trapped in loops ("the the the") — fine for short factual continuations only.
• Temperature $T$: divide logits by $T$ before softmax. $T \to 0$ approaches greedy; $T = 1$ is the model's honest distribution; $T > 1$ flattens toward chaos. The lab's most instructive experiment: the same checkpoint at T = 0.3 / 0.8 / 1.5 — coherence vs creativity as one knob.
• Top-k: zero out all but the k highest logits, renormalize — a hard cutoff on the tail where degenerate tokens live.
• Top-p (nucleus): keep the smallest set whose cumulative probability ≥ p — adaptive cutoff: narrow when the model is confident, wide when uncertain; generally preferred over fixed k.
• Combinations apply in order (temperature → top-k → top-p), and every serving stack (Phase 09) implements precisely this pipeline per token.

Chapter 7: Why RNNs Lost — and Where They Won

The scorecard against what's coming in Phase 04:

Transformers won on the training column — parallelism let them eat the whole internet, and the O(1) gradient path made long-range learning easy rather than heroic. But read the inference column: RNNs are the better inference shape, and that trade resurfaces constantly in your career — streaming/edge workloads, the KV-cache memory wall (Phase 09), and the state-space-model renaissance (Mamba) which is explicitly "RNN inference with parallelizable training." Phase 03 isn't history class; it's the other pole of a tradeoff you'll navigate professionally.

Lab Walkthrough Guidance

Lab 01 — Char-RNN (Karpathy's classic, built honestly):

1. Data first: character vocab over your corpus (Shakespeare is traditional), contiguous batching with correct (input, target=input-shifted-by-1) alignment — off-by-one here trains a copy machine; test the alignment on a tiny string.
2. Implement the vanilla RNN cell from Chapter 3's equations yourself (then optionally swap nn.LSTM); train with truncated BPTT — carry the hidden state across chunks (detached!) so the model sees long context without unbounded graphs.
3. Add gradient-norm clipping; log the pre-clip norm — watching it spike is the vanishing/exploding chapter made visible.
4. Track bits-per-character; sample at fixed prompts every N steps at several temperatures — the qualitative arc (noise → words → grammar → style) is the most instructive training curve in the curriculum; save the samples.
5. Compare vanilla vs LSTM on the same budget: loss curves and long-range behavior (does a quote opened 200 chars ago get closed?).

Success Criteria

You are ready for Phase 04 when you can, from memory:

1. Write the chain-rule factorization and explain why every LM reduces to next-token distribution modeling.
2. Connect cross-entropy ↔ likelihood ↔ perplexity and calibrate a bits-per-char number.
3. Derive (sketch) the Jacobian-product argument for vanishing/exploding gradients and name the four mitigations.
4. Explain the LSTM cell-state gradient path and why $f_t$ replaces $W_{hh}$-products.
5. Implement temperature/top-k/top-p from logits on a whiteboard, with what each fixes.
6. Reproduce the RNN-vs-transformer scorecard and argue the inference column's modern relevance.

Interview Q&A

Q: Why did transformers replace LSTMs — and what did we give up? Two structural wins: training parallelism across sequence positions (the GPU-era scaling unlock) and O(1) gradient paths between any two tokens (long-range learning by construction instead of through a gated bottleneck). We gave up O(1) inference state — transformers pay a KV cache that grows with context and dominates serving memory. Naming the loss is the senior half of the answer; SSMs/Mamba exist precisely to claw it back.

Q: Your generation loops endlessly repeating a phrase. Diagnose across the stack. Decoding first: greedy or near-zero temperature makes repetition self-reinforcing (the repeated phrase becomes ever more likely in context) — raise temperature, add top-p, or apply a repetition penalty. If it persists: degenerate model (undertrained, or trained on duplicated data — Phase 10's dedup matters here). The mechanism to articulate: argmax decoding + a model that locally overweights recent n-grams = a fixed point; sampling breaks the loop stochastically.

Q: What does gradient clipping actually do, and what does it not do? It rescales the gradient norm when it exceeds a threshold — direction preserved, magnitude capped — turning explosion (rare, catastrophic steps) into survivable ones. It does nothing for vanishing (you can't rescale a signal that's already ~0), which needs architectural fixes (gating, residuals) — the two pathologies are opposite and people conflate them; the distinction is the question's point. Still standard in LLM training (spikes from data/loss anomalies), so it's not retro knowledge.

References

• Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks (2015) — the lab's spiritual source; read it with samples open
• Hochreiter & Schmidhuber, Long Short-Term Memory (1997)
• Pascanu et al., On the difficulty of training recurrent neural networks (2013) — arXiv:1211.5063 — the vanishing/exploding analysis + clipping
• Cho et al., Learning Phrase Representations using RNN Encoder–Decoder (2014) — GRU
• Holtzman et al., The Curious Case of Neural Text Degeneration (2020) — arXiv:1904.09751 — nucleus sampling and why greedy degenerates
• Olah, Understanding LSTM Networks — colah.github.io — the canonical visual walkthrough
• Jurafsky & Martin, ch. 3 (n-grams) and ch. 8–9 (RNNs) for the textbook depth