Cheat Sheet — Frontier Pre-Training

Everything worth having memorized, on one page. If you can reproduce this from memory you can hold your own in a pre-training design review.

The core equations

Memory

KV cache = 2 · L · n_kv · d_head · T · B · bytes
ZeRO-1 shards optimizer states   (75% of the total — cheapest big win)
ZeRO-2 adds gradients
ZeRO-3 adds parameters           (an all-gather per layer)

70B model, mixed-precision Adam: 1.12 TB unsharded. An H100 has 80 GB. You must shard.

Hardware constants

Ridge point = peak FLOP/s ÷ bandwidth. H100 ≈ 296 FLOP/byte. Below it → memory-bound.

Energy ratios (Horowitz, ISSCC 2014): a DRAM read costs ~20,000× an 8-bit integer add. This is why quantization is an energy lever first.

Numbers to know

C = 6ND — where it breaks

MoE

capacity   = capacity_factor × tokens × k / E
L_balance  = E · Σ f_i · P_i        f = token-slot fraction (discrete)
                                    P = mean router probability (differentiable)
L_z        = mean( logsumexp(logits)² )
L_total    = L_task + α·L_balance + γ·L_z      α ≈ 0.01,  γ ≈ 1e-3

Prefill vs decode — never confuse them

This asymmetry is why prefill/decode disaggregation exists, and why the Flash 2.0 fix was pipelined prefill specifically.

The scaling-law workflow

1. FIX the recipe (how depth/LR/batch scale with N, D) — the law describes THIS
2. Design the ladder: geometric budgets, maximize log-C spread, replicate one point
3. IsoFLOPs: fix C, sweep N, fit a parabola in log N, take the vertex
4. Fit power laws N_opt ∝ C^a, D_opt ∝ C^b   →  CHECK a + b ≈ 1
5. Or fit L = E + A/N^α + B/D^β   — Huber loss, log-space residuals
6. Bootstrap a confidence interval (nonlinear model, unknown noise model)
7. Compare baseline vs candidate laws AT THE TARGET C; report the CROSSOVER
8. State the extrapolation distance out loud: "3 decades beyond our largest ablation"

Decision gate: a delta smaller than your confidence interval is not a result.

Kaplan vs Chinchilla, in one box

The bug: reading loss mid-run is a biased estimator, because much of the improvement comes from the LR decay at the end. A uniform bias would be absorbed into E and change nothing; it is the non-uniformity across the ladder that tilts the exponents.

Tokens per parameter, historically: GPT-3 1.7 → Chinchilla 20 → Llama-3-70B 214 → Llama-3-8B 1875. A 1000× swing driven first by a methodology fix, then by serving economics.

Feinberg's three verticals

Distillation in one sentence: variance reduction — a better teacher reduces bias.

Interview one-liners

• "Backward is exactly 2× forward" — each forward matmul becomes two backward matmuls, dX = dY·Wᵀ and dW = Xᵀ·dY, each the same size.
• "35% MFU is an accounting identity" — the matmul unit idles during vector ops, memory traffic, collectives and the optimizer step. Read the breakdown; it is your agenda.
• "Active for FLOPs, total for memory."
• "I wouldn't train both and compare" — fit a law for each, evaluate at the target C, report the delta with intervals plus the crossover, because curves cross.
• "That difference is inside our error bars."
• "Prefill is compute-bound, decode is memory-bound" — never apply one's optimization to the other.
• "Power is the bill" — a DRAM read costs ~20,000× an integer add.
• "A scaling law describes your recipe, not the universe."

The five things that will actually kill your run

1. Loss spike — decide in minutes: skip the batch, lower LR, or roll back.
2. Silent data corruption — no crash, model quietly degrades. Cross-replica checksums.
3. Data-iterator bug — wrong shard or an off-by-one. Days of compute on the wrong data.
4. Router collapse (MoE) — watch max_over_mean; above 3 is an emergency.
5. Slow checkpoints — cadence and restart speed are worth ~12 days of a 40-day run.