Glossary — Frontier Pre-Training

One line each. Terms are grouped by where they first matter, and cross-linked to the phase that builds them properly.

Compute and arithmetic

FLOP — one floating-point operation (a multiply or an add). "FLOPs" counts work; "FLOP/s" measures speed. Phase 00

FMA — fused multiply-add: one instruction, counted as two FLOPs.

C = 6ND — training compute for N parameters over D tokens: 2N forward + 4N backward, per token. The currency conversion of the entire field.

2N — inference FLOPs per token (forward only). Used for prefill and for the serving term of lifetime cost. Never mix it up with 6N.

Matmul — matrix multiply; 2·m·k·n FLOPs. ~99% of a transformer's arithmetic.

Systolic array — the TPU's fixed-size (e.g. 128×128) grid of multiply-accumulate cells. Dimensions that are not multiples of the tile get padded, and you pay for the padding.

Tensor cores — NVIDIA's equivalent matmul units.

Arithmetic intensity — FLOPs performed per byte moved. Compare against the ridge point to know whether you are compute- or memory-bound.

Ridge point — peak FLOP/s ÷ HBM bandwidth. H100 ≈ 296 FLOP/byte. Below it → memory-bound.

Roofline — the model that plots achievable throughput against arithmetic intensity, with a bandwidth-limited slope and a compute-limited ceiling.

Precision and memory

FP32 / BF16 / FP16 / FP8 / INT4 — number formats at 32/16/16/8/4 bits. BF16 beat FP16 for training because it keeps FP32's exponent range, and gradients span an enormous dynamic range.

HBM — High Bandwidth Memory, physically attached to the accelerator. The scarcest resource in serving. Two numbers matter: capacity (GB) and bandwidth (TB/s).

Mixed precision — bf16 for matmuls, fp32 for the optimizer's master weights and moments. Costs 16 bytes per parameter with Adam.

ZeRO / FSDP — sharding of optimizer states (stage 1), gradients (stage 2) and parameters (stage 3) across data-parallel replicas. Stage 1 is the cheapest big win because optimizer states are ~75% of training memory.

Activation checkpointing — discard intermediate activations and recompute them in the backward pass. Trades ~30% more FLOPs for a large memory saving, and is the reason MFU and HFU differ.

KV cache — the stored keys and values from earlier tokens, so generation does not recompute them. 2·L·n_kv·d_head·T·B·bytes. The serving wall.

Model shape

d_model — width of the residual stream. d_ff — the MLP's hidden width (~4× d_model, or ~2.7× for gated). n_heads / n_kv_heads / d_head — attention head counts and dimension. L — layers. V — vocabulary size.

MHA / GQA / MQA — Multi-Head, Grouped-Query, and Multi-Query attention: every query head gets its own KV head, a shared one per group, or a single shared one. The single biggest lever on KV-cache size, and it is frozen at pre-training time.

SwiGLU / gated MLP — an MLP with three matrices (up, gate, down) instead of two.

Non-embedding parameters — parameter count excluding the embedding and unembedding tables. The correct N for scaling-law work: a 256k vocabulary can be 89% of a small model.

Scaling laws

Scaling law — an empirical formula predicting test loss from resources, typically L = E + A/N^α + B/D^β. A property of your recipe, not of nature. Phase 01

Irreducible loss (E) — the entropy of the data itself; a floor no model beats. ~87% of the number at frontier scale.

Capacity term (A/N^α) — error from the model being too small. Data term (B/D^β) — error from not having seen enough.

Recipe — the full parameterized specification of a run: architecture scaling, data mixture, optimizer, schedule, numerics, parallelism. A law is meaningless without one fixed.

IsoFLOPs — fix C, sweep N, derive D = C/6N, fit a parabola in log N, take the vertex. Repeat across budgets, then fit N_opt ∝ C^a and D_opt ∝ C^b.

a + b ≈ 1 — a consistency check that falls directly out of C = 6ND. Free bug detector.

Kaplan (2020) — found N_opt ∝ C^0.73; concluded the industry should scale parameters over data. Chinchilla (2022) — found C^0.5 after fixing a measurement bias; concluded models were undertrained.

LR-decay / schedule-mismatch bias — reading loss part-way through a run whose learning-rate schedule targets a longer horizon. A uniform bias is absorbed into E; the non-uniform one tilts the fitted exponents. The mechanism behind Kaplan → Chinchilla.

Huber loss — quadratic near zero, linear in the tails. Used on log-space residuals so a single diverged run cannot dominate the fit.

Bootstrap — resample the ladder with replacement, refit, repeat, take percentiles. How you get a confidence interval when the model is nonlinear and the noise model is unspecified.

Crossover — the compute budget at which two fitted laws swap places. The deliverable of a recipe comparison; "candidate is better" silently assumes a scale.

Data-constrained law L(N, U, R) — splits D into unique tokens U and repeats R. A few epochs are nearly free; many are nearly worthless.

Inference-aware scaling — optimize 6N·D_train + 2N·D_inf rather than training cost alone. Complicated by the fact that D_inf is unforecastable (Jevons paradox, market expansion).

Jevons paradox — making a resource cheaper increases total consumption. Efficiency wins get eaten by more usage.

Mixture of Experts

MoE — replace each block's MLP with E parallel experts plus a router that activates top_k per token. Parameters scale with E; FLOPs scale with k. Phase 02

Router — the tiny d_model × E matrix that scores experts. The most fragile component in the model.

Top-k routing — select the k highest-scoring experts. k=1 is Switch; k=2 is the common default; k == E is a dense model in disguise.

Gate renormalization — rescale the chosen k gates to sum to 1. Without it, the layer's output is silently attenuated by a per-token amount.

Router collapse — the rich-get-richer loop in which one expert wins everything and the rest die. The default dynamic, not an exotic failure.

Load-balancing loss — L = E · Σ f_i·P_i, where f is the (discrete) token-slot fraction and P the (differentiable) mean router probability. Perfect balance gives exactly 1.0.

Router z-loss — a penalty on logsumexp(logits)². Prevents bf16 overflow and softmax saturation, which are two different failures.

Capacity factor — the safety margin on each expert's fixed buffer. Too low drops tokens; too high pads with zeros. No setting avoids both.

Token dropping — overflow past an expert's capacity. Silent: the token rides the residual and nothing is logged.

Shared expert — one that runs for every token unconditionally. Structurally eliminates fully dropped tokens and lets routed experts specialize harder.

Active vs total parameters — active goes into 6ND and serving cost; total goes into HBM, checkpoints and sharding. Off by 10–20× if swapped.

Expert parallelism (EP) — sharding experts across chips. Forces two all-to-all collectives per layer.

Parallelism and serving

DP / TP / PP / EP — data, tensor, pipeline and expert parallelism.

All-reduce / all-gather / reduce-scatter / all-to-all — the collective operations. All-to-all (every chip sends a different slice to every other) is the most expensive, and is what naive expert parallelism requires.

GSPMD — Google's compiler-driven sharding system: annotate tensors, let XLA infer the rest.

Prefill — process the entire prompt at once. Compute-bound.

Decode — generate one token at a time. Memory-bandwidth-bound, at every realistic batch size. This asymmetry governs all of serving.

Pipelined prefill — shard layers across chips rather than experts, and stream prompt chunks through, so transfers hide behind computation. The Flash 2.0 unlock, credited to Geng Yan.

Pipeline bubble — the idle time while a pipeline fills and drains. Utilization is M/(M + S − 1) for M chunks and S stages.

Prefill/decode disaggregation — running the two phases on separately-provisioned hardware, because they are bound by different resources.

Continuous batching / PagedAttention — serving techniques that raise decode throughput by amortizing weight reads across many concurrent requests.

Efficiency and operations

MFU (Model FLOPs Utilization) — useful model FLOPs achieved ÷ peak. 35–55% is normal at scale; it is an accounting identity, not a failure grade.

HFU (Hardware FLOPs Utilization) — the same, but counting recomputation from activation checkpointing as useful. Always ≥ MFU. Always ask which you are being shown.

Goodput — the fraction of wall-clock spent making forward progress. Driven mostly by checkpoint cadence and restart speed; 64% vs 94% is ~12 days of a 40-day run.

Loss spike — a sudden jump in training loss. May self-heal or may permanently damage the model; you have minutes to decide between skipping the batch, lowering LR, and rolling back.

Silent data corruption (SDC) — a chip computing wrong numbers without erroring. The worst failure mode, because only cross-replica checksums catch it.

Straggler — one slow node; at a synchronous all-reduce barrier, everyone waits for it.

PUE — Power Usage Effectiveness: total datacenter power ÷ IT power. 1.1–1.5 typically.

Compression

Distillation — training a student to match a teacher's full output distribution rather than a one-hot label. Best understood as variance reduction; a better teacher reduces bias. TRANSCRIPT-DISSECTED §Claim 3

Temperature — the softmax divisor T. Raising it flattens the distribution and amplifies the "dark knowledge" in small probabilities. The Hinton loss carries a T² factor to restore gradient magnitude.

Top-k logit store — keeping only the k largest teacher logits plus the renormalized tail mass. The only tractable way to store teacher outputs at trillion-token scale (full logits would be ~5 exabytes).

Quantization — representing weights (and sometimes activations) in fewer bits. An energy lever before a memory lever: a DRAM read costs ~20,000× an integer add, and power is ~99% of hardware TCO.

Affine / asymmetric quantization — map a group of values onto 2^b integer levels with a scale and zero-point. Max error is half a step, by construction.

Outliers — the handful of weights (and far worse, activation channels) that are ~100× larger than the rest, stretching the scale and destroying resolution for everything else. Every serious method (GPTQ, AWQ, SmoothQuant) is fundamentally an outlier-handling strategy.

Vector quantization — replacing groups of weights with a codebook index, exploiting correlations between them. Feinberg names this as an open frontier requiring little compute.

Research practice

Research as an MDP — planning under uncertainty over a stochastic dependency graph, as opposed to engineering's deterministic DAG where progress is monotone.

Research taste — well-calibrated priors on p(success) plus the habit of buying information before buying outcomes. Trainable, not mystical.

Value of information — the worth of an experiment that changes what you do next, over and above its direct payoff. Usually the dominant term.

Kill criteria — the stopping rule written before the experiment starts, because afterwards you will be attached to it.

Optimal experimental design — choosing where to place runs to minimize the variance of the extrapolation you actually care about. Spread beats density: ~26× at identical cost.

Regret — how much worse off you were than the best strategy in hindsight. Feinberg's framing for auditing whether sophisticated scaling work actually beat "pick a size and train on everything."

Tick-tock — Gemini's generational cadence: this generation's Flash should match last generation's Pro. A compression mandate with a deadline.