The Transcript, Dissected

What this document is. Vlad Feinberg's interview and his Princeton talk are dense with claims that are obvious to a frontier-lab engineer and opaque to everyone else. This document takes every substantive claim, in order, and does four things with each:

  1. Quote it (briefly) or state it plainly.
  2. Decode it for someone with zero background — every term defined from nothing.
  3. Go under the hood — the actual mechanism, with math and runnable code.
  4. Zoom out — why it matters, what it implies, what the common misreading is.

Nothing here assumes you know what a transformer is, what a FLOP is, or what "sharding" means. If you already do, the Under the hood subsections are where the value is.

Sources: the Developing Dev interview (video) and his Gemini Pretraining talk, Princeton, Apr 2025. Quotes are short and attributed; everything else is this track's own explanation.

Table of Contents

Part 0 — The Vocabulary You Need Before Anything Else

If you have no background, read this once. Nine words unlock the rest of the document.

FLOP — one floating-point operation: a single multiply or a single add on decimal numbers. "FLOPs" (plural) counts work; "FLOP/s" measures speed. An NVIDIA H100 does roughly 10¹⁵ FLOP/s (about a petaflop) on the number formats used for training. If a task needs 10²⁴ FLOPs and your machine does 10¹⁵ FLOP/s, it takes 10⁹ seconds ≈ 31 years — which is why you use 10,000 machines.

Parameter — one number inside the model that gets learned. N is the count. A "70B model" has 70 × 10⁹ parameters. At 2 bytes each (bf16) that is 140 GB just to hold the weights.

Token — a chunk of text (roughly ¾ of an English word). Models read and write tokens, not characters. D is the number of tokens the model is trained on. Modern runs use 10¹³–10¹⁴ tokens ("trillions and trillions").

Matmul (matrix multiply) — the operation that is ~99% of a transformer's arithmetic. Multiplying an (m × k) matrix by a (k × n) matrix costs 2·m·k·n FLOPs (each of the m·n outputs is a sum of k products, and each product-plus-add is 2 FLOPs).

HBM (High Bandwidth Memory) — the fast memory physically attached to an accelerator chip. An H100 has 80 GB; a TPU v5e has 16 GB. This is the scarcest resource in serving. Two numbers matter: capacity (GB) and bandwidth (TB/s — how fast you can read it).

Accelerator / chip — a GPU (NVIDIA H100, B200) or a TPU (Google's v5e, v6e). It contains a matmul unit (systolic array / tensor cores) that is enormously fast, a vector unit that is much slower, and HBM that is slower still relative to the matmul unit's appetite.

Pre-training — the first and most expensive stage: predict the next token over a huge corpus. This produces a "base model" that knows language and facts but is not yet a helpful assistant.

Post-training — everything after: supervised fine-tuning (SFT) on demonstrations, preference optimization (RLHF/DPO), reasoning training. Turns a base model into a product.

Inference / serving — running the trained model to answer requests. Split into prefill (process the user's whole prompt at once — compute-heavy) and decode (generate output one token at a time — memory-bandwidth-heavy). This split governs everything in Phases 05–06.

Scaling law — an empirical formula predicting model quality (test loss) from compute, parameters, and data. The tool that makes a $50M training run a calculated bet instead of a prayer.

Tip. Write these nine on a card. Every claim below is a combination of them.

Part 1 — "Pre-Training" Is Not "Training"

Before the claims, the shape of the job, because the interview assumes it.

A normal ML job looks like: get data → train → check validation → adjust → retrain → ship. The loop runs dozens of times a week.

A frontier pre-training job looks like this:

        ┌──────────────────────── months ────────────────────────┐
 recipe design → scaling-law ladder → forecast → GO/NO-GO → the run → compress → serve
   (weeks)         (many small runs)   (a number)  (one meeting)  (40 days)   (weeks)  (forever)
                                                                     ▲
                                                        this happens exactly ONCE

Everything upstream of "the run" exists to make one irreversible decision defensible. That is why scaling laws are the intellectual centre of the job, why forecasting is a research area, and why the SRE hat exists. Hold that picture; every claim below is a piece of it.

Claim 1 — Kernel and low-level work is in "voracious demand"

He says there is "voracious demand... across all the different labs" for "kernel development and low-level engineering to improve the runtime for these LLMs," because when you change the architecture "you just need to be able to implement these new techniques in efficient ways."

Decode it

A kernel is a small program that runs on the accelerator and does one job extremely fast — "multiply these two matrices," "apply softmax to this row," "do attention for this block." Your Python code (torch.nn.Linear) is a thin wrapper; the actual work happens in a kernel written in CUDA (NVIDIA), Triton, Pallas (TPU), or an assembly-adjacent language.

Why the demand exists. A researcher proposes a new attention variant on a whiteboard. To test it, someone must write a kernel for it. If no kernel exists, the idea runs 10× slower than the baseline in naive PyTorch, loses the comparison, and gets discarded — even if it was better. Missing kernels silently kill good research. That is the demand.

Under the hood — why naive code is slow

The classic example is attention. Naive attention materializes an n × n score matrix:

# Naive attention. n = sequence length, d = head dimension.
# The killer: `scores` is n×n and must be WRITTEN to HBM, then READ back. Twice.
scores  = Q @ K.T / math.sqrt(d)          # (n, n)  <- n² floats hit memory
weights = softmax(scores, axis=-1)        # (n, n)  <- read n², write n²
out     = weights @ V                     # (n, d)  <- read n² again

At n = 8192, n² = 67M floats = 134 MB in bf16 — per head, per layer. You blow past cache and pay HBM bandwidth three times.

FlashAttention never materializes it. It walks over blocks of keys/values, keeping a running softmax in fast on-chip memory using the online-softmax trick:

# Online softmax: fold a new block into a running (max, sum, weighted-output) triple.
# This is the whole idea behind FlashAttention, in 8 lines of pure Python.
def online_softmax_update(m_prev, l_prev, o_prev, new_scores, new_values):
    m_new = max(m_prev, max(new_scores))            # running max, for stability
    rescale = math.exp(m_prev - m_new)              # how much to shrink the old state
    exp_new = [math.exp(s - m_new) for s in new_scores]
    l_new = l_prev * rescale + sum(exp_new)         # running denominator
    o_new = [o * rescale for o in o_prev]           # rescale old numerator
    for e, v in zip(exp_new, new_values):           # add the new block's contribution
        o_new = [a + e * b for a, b in zip(o_new, v)]
    return m_new, l_new, o_new
# Final output = o_new / l_new. Never stores an n×n matrix.

Same mathematical result. Memory traffic drops from O(n²) to O(n). Speedups of 2–4× on long sequences, and it is why 100k+ context is affordable at all.

Takeaway. "Kernel work" is not plumbing beneath the science. It is the science: the set of architectures you can honestly evaluate is exactly the set someone has written a fast kernel for. Phase 11 builds a tiny tile-DSL so this stops being magic.

Claim 2 — Research is a stochastic DAG; engineering is a deterministic one

He frames "research as an MDP" (crediting Jacob Steinhardt's essay). Engineering: "the DAG is more or less deterministic... you can just make monotone progress." Research: "some of the nodes which might be research ideas... may or may not work out." The core skill is "building an intuition of how likely an approach is to work out without having yet done that approach" — research taste.

Decode it

A DAG (directed acyclic graph) is a to-do list with arrows: "must do A before B." Building a web service is a DAG — write the database layer, then the API, then the frontend. Each box you finish stays finished. Progress is monotone: it only goes up.

An MDP (Markov Decision Process) is the mathematical model for "make decisions when outcomes are random." It has states (what you know now), actions (which experiment to run), transition probabilities (this experiment works with probability p), rewards, and costs. The famous fact about MDPs: the greedy choice — highest immediate reward — is often wrong, because a cheap experiment that teaches you which branch to take can be worth more than an expensive one that might directly succeed.

Feinberg's point is that research planning is the MDP, not the DAG. You must "factor in the success rate and the time investment... as well as a priori estimating what those different rates are."

Under the hood — make "taste" a computation

Taste feels mystical. It is not: it is a prior over p(success) plus a habit of computing expected value per unit cost. Here it is in code you can actually run:

# A research portfolio, scored the way a pre-training lead scores it.
# value = what you gain if it works (e.g. % quality improvement on the flagship)
# p     = your prior that it works
# cost  = engineer-weeks (or TPU-days) to find out
experiments = [
    # name,                        value,  p,     cost
    ("new attention variant",      100,    0.15,  8),
    ("data mixture reweighting",    30,    0.70,  2),
    ("bigger LR + warmup tweak",    15,    0.50,  1),
    ("switch dense -> MoE",        250,    0.40, 20),
]

for name, value, p, cost in sorted(
        experiments, key=lambda e: -(e[1] * e[2] / e[3])):
    print(f"{name:30s} EV/cost = {value * p / cost:6.2f}")

data mixture reweighting        EV/cost =  10.50
bigger LR + warmup tweak        EV/cost =   7.50
switch dense -> MoE             EV/cost =   5.00
new attention variant           EV/cost =   1.88

Now the three refinements that separate a senior researcher from this toy:

  1. Value of information. An experiment whose result changes what you do next is worth more than its direct payoff. Running a cheap 400M-parameter MoE ablation has low direct value but tells you whether to spend the 20 weeks. Buy information before you buy outcomes.
  2. Kill criteria set in advance. "If the 1B ablation is not within 0.01 nats of baseline by 20B tokens, we stop." Written before you start, because after you start you will be attached to it. This is the single highest-leverage habit in research.
  3. Correlated failures. Five variants of one idea are not five independent bets. If the underlying premise is wrong, all five die together. Diversify across premises, not across implementations.

Takeaway. When someone says a researcher has "great taste," they mean the person's internal p values are well-calibrated and they instinctively buy information first. Both are trainable. Phase 12 makes you build the planner.

Claim 3 — Vertical one: distillation

Transferring "knowledge or some form of statistics about the underlying dataset through a teacher model into the student model." At scale this means running the teacher over "trillions and trillions of tokens," costing "millions and millions of dollars," so every operation matters "because every operation... is multiplied by such a large factor."

Decode it

You have a huge, excellent, expensive model (the teacher — say Gemini Pro). You want a small, fast, cheap model (the student — Flash) that behaves as much like it as possible.

The naive approach: train the student on the same text the teacher saw. The distillation approach: train the student to match the teacher's probability distribution over the next token.

Why that is dramatically better — this is the key insight and it is easy to miss:

Suppose the context is "The capital of France is". The ground-truth label is a single token: " Paris". That is one bit of supervision — "this one, not the other 255,999."

The teacher instead says:

" Paris"      0.92
" Lyon"       0.03
" Marseille"  0.02
" located"    0.01
" the"        0.005
...           (a full distribution over the whole vocabulary)

That is a rich, dense signal. It doesn't just say the answer — it says which wrong answers are reasonable. It encodes that Lyon is a French city while "banana" is not. Each token now carries hundreds of bits of supervision instead of one. This is the "some form of statistics about the underlying dataset" phrase: the teacher's distribution is a compressed summary of everything it learned from the corpus, delivered per token.

Under the hood — the actual loss

import math

def kl_distillation_loss(student_logits, teacher_logits, temperature=2.0):
    """KL(teacher_T || student_T) * T^2 — the standard Hinton distillation loss.

    Temperature T > 1 flattens both distributions, which amplifies the information
    carried by the small probabilities (the 'dark knowledge'). The T^2 factor
    restores the gradient magnitude, which otherwise scales like 1/T^2.
    """
    def softmax_t(logits, T):
        m = max(logits)                                   # max-subtraction: never overflow
        exps = [math.exp((z - m) / T) for z in logits]
        s = sum(exps)
        return [e / s for e in exps]

    p_teacher = softmax_t(teacher_logits, temperature)
    p_student = softmax_t(student_logits, temperature)

    kl = sum(pt * math.log(pt / ps)
             for pt, ps in zip(p_teacher, p_student) if pt > 1e-12)
    return kl * temperature ** 2


# Toy vocabulary of 4 tokens. The teacher is confident but not absolute.
teacher = [4.0, 1.0, 0.5, -1.0]
good_student = [3.6, 1.2, 0.4, -0.8]   # similar shape  -> low loss
bad_student  = [4.0, -5.0, -5.0, -5.0] # right argmax, wrong shape -> higher loss

print(round(kl_distillation_loss(good_student, teacher), 4))  # 0.031
print(round(kl_distillation_loss(bad_student,  teacher), 4))  # 2.1126

Note what that demonstrates: bad_student gets the answer right (same argmax) and still takes 68× the loss, because it has thrown away the shape. Distillation grades the shape, not the answer.

The engineering problem nobody warns you about

Do the arithmetic on "trillions of tokens":

  • 10 trillion tokens × a vocabulary of 256,000 × 2 bytes per logit
  • = 5 × 10¹⁸ bytes = 5 exabytes.

You cannot store that. Nobody can. So the real job is choosing an approximation:

StrategyStorage for 10T tokensCostFidelity
Full logits~5 EBimpossibleperfect
Top-k logits (k=64) + renormalized tail mass~10 TBcheapvery good
Online distillation (teacher runs alongside student)0teacher FLOPs every step, foreverperfect
Sampled sequences (teacher generates text, student trains on it)~20 TB textone-timeweakest

This is precisely the "infrastructure investments in storage systems and multi-datacenter operations" he refers to. The teacher may run in one datacenter while the student trains in another, so those top-k logits cross continents. The research idea is three lines; the system around it is a year of work. Phase 07 builds the top-k store and measures the fidelity loss.

Takeaway. Distillation is where "research" and "storage engineering" become the same job. If you want a differentiating skill, own that seam.

Claim 4 — Vertical two: inference co-design

Building "neural architectures that are efficient to run inference on" by choosing network topology, "shapes of the matrices," and "attention shapes, num heads" that "fully utilize the hardware."

Decode it

Two models with identical parameter counts can differ by 3× in serving speed, purely because of shape choices. Inference co-design is picking shapes with the chip in mind, before you train — because after you train, the shape is frozen forever.

Under the hood — three concrete shape decisions

Decision A: matmul dimensions should be multiples of the hardware tile. A TPU's matmul unit is a systolic array (e.g. 128×128). A GPU's tensor cores want multiples of 8/16/64. If your d_ff is 11,000, the hardware pads to 11,008 or worse, and you pay for arithmetic you throw away.

def tile_efficiency(dim, tile=128):
    """Fraction of the padded matmul that is real work."""
    padded = math.ceil(dim / tile) * tile
    return dim / padded

for d in (11000, 11008, 8192, 4096, 4097):
    print(f"d_ff={d:6d}  padded={math.ceil(d/128)*128:6d}  eff={tile_efficiency(d):.1%}")

d_ff= 11000  padded= 11008  eff=99.9%
d_ff= 11008  padded= 11008  eff=100.0%
d_ff=  8192  padded=  8192  eff=100.0%
d_ff=  4096  padded=  4096  eff=100.0%
d_ff=  4097  padded=  4224  eff=97.0%

Small here — but the same logic applied to head dimension, expert count, and shard boundaries compounds multiplicatively across dozens of layers, and the pathological cases are much worse than 97%.

Decision B: KV heads. During decode, you re-read the KV cache for every generated token. The cache size is:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × bytes_per_elem

Multi-Head Attention (MHA) gives every query head its own KV head. Grouped-Query Attention (GQA) shares one KV head across a group of query heads; Multi-Query Attention (MQA) uses exactly one. Quality barely moves; the cache shrinks by the group factor.

def kv_cache_gb(layers, kv_heads, head_dim, seq, batch, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * seq * batch * bytes_per / 1e9

# A 70B-class model, 8k context, batch of 32:
print(round(kv_cache_gb(80, 64, 128, 8192, 32), 1))  # MHA  : 687.2 GB -> nine H100s of cache
print(round(kv_cache_gb(80,  8, 128, 8192, 32), 1))  # GQA-8:  85.9 GB -> just over one H100
print(round(kv_cache_gb(80,  1, 128, 8192, 32), 1))  # MQA  :  10.7 GB -> trivially fits

That is one architecture decision moving memory by 64×. It has to be made before training starts. It is the single clearest example of what "inference co-design" means.

Decision C: depth vs width at fixed N. Deeper models are often slightly better per parameter, but depth is serial — layer k+1 cannot start until layer k finishes — so depth directly costs decode latency and creates more pipeline stages to sync. Width is parallel and matmul-friendly. Co-design usually means: as wide as quality allows, as shallow as quality tolerates.

Takeaway. Ask of every architecture choice: "what does this do to bytes-read-per-token at decode?" That single question is 80% of inference co-design.

Claim 5 — Vertical three: quantization, and the 99%-power fact

Reducing "the size that the neural nets take up in order to represent their weights" from FP32 down to four bits — which he calls "kind of a miracle." And the reason it matters: "99% of the total cost of operation for AI hardware comes from the power that it takes to run these chips."

Decode it

A number in a computer is stored in bits. FP32 = 32 bits per parameter. BF16 = 16. FP8 = 8. INT4 = 4. Quantization means storing the same model with fewer bits per number, accepting a small rounding error.

A 70B model: 280 GB in FP32, 140 GB in BF16, 70 GB in FP8, 35 GB in INT4. The last one fits in a single H100. The first needs four.

Under the hood — how you actually do it

The simplest scheme, affine (asymmetric) quantization, per group of weights:

def quantize_affine(values, n_bits=4):
    """Map a group of floats onto 2^n_bits integer levels. Returns (codes, scale, zero)."""
    qmin, qmax = 0, 2 ** n_bits - 1
    lo, hi = min(values), max(values)
    if hi == lo:                                  # degenerate group: everything identical
        return [0] * len(values), 1.0, lo
    scale = (hi - lo) / (qmax - qmin)             # how much real value one code step is worth
    zero = lo                                     # what code 0 means
    codes = [min(qmax, max(qmin, round((v - zero) / scale))) for v in values]
    return codes, scale, zero

def dequantize_affine(codes, scale, zero):
    return [c * scale + zero for c in codes]

w = [0.12, -0.45, 0.88, -0.03, 0.51, -0.77, 0.20, 0.05]
codes, scale, zero = quantize_affine(w, n_bits=4)
recon = dequantize_affine(codes, scale, zero)
err = max(abs(a - b) for a, b in zip(w, recon))
print(codes)              # [8, 3, 15, 7, 12, 0, 9, 7]
print(round(err, 4))      # 0.05  -> max error is half a step, as it must be

Note the error bound: with 16 levels spanning a range of 1.65, one step is 0.11, and rounding can never be off by more than half a step (0.055). That is the guarantee you test for — max_error <= scale/2 is an invariant, not an empirical observation.

Why it works at all (the "miracle"): neural network weights are massively redundant and roughly bell-shaped. The network was trained with noise (dropout, stochastic gradients, bf16 rounding) so it is already robust to small perturbations. What matters is not the precision of any single weight but the statistical structure of the whole layer.

Why it stops working: outliers. A handful of weights (and far more importantly, a handful of activation channels) are 100× larger than the rest. In a per-group scheme, one outlier stretches scale and destroys the resolution of the other 63 values. Every serious method — GPTQ, AWQ, SmoothQuant — is fundamentally an outlier-handling strategy.

The 99% fact, unpacked

This claim surprises people, so let us be careful about what it means. It is a claim about the total cost of operation — the ongoing cost of running a fleet, amortizing chips over years — and it lumps together electricity for the chips, electricity for cooling, and the power-delivery infrastructure sized to that draw. Under that accounting, the marginal cost of serving is energy, not silicon.

Now, why quantization is such an outsized lever on energy — this is a hardware fact worth memorizing. Moving a number costs vastly more energy than computing with it. Approximate energies at 45nm (Horowitz, ISSCC 2014 — the canonical reference; ratios still hold):

OperationEnergyRelative
8-bit integer add0.03 pJ1×
32-bit float add0.9 pJ30×
32-bit float multiply3.7 pJ123×
Read 32 bits from on-chip SRAM5 pJ167×
Read 32 bits from off-chip DRAM640 pJ~21,000×

Read that last row again. A DRAM read costs ~20,000× an integer add. So when you halve the bits, you halve the dominant term. Going FP16 → INT4 cuts weight-movement energy ~4×, and weight movement is most of decode.

# Rough marginal-energy model for decoding one token from an N-parameter model.
# Decode is memory-bound: the cost is dominated by reading every weight once.
PJ_PER_BIT_DRAM = 640 / 32                      # ≈ 20 pJ per bit moved

def decode_energy_joules(n_params, bits_per_param):
    return n_params * bits_per_param * PJ_PER_BIT_DRAM * 1e-12

for bits, name in [(32, "FP32"), (16, "BF16"), (8, "FP8"), (4, "INT4")]:
    j = decode_energy_joules(70e9, bits)
    # 1e9 tokens/day at $0.12/kWh
    daily = j * 1e9 / 3.6e6 * 0.12
    print(f"{name}: {j*1000:8.1f} mJ/token   ${daily:9,.0f}/day at 1B tokens/day")

FP32:  44800.0 mJ/token   $    1,493/day at 1B tokens/day
BF16:  22400.0 mJ/token   $      747/day at 1B tokens/day
FP8 :  11200.0 mJ/token   $      373/day at 1B tokens/day
INT4:   5600.0 mJ/token   $      187/day at 1B tokens/day

The absolute numbers are a deliberate over-estimate in one direction (real systems reuse weights across a whole batch, which is exactly why batching exists) and an under-estimate in another (this counts only DRAM traffic, not cooling, power delivery, or the rest of the chip). The ratios are the point: FP32→INT4 is an 8× reduction in the dominant cost term. Scale that to Google's actual token volumes and it is a nine-figure line item.

He also notes the compounding: reducing activation precision on top of weights "multiplies" the gain — because now you shrink the other thing crossing the memory bus. It is also much harder, because activations have far worse outliers than weights.

Takeaway. Quantization is not a compression trick. It is the primary energy lever in AI, and energy is the bill. Phase 08 builds the quantizer and the power model.

Claim 6 — MFU, and why 100% is not the goal

He explicitly addresses Twitter confusion about "low" MFU numbers. To hit 100% you would need to be "doing a bunch of matmuls in a loop without reading any memory," which is not a neural network, because real nets "have to apply activation functions or do attention or write intermediate outputs."

Decode it

MFU = Model FLOPs Utilization.

MFU = (useful model FLOPs performed per second) / (chip's peak FLOP/s)

If your run does 3 × 10¹⁴ useful FLOP/s on hardware rated at 10¹⁵, your MFU is 30%. Published large-scale numbers commonly land in the 30–55% band. People see "35%" and conclude someone is incompetent. They are wrong, and here is exactly why.

Under the hood — where the other 65% goes

A chip is not one unit. It is several, with wildly different throughput:

┌──────────────────────────────────────────────┐
│  ACCELERATOR                                 │
│                                              │
│  ┌───────────────┐   very fast (the peak)    │
│  │ MATMUL UNIT   │   ~1000 TFLOP/s           │
│  └───────────────┘                           │
│  ┌───────────────┐   ~50-100× slower         │
│  │ VECTOR UNIT   │   (gelu, softmax, norms)  │
│  └───────────────┘                           │
│  ┌───────────────┐   ~3 TB/s                 │
│  │ HBM           │   (weights, activations)  │
│  └───────────────┘                           │
│  ┌───────────────┐   ~0.05-0.9 TB/s          │
│  │ INTERCONNECT  │   (chip-to-chip)          │
│  └───────────────┘                           │
└──────────────────────────────────────────────┘

"Peak FLOP/s" is the matmul unit's number alone. But your model must also:

  • apply GELU/SwiGLU — vector unit, no matmul FLOPs credited
  • compute softmax in attention — vector unit, plus exponentials
  • compute layer norms / RMS norms — vector unit, plus a reduction (which serializes)
  • write and read intermediate activations — HBM traffic, no FLOPs credited
  • all-reduce gradients — interconnect, no FLOPs credited
  • pay the optimizer step (Adam: several element-wise passes over all parameters)

Every one of those is time during which the matmul unit is idle. So:

def mfu_budget(matmul_s, vector_s, hbm_s, comms_s, optimizer_s):
    total = matmul_s + vector_s + hbm_s + comms_s + optimizer_s
    return {
        "MFU (matmul busy fraction)": matmul_s / total,
        "lost to vector ops":         vector_s / total,
        "lost to memory traffic":     hbm_s / total,
        "lost to collectives":        comms_s / total,
        "lost to optimizer":          optimizer_s / total,
    }

for k, v in mfu_budget(matmul_s=100, vector_s=45, hbm_s=60,
                       comms_s=50, optimizer_s=25).items():
    print(f"{k:32s} {v:6.1%}")

MFU (matmul busy fraction)        35.7%
lost to vector ops                16.1%
lost to memory traffic            21.4%
lost to collectives               17.9%
lost to optimizer                  8.9%

35.7% MFU is not a failure — it is an accounting identity. And notice what the breakdown gives you: an agenda. Comms at 18% says overlap them. Memory at 21% says fuse kernels. Vector at 16% says fuse the norm into the matmul epilogue.

His deeper point connects straight back to co-design: different shapes stress different units. The job is "choosing shapes for this neural net that fully saturate all of those hardware units" — not maximizing one number.

Two traps. (1) MFU is comparable only within a hardware/precision/model class — MFU on FP8 with sparsity is a different denominator. (2) Some teams quote HFU (Hardware FLOPs Utilization) which counts recomputation from activation checkpointing as useful work. HFU is always ≥ MFU. Always ask which one you are being shown.

Claim 7 — Pre-training is a one-shot extrapolation problem

His slides put it starkly. Before: "Maybe 2 stages; toy problem for iteration (CIFAR10) then you apply to Imagenet. LR searches by doing multiple 'final runs'. The last data point is our test set!" Now: "every single time you go up for a pre-training run, you're about to put in more FLOPs into this run than you've ever done before." So "every next run requires extrapolation."

Decode it

In classical ML, "the last data point is our test set" is a joke about a real practice: you try 20 learning rates, and the best one on the held-out set is your answer. You interpolate within a region you have already explored.

Pre-training breaks this in a way that is genuinely new:

  • Each flagship run costs $10M–$100M+ and takes 1–3 months.
  • You get one shot per generation.
  • The run is larger than anything you or anyone has run before — by construction, since the whole point is to push the frontier.

So you are not interpolating. You are extrapolating beyond every data point you own. That is a fundamentally different statistical problem, and it is why "scaling laws" is a research area rather than a spreadsheet.

His slides add the crucial qualifier, which most summaries drop:

"Analysis made in the context of a parameterized LLM training recipe! Must already have architecture scaling, schedule defined for N, D. Loss forecast implies model/recipe selection capability!"

Unpacked: a scaling law is not a law of nature. It is a property of your recipe. Before you can fit one, you must have already decided how every hyperparameter scales with N and D — how depth grows with width, how LR decays with batch size, how warmup scales. The law then describes that family. Change the family and you must refit.

And the last sentence is the punchline of the whole field: if you can forecast loss, you can select recipes. Forecasting is not a reporting tool. It is the decision procedure.

Under the hood — the ladder, in code

# You cannot run the flagship twice. So you run a LADDER of small models,
# fit a curve, and extrapolate. Here is the shape of that in miniature.

def run_ladder(budgets):
    """Pretend-train at several compute budgets, recording (C, best_loss)."""
    results = []
    for C in budgets:
        # In reality: sweep N at fixed C, take the minimum. See Phase 01.
        loss = 1.69 + 406.4 / (compute_optimal_N(C) ** 0.34) \
                    + 410.7 / (compute_optimal_D(C) ** 0.28)   # Chinchilla-form
        results.append((C, loss))
    return results

def compute_optimal_N(C):  return 0.6 * (C ** 0.5)   # Chinchilla-ish: N ∝ C^0.5
def compute_optimal_D(C):  return C / (6 * compute_optimal_N(C))

ladder = run_ladder([1e18, 1e19, 1e20, 1e21])         # cheap: hours, not months
flagship = 1e24                                        # the real run: months, $$$
print(run_ladder([flagship]))                          # THE FORECAST

The whole game: make the extrapolation from 1e21 to 1e24 — three orders of magnitude — trustworthy. Phase 01 makes you do it for real, with the fit, the error bars, and the failure modes.

Tip for interviews. If asked "how would you decide between two pre-training recipes?", the senior answer is never "train both and compare." It is: "fit a scaling law for each over a ladder of small runs, compare the fitted curves at the target FLOP count, and report the crossover point and the confidence interval." That is literally what his slides show: "To make a change, compare baseline vs candidate laws."

Claim 8 — C = 6ND, derived

From his slides: "for a transformer C = 6 * N * D is a very good approximation of FLOPs." The footnote: "Excluding self-attention, an N-parameter decoder-only model requires 6N matmul FLOPs per token seen (2N for forward and 4N for backward), because each matmul performs one multiplication and one addition per pair of input values, and the backward pass includes two matmuls for each one in the forward pass."

This is the most important equation in the field. Derive it once and you own it forever.

Step 1 — a single matmul costs 2 × (number of weights) FLOPs per token

Take a linear layer mapping a d_in-dimensional vector to d_out. Its weight matrix has d_in × d_out entries. For one input vector:

output[j] = Σ_i  input[i] * W[i][j]        for each of d_out outputs

Each of the d_out × d_in terms is one multiply and one add = 2 FLOPs. Total: 2 × d_in × d_out = 2 × (#weights). Hence, over the whole model: 2N FLOPs per token, forward.

Step 2 — backward costs twice forward

Backpropagation through the same linear layer needs two matmuls, not one:

forward :  Y = X · W                         (1 matmul)
backward:  dX = dY · Wᵀ    <- to pass gradient to the previous layer
           dW = Xᵀ · dY    <- to update this layer's weights
                                             (2 matmuls)

Both are the same size as the forward matmul. So backward = 2 × 2N = 4N FLOPs per token.

Step 3 — add them

6N FLOPs per token  ×  D tokens  =  C = 6ND

def training_flops(n_params, n_tokens):
    return 6 * n_params * n_tokens

def days_on_cluster(flops, n_chips, peak_flops_per_chip, mfu=0.4):
    return flops / (n_chips * peak_flops_per_chip * mfu) / 86400

# Llama-3-70B-scale: 70B params, 15T tokens
C = training_flops(70e9, 15e12)
print(f"{C:.2e} FLOPs")                                        # 6.30e+24 FLOPs
print(f"{days_on_cluster(C, 16000, 1e15):.1f} days on 16k H100")  # 11.4 days
print(f"{days_on_cluster(C, 1000, 1e15):.1f} days on 1k H100")    # 182.3 days

Now you can answer his opening question — "if I give you 1000 H100 for 30 days, what is the best LLM you can train?" — because you can convert chips × days into C, and Phase 01 turns C into (N, D).

Where 6ND breaks — know these four

  1. Attention is excluded. The QKᵀ and attn·V matmuls cost roughly 12 · L · n_ctx · d_model FLOPs per token, which does not scale with N. At short context this is a few percent; at 128k context it dominates. His slides show the exact per-step count: 18BTDF + 24BTDNH = 6·BT·(3DF + 4DNH), where B=batch, T=sequence, D=d_model, F=d_ff, N=num heads, H=head dim — the first term is the MLP, the second is attention projections.
  2. MoE. N must be the active parameter count (what each token actually routes through), not the total. A 400B-total / 40B-active MoE costs like a 40B dense model to train, and like a 400B model to store. This is exactly the audience question on his slide: "What about MoEs?"
  3. Embeddings. The input embedding is a lookup (~free); the output unembedding is a real matmul (2 · d_model · vocab per token). At small N with a 256k vocab this is a large fraction — a common source of wrong small-model FLOP counts.
  4. Activation checkpointing adds a partial extra forward pass, pushing you toward ~8ND of hardware FLOPs while the model FLOPs stay 6ND. (This is the MFU/HFU gap from Claim 6.)

Takeaway. C = 6ND is the currency conversion of the entire field: it turns money and time into model size and data. Memorize the derivation, not the formula.

Claim 9 — Kaplan said scale parameters; Chinchilla said scale both

His slides walk through this in detail, and it is the single most instructive story in scaling research — because it is a story about a methodological bug producing a wrong industry-wide strategy for two years.

Kaplan et al., 2020

Found that loss follows clean power laws in N, D, and C. Their compute-optimal allocation, in his slides' words: "With a 10x compute budget, parameters should increase by 5.37x and the amount of data by 1.86x." Their own line: data requirements grow "very slowly as D ∼ C^0.27."

Industry consequence (his slide states it directly): "We should heavily invest in scaling the model size rather than the data size!" This is why 2020–2022 was the era of ever-bigger, relatively under-fed models — GPT-3 at 175B trained on ~300B tokens.

His slide also lists the caveat, which everyone ignored at the time:

  • "These 'laws' are only empirical"
  • "The fitting of these laws depends a lot on the experimental setup as well as the implicit assumptions being made there."

Chinchilla (Hoffmann et al., GDM, March 2022)

His slide names the bug precisely:

"Kaplan et al. run a single training run per model size and uses intermediate losses to estimate the loss at different token horizon. ... This is a bad approximation as you can get much better losses through proper learning rate decay. Only the final loss value is optimal."

Here is why that is fatal, and it is worth being very concrete because it is subtle.

Learning-rate schedules decay to near zero at the end of training. That final decay phase is where a big chunk of the loss improvement happens — the model stops bouncing around the minimum and settles into it. So:

  • A model mid-run at 100B tokens (LR still high, still bouncing) has a much worse loss
  • than a model whose entire schedule was designed to end at 100B tokens (LR fully decayed).

Kaplan used the first as a proxy for the second. That systematically overstates how bad it is to train on more data — every data point in his "more tokens" direction was unfairly penalized. Correct for it, and the optimal shifts toward more tokens.

# Why reading loss mid-run is a biased estimator of "loss if I had stopped here".
def loss_at(tokens, horizon):
    """Toy: base curve + a penalty for not having decayed the LR yet."""
    base = 3.0 / (tokens ** 0.1)
    frac_done = tokens / horizon
    lr_penalty = 0.15 * (1 - frac_done)     # high LR = still noisy = worse loss
    return base + lr_penalty

# Kaplan-style: peek at the 300B-token run when it has seen 100B tokens
print(round(loss_at(100e9, horizon=300e9), 4))   # 0.3383  <- biased HIGH
# Chinchilla-style: a run actually designed to end at 100B tokens
print(round(loss_at(100e9, horizon=100e9), 4))   # 0.2383  <- the truth
# The gap (0.1 nats) is enormous at this scale, and it is pure methodology.

And note the direction of the bias, which is what makes it fatal rather than merely noisy: the penalty is proportional to (1 - frac_done), so it is largest exactly for the points with the most tokens relative to their horizon. The error is not random — it systematically tilts the fitted curve against training on more data.

The IsoFLOPs method — his slides' six steps

Chinchilla's cleanest approach, exactly as his slides enumerate:

1. Fix a target FLOPs budget                     ── e.g. C = 1e20
2. Train a few models, vary model size           ── N = 100M, 300M, 1B, 3B (D = C/6N each)
3. Fit a parabola and find the minimum           ── loss vs log(N) is U-shaped; take the vertex
4. Repeat 1–3 for various FLOPs budgets          ── C = 1e19, 1e20, 1e21, 1e22
5. Fit a power law: FLOPs budget → optimal N     ── N_opt ∝ C^a
6. Fit a power law: FLOPs budget → optimal D     ── D_opt ∝ C^b

Why a parabola? Because at fixed C, there is a genuine trade-off with a single interior minimum:

  • Too small N: you have tons of data but not enough capacity to absorb it → underfit.
  • Too large N: enormous capacity but you starve it of data → also bad.
  • In between: the sweet spot. Plotting loss against log N gives a clean U.
# The IsoFLOPs inner loop, in full.
def isoflop_curve(C, sizes):
    pts = []
    for N in sizes:
        D = C / (6 * N)                               # the budget constraint
        L = 1.69 + 406.4 / N**0.34 + 410.7 / D**0.28  # Chinchilla parametric form
        pts.append((N, D, L))
    return pts

for N, D, L in isoflop_curve(1e21, [1e8, 3e8, 1e9, 3e9, 1e10, 3e10]):
    print(f"N={N:8.1e}  D={D:8.1e}  loss={L:.4f}")

N= 1.0e+08  D= 1.7e+12  loss=2.6198   <- too small: underfit
N= 3.0e+08  D= 5.6e+11  loss=2.4344
N= 1.0e+09  D= 1.7e+11  loss=2.3400
N= 3.0e+09  D= 5.6e+10  loss=2.3363   <- the minimum
N= 1.0e+10  D= 1.7e+10  loss=2.4160
N= 3.0e+10  D= 5.6e+09  loss=2.5687   <- too big: data-starved

Notice how flat the bottom of that U is: 1e9 and 3e9 differ by only 0.004 nats. That flatness is a gift and a trap. A gift, because you can move off the exact optimum for serving reasons (Phase 02) at almost no quality cost. A trap, because with noisy measurements the fitted minimum can wander by a factor of 3 — which is precisely why the parabola fit and its confidence interval matter more than the single best point.

The result and its consequence

His slide: "the exponent in the power law is ~0.5, meaning model and data size should be scaled at the same rate! This is widely different from Kaplan et al." And he labels the old regime on the plot with one word: UNDERTRAINED!

"Consequences: Given a compute budget, models should be smaller and trained for longer. Kaplan's scaling laws meant that models were undertrained — which is obviously bad given bigger models are more expensive to serve and use downstream!"

That final clause is the bridge to his entire research agenda. Chinchilla didn't just improve loss-per-FLOP — it made models smaller at the same quality, which makes them cheaper to serve, which is the thing his team optimizes for.

Concretely: Chinchilla (70B, 1.4T tokens) beat Gopher (280B, 300B tokens) at the same training compute — with a model 4× smaller to serve.

Takeaway. The most consequential result in scaling laws came from fixing an experimental-design flaw, not from a new idea. Feinberg's own listed research direction — "Least squares vs MLE & formal stats model imply different scaling recommendations! Formalize." — says the field still has this class of bug in it. Methodology is the frontier.

Claim 10 — Scaling laws are brittle and dataset-dependent

From his closing slides: "Scaling laws are brittle, dataset dependent." And: "L(N, D, etc.) — of course we can add more dims to improve fit. Least squares vs MLE & formal stats model imply different scaling recommendations! Formalize." Plus: "Rather than grid (N, D) where do we get max info gain? Active learn…"

Decode it

Three distinct criticisms hiding in there. Take them one at a time.

(a) The fit depends on your loss function. You have ~30 noisy (N, D, L) points and you want parameters (A, B, E, α, β) for L = E + A/N^α + B/D^β. How you measure "fit" changes the answer:

MethodWhat it minimizesBias
Least squares on LΣ (L_pred − L_obs)²dominated by large-loss (small-model) points
Least squares on log Lrelative errortreats all scales equally
Huber loss on log Lrelative error, outlier-robustwhat Chinchilla actually used
MLE with an explicit noise modellikelihood under stated assumptionsrequires you to state the noise model

These give materially different exponents on the same data — and therefore different recommendations for the flagship run. Feinberg's "Formalize" is a call to stop hand-waving: write down the statistical model, then the estimator follows.

(b) The design points are chosen badly. Everyone runs a grid: N ∈ {100M, 300M, 1B, 3B} × C ∈ {1e19, 1e20, 1e21}. But a grid is not an efficient experiment. Optimal experimental design asks: given my current uncertainty, which next run most reduces the variance of my extrapolation at C = 1e24? Almost always the answer is "the largest one you can afford, plus one that breaks a collinearity" — not "fill in the grid." This is the "active learn" remark, and it is a genuinely open, publishable, cheap research direction.

# The intuition behind active learning for scaling laws, in miniature.
# Fitting a line from points clustered together gives a terrible slope estimate.
def slope_variance(x_points):
    n = len(x_points)
    xbar = sum(x_points) / n
    sxx = sum((x - xbar) ** 2 for x in x_points)
    return 1.0 / sxx                        # Var(slope) ∝ 1 / Σ(x - x̄)²

clustered = [19.0, 19.2, 19.4, 19.6]        # four runs, all about the same size
spread    = [18.0, 19.0, 20.0, 21.0]        # same COUNT of runs, spread out
print(round(slope_variance(clustered), 3))  # 5.0   <- bad extrapolation
print(round(slope_variance(spread), 3))     # 0.2   <- 25x better, same budget

Same number of runs. 25× lower variance on the slope purely from where you placed them. That is what "where do we get max info gain?" means, and it costs nothing to apply.

(c) D is not what you think it is. This leads directly into Claim 16 — his slide says D "was opaque and recipe-specific. You wouldn't be blamed for assuming iid." It is not iid: repeated data, deduplication, and mixture weights all change what a "token" is worth.

Takeaway. Three fundable research projects sit in one slide, and none of them needs a supercomputer. This is the most actionable slide in the whole talk.

Claim 11 — MoE: what it is, why it wins, why it hurts

This is the concept the user asked about by name, so we build it completely from zero.

The problem MoE solves

In a dense transformer, every parameter participates in every token. If you want more knowledge in the model, you add parameters — and every token now costs more to process. Cost and capacity are welded together.

Mixture of Experts (MoE) breaks the weld. Replace the feed-forward block with E parallel copies ("experts") plus a small router. For each token, the router picks the top-k experts (typically k = 1 or 2) and only those run.

DENSE FFN                          MoE FFN (E=8, k=2)

  token                              token
    │                                  │
    ▼                                  ▼
┌────────┐                        ┌─────────┐
│  FFN   │  all params            │ ROUTER  │  tiny: d_model × E
│ (100%) │  run for every         └────┬────┘
└────────┘  token                      │ scores 8 experts, picks best 2
    │                          ┌───┬───┼───┬───┬───┬───┬───┐
    ▼                          ▼   ▼   ▼   ▼   ▼   ▼   ▼   ▼
  output                      E0  E1  E2  E3  E4  E5  E6  E7
                               ·   ✓   ·   ·   ✓   ·   ·   ·
                                   └───────┬───────┘
                                           ▼  weighted sum
                                        output

  8× the parameters. 2/8 = 25% of the compute per token.

The trade in one line: parameters (memory) scale with E; FLOPs scale with k.

The router, in full

import math

def softmax(xs):
    m = max(xs)
    e = [math.exp(x - m) for x in xs]
    s = sum(e)
    return [v / s for v in e]

def route(token_vec, router_w, k=2):
    """router_w: E x d_model. Returns [(expert_id, gate_weight), ...] of length k."""
    logits = [sum(w * t for w, t in zip(row, token_vec)) for row in router_w]
    probs = softmax(logits)
    top = sorted(range(len(probs)), key=lambda i: -probs[i])[:k]
    # Renormalize over the chosen k so the gates sum to 1.
    total = sum(probs[i] for i in top)
    return [(i, probs[i] / total) for i in top]

token = [0.5, -0.2, 0.9, 0.1]
router_w = [                       # 6 experts, d_model = 4
    [1.0, 0.0, 0.0, 0.0],
    [0.0, 1.0, 0.0, 0.0],
    [0.0, 0.0, 1.0, 0.0],
    [0.0, 0.0, 0.0, 1.0],
    [0.5, 0.5, 0.0, 0.0],
    [0.0, 0.0, 0.5, 0.5],
]
print(route(token, router_w, k=2))
# [(2, 0.5987...), (0, 0.4013...)]  -> experts 2 and 0, with those gate weights

Then the layer output is Σ_over_chosen gate_i × expert_i(token).

Load balancing — the thing that actually breaks

Left alone, routers collapse. Early in training one expert is randomly slightly better, so it gets more tokens, so it trains more, so it gets better, so it gets more tokens. Within a few thousand steps you have one expert doing everything and E−1 dead ones — you paid for 8× the parameters and got a dense model.

The fix is an auxiliary load-balancing loss added to the training objective:

def load_balance_loss(assignments, gate_probs, n_experts):
    """Switch-Transformer style: L_aux = E * Σ_i f_i * P_i.

    f_i = fraction of TOKENS routed to expert i        (discrete, no gradient)
    P_i = mean ROUTER PROBABILITY assigned to expert i (continuous, differentiable)

    Multiplying them makes the loss differentiable through P while being driven
    by the actual imbalance in f. Minimized when both are uniform (= 1/E each),
    giving L_aux = E * E * (1/E) * (1/E) = 1.0.
    """
    n_tokens = len(assignments)
    f = [0.0] * n_experts
    for e in assignments:
        f[e] += 1.0 / n_tokens

    P = [0.0] * n_experts
    for probs in gate_probs:
        for i, p in enumerate(probs):
            P[i] += p / n_tokens

    return n_experts * sum(fi * Pi for fi, Pi in zip(f, P))


E = 4
balanced   = [0, 1, 2, 3] * 2
collapsed  = [0] * 8
uni  = [[0.25] * 4] * 8
skew = [[0.97, 0.01, 0.01, 0.01]] * 8
print(round(load_balance_loss(balanced,  uni,  E), 4))   # 1.0000  <- ideal
print(round(load_balance_loss(collapsed, skew, E), 4))   # 3.8800  <- heavily penalized

You add α · L_aux (typically α ≈ 0.01) to the main loss. Too small and you collapse; too large and you damage quality by forcing nonsensical routing. This coefficient is one of the most finicky hyperparameters in modern pre-training.

Capacity factor — the batching constraint nobody mentions

Hardware wants fixed-size tensors. So each expert gets a fixed buffer:

capacity = capacity_factor × (tokens_in_batch × k / n_experts)

If more tokens route to an expert than fit, the overflow is dropped (passed through by the residual connection, unprocessed). If fewer arrive, the buffer is padded with zeros — wasted compute.

def capacity_stats(assignments, n_experts, capacity_factor=1.25, k=1):
    n_tokens = len(assignments)
    cap = int(capacity_factor * n_tokens * k / n_experts)
    counts = [0] * n_experts
    for e in assignments:
        counts[e] += 1
    dropped = sum(max(0, c - cap) for c in counts)
    padded  = sum(max(0, cap - c) for c in counts)
    return {"capacity_per_expert": cap, "dropped": dropped, "padded_slots": padded,
            "drop_rate": dropped / n_tokens}

# 100 tokens, 4 experts, mildly imbalanced routing
assign = [0]*40 + [1]*30 + [2]*20 + [3]*10
print(capacity_stats(assign, 4, capacity_factor=1.25))
# {'capacity_per_expert': 31, 'dropped': 9, 'padded_slots': 33, 'drop_rate': 0.09}

9% of tokens silently skip the FFN. This is a real, live source of quality loss in production MoE models, and tuning capacity_factor against drop rate is routine pre-training work.

What his slides say about MoE scaling

"MoE scaling laws are better, but have implications for token hunger. We're running out of internet! ... Notice relative data hunger compared to dense! At same active param count and fixed 100B token training, MoE 64E improves on dense."

Decoded: for a fixed compute budget, an MoE reaches lower loss than a dense model — the scaling law is strictly better. But the compute-optimal D for an MoE is larger. MoE converts "we have compute" into "we need more unique tokens," and unique high-quality tokens are the resource that is actually running out. MoE trades a compute problem for a data problem. (The reference here is Clark et al., Unified Scaling Laws for Routed Language Models, 2022.)

Takeaway. MoE is not free capacity. You pay in HBM, in routing instability, in dropped tokens, in data hunger, and — the big one, next — in communication. Phase 03 builds all of this; Phase 06 fixes the communication.

Claim 12 — The MoE serving bottleneck, and Geng Yan's pipeline-prefill fix

This is the most technically specific story in the interview, and it is worth full precision.

The problem

An MoE has many more parameters than a dense model of equal compute. Those parameters must live in HBM. A single chip's HBM (16 GB on a v5e; 80 GB on an H100) cannot hold them. So you shard the experts across chips — expert 0–7 on chip 0, experts 8–15 on chip 1, and so on. This is expert parallelism (EP).

Now trace one token through one layer:

Layer ℓ:   token lives on chip 0
           router says "you need expert 37"
           expert 37 lives on chip 4
           ──> send the token's activation vector to chip 4     [NETWORK]
           chip 4 computes
           ──> send the result back to chip 0                   [NETWORK]

Layer ℓ+1: router says "you need expert 12" (on chip 1)
           ──> send to chip 1                                   [NETWORK]
           ──> send back                                        [NETWORK]

... repeat for every one of ~60 layers.

His description of exactly this: "that token might live on the first TPU, but it needs to go to the last TPU." Every layer, for every token. The collective involved is an all-to-all (every chip sends a different slice to every other chip), which is the most expensive collective there is, and its cost, as he notes, "increases dramatically with N."

def moe_alltoall_cost(layers, tokens, d_model, n_chips, bytes_per=2,
                      link_bw_gbps=100, latency_us=5):
    """Rough per-forward-pass communication cost of naive expert parallelism."""
    bytes_per_hop = tokens * d_model * bytes_per
    # Each layer: dispatch to experts + combine back = 2 all-to-alls.
    total_bytes = layers * 2 * bytes_per_hop
    transfer_s = total_bytes / (link_bw_gbps * 1e9 / 8)
    latency_s = layers * 2 * latency_us * 1e-6      # fixed cost, paid per collective
    return {"GB moved": total_bytes / 1e9,
            "transfer_s": transfer_s,
            "latency_s": latency_s,
            "total_s": transfer_s + latency_s}

print(moe_alltoall_cost(layers=60, tokens=8192, d_model=8192, n_chips=16))

{'GB moved': 16.1, 'transfer_s': 1.288, 'latency_s': 0.0006, 'total_s': 1.289}

1.3 seconds of pure network time, before a single useful FLOP. For a product with a sub-second latency budget, that is fatal. This is the wall the Flash team hit.

The insight

He credits Geng Yan — described as a junior member of the team — with the fix. Restated precisely:

Stop parallelizing across experts. Parallelize across layers instead.

That is pipeline parallelism, applied at prefill time. Chip 0 holds layers 1–10 (with all their experts). Chip 1 holds layers 11–20. And so on.

Now the token does not hop around per layer. It flows forward through the pipeline, once:

NAIVE EXPERT PARALLELISM              PIPELINED PREFILL
(shard experts, keep all layers)      (shard layers, keep all experts local)

chip0 ⇄ chip1 ⇄ chip2 ⇄ chip3         chip0 ──> chip1 ──> chip2 ──> chip3
  ↕      ↕       ↕       ↕            L1-15    L16-30    L31-45    L46-60
 all-to-all EVERY layer (×60)         one hop per STAGE (×3), point-to-point

And now the crucial second half of the idea — his exact framing: "layer one on the first chip is processing the second thousand tokens... while layer two is working on the first thousand tokens." Chunk the prompt and stream the chunks through the pipeline, so every stage is busy on a different chunk simultaneously:

time ──────────────────────────────────────────────────>
chip0 (L1-15) : [chunk1][chunk2][chunk3][chunk4][chunk5]
chip1 (L16-30):        [chunk1][chunk2][chunk3][chunk4]
chip2 (L31-45):               [chunk1][chunk2][chunk3]
chip3 (L46-60):                      [chunk1][chunk2]
                └ bubble ┘  └──── steady state, all chips busy ────┘

The chip→chip transfer of chunk i overlaps with computation on chunk i+1.
Communication is HIDDEN, not eliminated.

His summary of the effect: communication went "from something that required a lot of token exchange on every single layer to something that actually can be hidden behind other computation."

def pipeline_prefill(n_chunks, n_stages, compute_per_stage_ms, transfer_ms):
    """Steady-state pipeline: total time = fill + (chunks * per-chunk cost)."""
    step = max(compute_per_stage_ms, transfer_ms)   # overlapped -> the max, not the sum
    fill = (n_stages - 1) * step                    # the 'bubble' at the start
    return fill + n_chunks * step

serial = 4 * 8 * (50 + 20)                          # 4 stages, 8 chunks, no overlap
piped  = pipeline_prefill(8, 4, 50, 20)
print(serial, "ms serial  ->", piped, "ms pipelined")   # 2240 ms -> 550 ms

Bubble efficiency is the thing to reason about: with S stages and M chunks, utilization is M / (M + S − 1). With 4 stages and 8 chunks: 8/11 = 73%. With 32 chunks: 32/35 = 91%. More chunks = smaller bubble, but each chunk is a smaller matmul with worse arithmetic intensity. That trade-off is the design.

Why prefill specifically

This is the part that makes the insight clever rather than obvious, and most summaries miss it.

  • Prefill processes the whole prompt at once. It is compute-bound and has thousands of tokens available to chunk. Pipelining has plenty of work to hide communication behind. ✅
  • Decode produces one token at a time. It is memory-bandwidth-bound, and there is no parallel work to overlap with. A pipeline here just adds S−1 serial hops to every single token. ❌

So you use different parallelism strategies for the two phases of the same request — which is exactly the modern prefill/decode disaggregation design. His remark that this "serving-time innovation" is what "made [an MoE Gemini 2.0 series] possible" is the payoff: the serving strategy unlocked the architecture choice.

Takeaway — the transferable lesson. The fix was not a better kernel or a better model. It was changing which axis you shard along, informed by which phase of inference you are in. Note also who found it: a junior engineer. Feinberg's response was to run "a very transparent technical process to get to the bottom of this" — the leadership behavior that lets a junior insight become a flagship decision. Phase 06 builds this scheduler.

Claim 13 — The Flash 2.0 war story: 40 days, five people, two continents

The largest model they had ever trained at the Flash scale. "40 days of grueling work for a really, really small team" — "five people on the rotation," rotating "day by day, handing off all of this SRE-style work of keeping the training job alive." They monitored data iterators and fixed indexing issues to avoid "wasting all of this GPU time." "We did not do a lot of sleeping" — "dual shifts across the Paris office and Mountain View."

Decode it — why does a training run need a pager?

A newcomer's model of training is model.fit(). At this scale the reality is a distributed system with thousands of nodes running for weeks, and the failure modes are exotic:

FailureWhat it looks likeWhy it is brutal
Loss spikeloss jumps 2→8 in one stepMay recover, may permanently damage the model. You have minutes to decide whether to roll back.
Hardware failureone chip of 10,000 diesSynchronous training means the whole job stalls. At 10k chips with a 3-year MTBF, expect a failure every few hours.
Silent data corruption (SDC)a chip computes wrong numbers without erroringThe worst one. No crash, no alert; the model just gets subtly worse. Detected only by cross-replica checksums.
Data iterator bugwrong shard, an off-by-one, a repeated segmentThis is the one he names. Silent. You can burn days of compute training on the wrong data.
Stragglersone slow nodeEveryone waits at the all-reduce barrier; throughput collapses to the slowest node.
Checkpoint corruptionthe save itself failsYour rollback point is gone. Discovered at the worst moment.

Under the hood — goodput, the number that matters

def goodput(total_hours, crash_count, restart_minutes, checkpoint_interval_min):
    """Fraction of wall-clock actually spent making forward progress.

    Every crash costs: the restart itself, PLUS the work done since the last
    checkpoint (lost), PLUS on average half a checkpoint interval of redo.
    """
    lost_per_crash_h = (restart_minutes + checkpoint_interval_min / 2) / 60
    lost = crash_count * lost_per_crash_h
    return max(0.0, (total_hours - lost) / total_hours)

# 40 days = 960 hours. Compare a fragile setup to a hardened one.
print(round(goodput(960, crash_count=200, restart_minutes=45,
                    checkpoint_interval_min=120), 3))   # 0.635
print(round(goodput(960, crash_count=200, restart_minutes=10,
                    checkpoint_interval_min=15), 3))    # 0.939

Same hardware, same number of crashes: 64% vs 94% goodput. That 30-point gap is 12 days of a 40-day run — or, in money, a seven-figure swing, and quite possibly the difference between shipping before a competitor and shipping after. The entire delta comes from two boring engineering decisions: fast restarts and frequent (asynchronous) checkpoints. This is why the SRE hat is not beneath the researcher; it is a large fraction of the deliverable.

Note also which lever matters more. Halving restart time saved ~35 minutes per crash; going from 2-hour to 15-minute checkpoints saved ~52 minutes per crash. Checkpoint cadence is the bigger lever, and it is limited by how fast you can write terabytes of optimizer state — which is why asynchronous and sharded checkpointing is a real engineering discipline, not a config flag.

The loss-spike playbook

The single most common 3 a.m. event. A working decision procedure:

def spike_response(loss_history, window=100, z_threshold=6.0):
    """Detect a spike against a rolling baseline, and pick an action."""
    if len(loss_history) < window + 1:
        return "warmup: insufficient history"
    recent = loss_history[-window - 1:-1]
    mean = sum(recent) / len(recent)
    var = sum((x - mean) ** 2 for x in recent) / len(recent)
    std = var ** 0.5 or 1e-9
    z = (loss_history[-1] - mean) / std

    if z < z_threshold:
        return "normal"
    if z < 15:
        return "WATCH: skip this batch, log it, continue"       # often self-heals
    return "ROLLBACK: restore last checkpoint, skip N batches, lower LR"

Standard mitigations, in escalation order: (1) skip the offending batch (a bad data shard is a common cause); (2) lower the learning rate and re-warm; (3) roll back to the last checkpoint and skip forward past the data; (4) if it recurs at the same step, it is the data, not luck — go find it. Real interventions from published reports include z-loss regularization, QK-norm, and clipping gradients more aggressively.

The competitive coda

Flash 2.0 landed around the same time as DeepSeek-V3. He notes that a Wall Street Journal piece had "some elided rows" in its leaderboard comparison, making Gemini look badly positioned — while the actual leaderboard showed "Flash 2.0 Thinking up in the top right corner, way far ahead of DeepSeek-V3."

The transferable lesson has nothing to do with either company: leaderboard screenshots are marketing artifacts. Always ask what was filtered, which variant was tested, what the axes are, and whether the comparison controls for cost. His own slides make the same point from the other side — "LMSys is not the end-all-be-all," and Llama 4 Maverick showed ranking can be "volatile and overfit to human preference."

Takeaway. Frontier pre-training is an operations discipline wearing a research hat. If you want a way in that is less crowded than "research scientist," become excellent at training-run reliability. Phase 09 builds the watchdog.

Claim 14 — Real-time products force small models (the napkin math)

His slides do this calculation live, and it is the single best worked example in the talk of how a product requirement becomes an architecture constraint. Reproduced and extended:

The setup

A web-interaction agent with:

  • 128k prefill, but only 8k incremental per turn
  • 128 decode tokens (enough to emit an action)
  • No more than 1 second of latency between actions
  • 250ms of that goes to "scaffolding, load balancing, request validation, kv cache retrieval" — and he flags this as "optimistic!"

Experiment: Llama3-70B on v5e chips. Assume fully compute-bound on prefill, HBM-bound on decode.

The arithmetic

# --- Chip and model constants -------------------------------------------------
V5E_FLOPS = 197e12          # bf16 peak FLOP/s for one TPU v5e
V5E_HBM_BW = 819e9          # bytes/s of HBM bandwidth
V5E_HBM_GB = 16             # GB capacity  <- note: a 70B model in bf16 needs 140 GB

N = 70e9                    # Llama3-70B parameters
BYTES_PER_PARAM = 2         # bf16

def prefill_seconds(n_tokens, n_params, n_chips, mfu=1.0):
    """Prefill is COMPUTE bound: 2N FLOPs per token (forward only).
    mfu=1.0 is his stated idealization: 'assume fully compute bound on prefill'."""
    flops = 2 * n_params * n_tokens
    return flops / (n_chips * V5E_FLOPS * mfu)

def decode_seconds(n_tokens, n_params, n_chips, efficiency=1.0):
    """Decode is MEMORY bound: every generated token re-reads EVERY weight.
    With the model sharded over n_chips, each chip reads its 1/n_chips slice."""
    bytes_moved = n_params * BYTES_PER_PARAM * n_tokens
    return bytes_moved / (n_chips * V5E_HBM_BW * efficiency)

for chips in (1, 4, 16, 64, 128):
    p = prefill_seconds(8192, N, chips)      # the 8k incremental prefill
    d = decode_seconds(128, N, chips)        # 128 tokens to emit one action
    total = p + d + 0.25                     # + the 250ms scaffolding budget
    verdict = "OK " if total <= 1.0 else "MISS"
    print(f"{chips:4d} chips: prefill {p:7.3f}s  decode {d:7.3f}s  "
          f"total {total:7.3f}s  {verdict}")

   1 chips: prefill   5.822s  decode  21.881s  total  27.953s  MISS
   4 chips: prefill   1.455s  decode   5.470s  total   7.175s  MISS
  16 chips: prefill   0.364s  decode   1.368s  total   1.982s  MISS
  64 chips: prefill   0.091s  decode   0.342s  total   0.683s  OK 
 128 chips: prefill   0.045s  decode   0.171s  total   0.466s  OK

The prefill column reproduces his slide exactly: "Uh oh… 5.7 seconds for 1 chip. So to hit 0.5 sec api limit we already need to have a 4x4 prefill station of v5e." One chip gives 5.8s; a 4×4 = 16-chip station brings prefill to 0.36s, under his 0.5s API limit. And the audience question he poses next is the right one: "how would we shard on 4x4?" — 16 chips is not a number, it is a topology (tensor/pipeline split, mesh shape, and the collectives that follow). Phase 04 answers it.

The five conclusions that follow

  1. The prefill station alone needs 16 chips to serve one conversation inside the latency budget. Not 16 chips for the service — 16 for one user.
  2. Decode is worse than prefill at batch 1, by ~3.8×. Every generated token re-reads all 140 GB of weights to do a trivial amount of arithmetic. This is the memory-bound regime from Phase 00, and it is why the full budget needs ~64 chips, not 16.
  3. The two phases want different hardware allocations — which is the entire argument for prefill/decode disaggregation, and the setup for Claim 12.
  4. Therefore: make the model smaller. Halving N halves both columns linearly. This is the entire economic case for Flash and Flash-Lite, and it is why his job exists.
  5. The 70B doesn't even fit. 140 GB of weights across 16 GB chips means ≥9 chips just to hold it, before any latency consideration. Capacity and latency push the same way.

Caveat, stated honestly: batch-1 decode is the worst case. Real serving batches many requests, which amortizes the weight reads across all of them and dramatically improves the decode column — that is why continuous batching exists. But an interactive agent that must respond in under a second cannot always wait to fill a batch. The tension between latency (small batch) and throughput (large batch) is the central trade-off of serving, and this table is where you feel it.

Takeaway. This is the calculation that converts "we want a real-time agent" into "we need a distilled, quantized, inference-co-designed small model." Do this arithmetic before you pick a model, not after. Phase 05 makes you build the full version with a roofline.

Claim 15 — Chinchilla ignores inference cost

His slide: "Chinchilla-style Scaling Ignores Inference Cost." The direct fix: "Globally optimize FLOPs between training and inference?" — citing Sardana et al., Beyond Chinchilla-Optimal, 2024.

Decode it

Chinchilla minimizes loss for a fixed training budget. But a deployed model's lifetime cost is training plus all inference, forever. If you serve a lot, it is worth overtraining a smaller model — spending more training FLOPs than Chinchilla says, on a model smaller than Chinchilla says — because you amortize that over trillions of served tokens.

def total_lifetime_flops(N, D_train, D_inference):
    return 6 * N * D_train + 2 * N * D_inference     # 6ND to train, 2N/token to serve

# Two models targeting the SAME quality (illustrative, from published fits).
chinchilla = dict(N=70e9,  D_train=1.4e12)          # "compute-optimal"
overtrained = dict(N=20e9, D_train=8.0e12)          # smaller, trained much longer

for name, m in [("Chinchilla-optimal", chinchilla), ("Overtrained-small", overtrained)]:
    for served in (1e12, 1e14, 1e16):
        total = total_lifetime_flops(m["N"], m["D_train"], served)
        print(f"{name:20s} served={served:.0e}  lifetime={total:.3e} FLOPs")
    print()

Chinchilla-optimal   served=1e+12  lifetime=7.280e+23 FLOPs
Chinchilla-optimal   served=1e+14  lifetime=1.459e+25 FLOPs
Chinchilla-optimal   served=1e+16  lifetime=1.401e+27 FLOPs

Overtrained-small    served=1e+12  lifetime=1.000e+24 FLOPs
Overtrained-small    served=1e+14  lifetime=4.960e+24 FLOPs
Overtrained-small    served=1e+16  lifetime=4.010e+26 FLOPs

At 10¹² served tokens the Chinchilla model wins. By 10¹⁴ the overtrained small model is ~2.9× cheaper overall, and by 10¹⁶ it is 3.5× cheaper. The crossover is the decision, and it depends entirely on a number the research team does not control: how many tokens the product will serve.

The three problems he raises with this idea

His slides are notably skeptical of the clean version, and the objections are the interesting part:

(1) Non-homogeneity of compute. "Inference-optimized chips. Also global optimization is not how cross-org planning actually works." Training FLOPs and inference FLOPs are not the same currency — they happen on different chips, in different datacenters, on different budgets, owned by different VPs. "But in principle can adjust the formulas for 'business cost'" — i.e. the real objective is dollars, and the FLOP-exchange-rate is an org-chart question as much as a physics one.

(2) Non-forecastability of D_inf. You cannot know how many tokens you will serve. Two named reasons:

  • Jevons paradox — making a resource cheaper increases total consumption. Every efficiency win you deliver gets eaten by more usage.
  • Market expansion from quality improvements — a better model unlocks use cases that did not exist, so demand is a function of the very quality you are optimizing.

So D_inf appears in your objective and is caused by your objective. Feedback loop, no fixed point. This is a genuinely unsolved problem, not a modelling nicety.

(3) Badness of fit. He points at the paper's own Fig 5 / Table 1. The inference-aware laws are extrapolating into the heavily-overtrained regime — far past where the fits were calibrated — and that is exactly where the classical functional forms behave worst. Which leads directly to the next claim.

Takeaway. The right objective is lifetime cost, not training cost. But the honest version of the calculation contains a term nobody can forecast, so in practice you do scenario analysis over D_inf and pick something robust across scenarios. Phase 02 builds this.

Claim 16 — "We're running out of internet"

His slide, on MoE's data hunger: "We're running out of internet!" And the fix he spends most of his time on: "Unsurprisingly, this is where we spend most of our time, even as modelling people. Probably half my focus this year so far." — namely (1) multimodal data (audio, visual, 3D, video) and (2) synthetic data.

Note that line carefully: the pre-training lead spends half his time on data. Not on architecture. Not on optimizers. On data.

The data-constrained law

The reference is Muennighoff et al., Scaling Data-Constrained Language Models (2023). His slide's framing is the sharpest summary you will find of why it matters:

"D was opaque and recipe-specific. You wouldn't be blamed for assuming iid." "New dimension: intentionally unique data, L(N, U, R)." "Upshot: yet smaller models, more resilient to repeats."

Decoded: everyone writes L(N, D) where D = "tokens seen." But 1 trillion tokens seen could mean 1T unique tokens once, or 200B unique tokens five times. Those are not the same experiment, and the classical law cannot tell them apart. So you split the axis:

  • U = unique tokens in your corpus
  • R = number of repeats (epochs)
  • D = U × R

And the empirical finding: repeated tokens are worth almost as much as fresh ones for the first few epochs, then their value decays fast.

import math

def effective_tokens(U, R, half_life=5.0):
    """Value of R epochs over U unique tokens, with exponentially decaying returns.

    Matches the paper's qualitative finding: ~4 epochs is nearly free,
    ~16 epochs adds almost nothing, and after that you may be actively hurting.
    """
    return U * sum(math.exp(-(r - 1) / half_life) for r in range(1, R + 1))

U = 100e9
for R in (1, 2, 4, 8, 16, 32):
    eff = effective_tokens(U, R)
    print(f"R={R:2d}  raw D={U*R/1e9:6.0f}B  effective={eff/1e9:6.1f}B  "
          f"efficiency={eff/(U*R):5.1%}")

R= 1  raw D=   100B  effective= 100.0B  efficiency=100.0%
R= 2  raw D=   200B  effective= 181.9B  efficiency= 90.9%
R= 4  raw D=   400B  effective= 303.8B  efficiency= 75.9%
R= 8  raw D=   800B  effective= 440.3B  efficiency= 55.0%
R=16  raw D=  1600B  effective= 529.2B  efficiency= 33.1%
R=32  raw D=  3200B  effective= 550.7B  efficiency= 17.2%

At R = 32 you have spent 32× the compute for 5.5× the effective data — and the last 16 epochs bought you 4% more. His slide's "5 epochs" annotation sits right at the elbow of that curve, which is not a coincidence.

This is a toy model, deliberately. The exponential-decay form is a stand-in that reproduces the paper's qualitative finding (a few epochs are nearly free, many are nearly worthless). The real fit in Muennighoff et al. has a different functional form with fitted half-life parameters. Use this to build intuition; use the paper to make decisions.

Why this changes the architecture decision

If unique data is the binding constraint, big models are the wrong answer, because big models are precisely the ones that need lots of fresh data to justify their capacity. Hence his slide's conclusion: "yet smaller models, more resilient to repeats." Data scarcity and serving economics push in the same direction — which is very convenient, and is a large part of why the industry converged on Flash-class models.

The two escape hatches

Multimodal data. Audio, images, video, 3D. Text on the internet is finite; video is effectively not. It also carries information text never encodes (physical dynamics, spatial relations). The cost: tokenization and encoder design get much harder, and token counts explode (a second of video can cost hundreds of tokens).

Synthetic data. He makes a subtle point most people get wrong:

"Without filter, it can help in the Stein's paradox sense (Jain et al 2024). Tradeoff: Generation Quality vs. Filtering."

Stein's paradox is the famous statistical result that a biased estimator can have lower total error than an unbiased one — shrinking your estimates toward a common point beats using each observation on its own. Applied here: synthetic data is biased (it reflects the generator's distribution, not the truth) but it is lower variance. Adding it can reduce total error even though it adds bias. That is a much more precise and more defensible argument for synthetic data than "we ran out of text."

The trade-off he names is real: generate more carefully (expensive) or generate cheaply and filter hard (also expensive, and filtering can be its own source of bias).

The counterpoint on his own slide

"Llama3: D_inf = inf!" — quoting the Llama 3 paper: "Both our 8B and 70B parameter models continued to improve log-linearly after we trained them on up to 15T tokens."

"Could be quite valid for open source! Just pick sizes and train on all your data! We could be doing research with those FLOPs! Use this forecast to estimate how much regret we got."

That is an honest and slightly self-critical note. If returns have not saturated, the simplest strategy — pick a size and train on everything — is close to optimal, and all the sophisticated scaling work buys you less than it appears to. His framing of "regret" (how much worse off you were than the best strategy in hindsight) is the right way to hold this. And "job is to push the curves right" is the mission statement.

Takeaway. Data is where a pre-training lead's time actually goes. If you want to be useful to a frontier lab fast, get extremely good at data quality, deduplication, mixture weighting, and synthetic-data filtering. It is less glamorous than architecture and worth more.

Claim 17 — Distillation scaling laws and the capacity gap

He cites Busbridge et al., Distillation Scaling Laws (2025), and the question it poses: "How to spend FLOPs with teacher?" Then he pushes back on one of its findings, and the pushback is a masterclass in reading a paper critically.

The question

You have a compute budget. You can spend it on: (a) making the teacher better, (b) running the teacher over more tokens to generate more supervision, or (c) training the student longer. It is a three-way allocation problem, and it has an optimum.

The "capacity gap" and his three objections

The claimed phenomenon: if the teacher is too much better than the student, distillation gets worse, not better — the student cannot represent the teacher's function, so chasing it hurts.

His response, point by point (from his slide, annotated):

(1) "very weak effect from up-trend; and not typical regime." The effect is small in the data, and the region where it appears is not where anyone actually operates. Lesson: always ask whether a reported effect is in the regime you care about.

(2) The temperature objection — this one is elegant. "Teacher pplx can be arbitrarily weakened by just adding temperature! Take a really good teacher → Eq8 predicts bad distill → but add high temp and it will be good?"

Unpacked: the law expresses the capacity gap in terms of teacher perplexity. But you can change a teacher's effective perplexity for free by raising the softmax temperature — that flattens its distribution without changing what it knows. So the formula predicts you could fix the capacity gap by turning a knob that carries no information. That is a reductio: if a free, information-free transformation moves your predictor, your predictor is parameterized on the wrong variable.

def perplexity(logits, T=1.0):
    """Raising temperature flattens the distribution and RAISES perplexity —
    without the teacher knowing anything less."""
    m = max(logits)
    e = [math.exp((z - m) / T) for z in logits]
    s = sum(e)
    p = [x / s for x in e]
    H = -sum(pi * math.log(pi) for pi in p if pi > 0)
    return math.exp(H)

teacher = [8.0, 2.0, 1.0, 0.5, 0.0]
for T in (1.0, 2.0, 4.0, 8.0):
    print(f"T={T}: perplexity={perplexity(teacher, T):.3f}")

T=1.0: perplexity=1.033
T=2.0: perplexity=1.626
T=4.0: perplexity=3.429
T=8.0: perplexity=4.586

Same teacher, same knowledge, perplexity moved 4.4× from a knob you set at inference time. Any law keyed on that number inherits the knob.

(3) "In practice, you can James-Stein this away with weight tuning with supervised objective." In production you never distill purely — you mix the distillation loss with the ordinary next-token loss:

def combined_loss(student_logits, teacher_logits, true_token, lam=0.5, T=2.0):
    """lam=1 -> pure distillation; lam=0 -> pure supervised. Reality lives in between."""
    distill = kl_distillation_loss(student_logits, teacher_logits, T)
    ce = cross_entropy(student_logits, true_token)
    return lam * distill + (1 - lam) * ce

Tuning λ is literally a shrinkage estimator — the James-Stein reference is exact, not metaphorical. You are trading bias (teacher's errors) against variance (single-label noise). And that gives his cleanest reframe of the entire topic:

"Distill as variance reduction. Better teacher will just reduce bias."

That one sentence is the best mental model for distillation in existence. The teacher's dense distribution is a low-variance estimate of the true next-token distribution — enormously less noisy than a one-hot label. The teacher's imperfection is the bias. Better teacher → less bias. Distillation at all → less variance. Everything else is engineering.

Takeaway. Watch how he read that paper: check the regime, look for a free transformation that breaks the parameterization, and check whether the effect survives normal practice. That is what "mathematical maturity" means in his hiring criteria, made concrete.

Claim 18 — The Gemini tick-tock

His slide: "Gemini tick-tock (Flash goal to match Pro of previous gen)." And: "Scaling Work has two flavors: (1) Adding points to Quality × Model Size plot. (2) Increasing the slope of the plot."

Decode it

Borrowed from Intel's old CPU cadence. The pattern:

Gen N   :  Pro  ──── quality X ────┐
                                   │  next generation must deliver X
Gen N+1 :  Flash ──── quality X ───┘  at a fraction of the size and cost
           Pro   ──── quality X+Δ

Every generation, the small cheap model must reach what the previous generation's flagship reached. That is a compression mandate with a deadline, and it is why his three verticals are what they are: distillation, quantization, and serving-friendly architecture are the only three ways to hit it.

The two "flavors" distinction is worth internalizing:

FlavorWhat it meansHow you do it
Adding pointsfill in the quality-vs-size curve at a new sizetrain another model at a new N
Increasing the slopemake the whole curve better — more quality per parameterbetter architecture, better data, better distillation, better optimizer

His closing note ties them: "Inference Efficiency Work: Compression work grows with both scaling aspects" — via (1) better distillation recipes, (2) quantization, (3) serving-friendly model design changes.

Takeaway. Adding points is production. Moving the slope is research. Know which one you are being asked for, and know which one you are doing.

Claim 19 — You can do pre-training research without a supercomputer

His slide is titled "Future Pretrain Research Ideas – Without Big Costs!" and opens with the objection it demolishes: "Common refrain: pretraining is expensive, only can be researched in industry." Then four counterexamples. If you want research to put on a resume, this is the list.

1. Kernels and kernel languages. "Developing hardware-focussed kernels is the hot-loop for research now. Kernel programming languages, compiler tools, developer tools that make this easier are crucial. Or come up with the next flash attention." — Needs one GPU. High impact. Directly hireable. (Phase 11.)

2. Vector quantization. "Quantization entering a new frontier from vector quant." Scalar quantization rounds each weight independently. Vector quantization replaces groups of weights with a codebook index, exploiting correlations between them. Far better rate-distortion in principle; largely unexploited in LLMs. Needs a laptop to prototype. (Phase 08.)

3. FunSearch-style inference-vs-quality trade-offs. "For LLM-in-the-loop for search." How much extra inference compute (samples, search width, verification) buys how much quality? This is test-time-compute scaling, and it is an open, cheap research area.

4. The statistics of scaling laws. Covered in Claim 10 — the estimator question and the optimal-design question. Pure statistics on published data. Zero GPUs. Possibly the highest ratio of impact-to-cost on the entire list.

Takeaway. The barrier to entry is not compute. It is knowing which questions are open. He just told you four of them.

Claim 20 — Hiring: intent, mathematical maturity, grit

He screens for three things: "intent, mathematical maturity, grit." Mathematical maturity means "being able to dive into a paper of that level and then understand it, being able to take a research idea from a paper and implement it yourself." He also stresses "having read and having the skills to effectively traverse the historical citation tree for a particular topic" and knowing "what are the high-value papers."

The concrete asks, and how to actually execute them

(a) Do How To Scale Your Model ("The Scaling Book") exercises — handwritten, on video. He publicly offered interviews for this, and referrals when he lacked headcount. This is an open, standing, verifiable offer, and it is stunning how few people take it.

How to do it well: work the arithmetic by hand, show the units at every step, and narrate why each quantity matters, not just what it equals. The video is evidence you did it yourself; the narration is evidence you understood it.

(b) Implement a transformer from scratch. Demonstrates "willingness to get into the weeds engineering-wise" and the "bread-and-butter math that we use every day to size these LLMs."

How to do it well: not a 200-line nanoGPT copy. Include the FLOP counter, the memory accounting, a KV cache, and a benchmark showing you know where time actually goes. The differentiator is the instrumentation, not the model.

(c) Contribute to vLLM / SGLang / TensorRT. "Actual evidence that you've created something of use to other people" is what he wants — an improvement "for this and that setting."

How to do it well: pick a narrow, measurable win (a kernel for an under-served shape, a scheduler edge case, a quantization format), benchmark it honestly, and write the PR description like a small paper — problem, measurement, fix, measurement.

(d) Learn to traverse a citation tree. This is a teachable procedure, not a vibe:

1. Find the newest strong survey or the most-cited recent paper on the topic.
2. Read its related-work section. Note which papers it treats as LOAD-BEARING
   (described in detail) vs merely listed.
3. Follow those backward to the origin papers. Read the origin paper in full.
4. Use a citation index to walk FORWARD from the origin: who cites it, and
   which of those are highly cited themselves? That intersection is the spine.
5. Build a timeline: what changed at each step, and WHY the previous answer
   was insufficient. That "why" chain IS the field's argument.
6. Find the paper that CONTRADICTS the consensus. Understand why it lost —
   or whether it actually did.

Step 6 is the one that separates people. Kaplan→Chinchilla is exactly that shape, and Feinberg's critique of the distillation capacity gap (Claim 17) is him doing step 6 live.

Takeaway. All four asks share a property: they produce an artifact someone else can check. That is the entire signal. Not a credential, not a course certificate — a thing that exists, that works, that someone else can use.

Claim 21 — The internal-transfer play

For someone at a big company but outside the frontier team: don't chase the transfer. Ask "how do I help my product area adopt this technology as effectively as possible?" You become "the partner that we work with on the research side." He cites Nate Lintz, who transferred in and "owns so much of what we do... in terms of inference." And he adds that the transfer may not even be necessary, because integrating new technology into real products people use is itself cutting-edge work.

Why this works — the mechanism

Frontier research teams have a structural problem: they build things and need them adopted, but they don't understand your product's constraints, data, users, or latency budget. If you become the person who does understand both sides, you are not applying for a job — you are already doing half of it, and they already depend on you.

The playbook:

1. Become the LLM person for your product area. Actually deploy something.
2. Hit a real wall (latency, cost, quality, a serving limitation).
3. Bring the research team a PROBLEM WITH DATA — not a request.
   "Our p99 is 2.1s; here's the profile; the prefill is 80% of it."
4. Collaborate on the fix. Now you have a joint artifact and a joint author list.
5. Repeat. You are now the default partner for that surface.
6. The transfer, if you still want it, is a formality — they will ask you.

Step 3 is the whole thing. A profile and a number gets a researcher's attention. A request does not.

Takeaway. The shortest path into a frontier team usually runs through the product, not around it. And it is far less competitive than the front door.

Claim 22 — On AI replacing engineers

He calls the discourse "FUD everywhere, especially with some of the approach to marketing that some people have." His counter is accountability: "there's an element of making decisions around how we allocate these resources that will always be something that needs to be attributable to a human making that decision" — "you can't hand off blame to AI." The lawyer example: they remain necessary because "they can't be disbarred." And his prescription: "we all have agency over our future and we can start investing in skills that matter for tomorrow today."

The argument, made precise

It is not "AI is bad at this." It is a claim about institutions. Professions with liability — law, medicine, engineering sign-off, financial audit — are structured around a person who can be sanctioned. Licensure, malpractice, disbarment, fiduciary duty. A model cannot hold a license, cannot be sued, cannot be struck off. So even a perfectly capable model needs a human principal to attach accountability to.

The honest counterargument, which you should hold too: this bounds the floor, not the size. One accountable lawyer with excellent tools may replace ten. Accountability preserves the role, not the headcount.

Where that leaves you, practically:

  • The durable skills are judgment under uncertainty (Claim 2), verification (can you tell whether the output is right?), and accountability (will you sign your name to it?).
  • The compressible skills are pure translation — spec-to-code, paper-to-implementation.
  • His own field is a good example of the floor: nobody is going to let a model unilaterally spend $50M of TPU time. Someone signs.

Takeaway. He is not saying "relax." He is saying stop consuming the discourse and start compounding skill — which is, notably, the same thing he says about everything else.

Claim 23 — The career philosophy

Two pieces of advice, both anti-intuitive, both worth more than the technical content.

(1) Chase real problems, including the menial ones

"Chase the problems that people are facing in the world today. Go after the challenges that people see in everyday life, and don't be afraid to tackle a smaller part of this problem or maybe a more menial sounding part."

He is describing his own path. He was in pure research maximizing first-author papers at NeurIPS/ICML/ICLR. His manager Rohan Anil pushed him toward Bard. The work was hyperparameter tuning and getting a model to run on old TPUs — about as unglamorous as frontier AI gets. Jeff Dean gave him a spot bonus for it, and it put him on the path to leading Gemini pre-training.

The mechanism behind why this works:

  • Menial work sits close to reality, so it teaches you constraints nobody has written down.
  • It is under-supplied, because status-seekers avoid it, so your marginal value is high.
  • It makes you legible to the people who ship, which is where decisions are made.
  • And it seeds real problems: "quantizing the Ads DNN for pCTR serving efficiency" is not a glamorous line, and it is the direct intellectual ancestor of leading Flash.

(2) Be someone people want to see succeed

"Be the kind of co-worker that people would want to see succeed." Leverage other people's complementary skills "in ways that help them shine" — and then "people will notice, people will want to contribute to projects that you come up with in the future." He explicitly contrasts this with the "workplace psychopath" / Machiavellian model, and credits mentors like Todd Lipkin, who first got him into computer science, as the kind of person who "genuinely inspire[s] me to want to help them succeed."

The game-theoretic version, since it sounds soft but is not: careers are iterated games with reputation and information asymmetry. Zero-sum play works in one-shot games with observable payoffs. Neither condition holds in a career. What actually determines your opportunities is whether someone who already trusts you brings you in — and that is a pure reputation effect, accumulated over years.

Note also the Geng Yan story from Claim 12 in this light: a junior engineer's idea became the central technical bet of a flagship model, and the lead's contribution was to run "a very transparent technical process" around it. That is this philosophy operating at the level of technical decisions.

Takeaway. The technical content of this document is depreciating — architectures change. The two paragraphs above are not.

Master Takeaways

The whole document in twenty lines. If you internalize nothing else:

  1. C = 6ND. Derive it. It converts money into models.
  2. Pre-training is one-shot extrapolation. Every run is bigger than every run you have data for. This is why forecasting is a research field.
  3. A scaling law is a property of your recipe, not of nature. Change the recipe, refit.
  4. Loss forecasting is recipe selection. Compare fitted laws at the target FLOP count, not runs at small scale.
  5. Chinchilla beat Kaplan by fixing an experimental-design bug, not by having a better idea. Methodology is the frontier.
  6. Chinchilla-optimal is the wrong objective if you serve at scale. Optimize lifetime cost — but know that D_inf is unforecastable (Jevons, market expansion).
  7. MoE buys capacity with memory, communication, instability, and data hunger. Nothing is free; it just moves.
  8. The Flash 2.0 unlock was changing the sharding axis — layers instead of experts, at prefill — so communication hides behind compute.
  9. Prefill and decode are different machines. Compute-bound vs bandwidth-bound. Never apply one's optimization to the other.
  10. MFU of 35% is an accounting identity, not a failure. Read the breakdown; it is your agenda.
  11. Power is the bill. ~99% of TCO. A DRAM read costs ~20,000× an integer add. Quantization is an energy lever first and a memory lever second.
  12. Distillation is variance reduction; a better teacher reduces bias. The teacher's shape is the signal, not its argmax.
  13. Distillation's hard part is storage, not the loss. 10T tokens × 256k vocab is 5 EB. Top-k or online.
  14. Data is where the pre-training lead's time goes — half his year. Not architecture.
  15. D is not iid. L(N, U, R): unique tokens and repeats are different axes. ~4 epochs is nearly free; 32 is mostly waste.
  16. Synthetic data helps in the Stein's-paradox sense — biased but lower variance.
  17. Training runs need SREs. Goodput 78% vs 94% on identical hardware is checkpoint cadence and restart speed. That is days of a 40-day run.
  18. Research is an MDP. Buy information before outcomes. Write kill criteria in advance. Diversify across premises, not implementations.
  19. You can do real pre-training research on one GPU: kernels, vector quantization, test-time-compute trade-offs, and the statistics of scaling laws.
  20. The signal that gets you hired is an artifact someone else can check — and the career that compounds is built on doing unglamorous work well and making other people shine.

References

Primary sources for this document

Papers named in the talk or interview

  • Kaplan et al., Scaling Laws for Neural Language Models, 2020 — https://arxiv.org/abs/2001.08361
  • Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), 2022 — https://arxiv.org/abs/2203.15556
  • Sardana et al., Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws, 2024 — https://arxiv.org/abs/2401.00448
  • Muennighoff et al., Scaling Data-Constrained Language Models, 2023 — https://arxiv.org/abs/2305.16264
  • Busbridge et al., Distillation Scaling Laws, 2025 — https://arxiv.org/abs/2502.08606
  • Clark et al., Unified Scaling Laws for Routed Language Models, 2022 — https://arxiv.org/abs/2202.01169
  • Pope et al., Efficiently Scaling Transformer Inference, 2022 — https://arxiv.org/abs/2211.05102
  • Xu et al., GSPMD: General and Scalable Parallelization for ML Computation Graphs, 2021 — https://arxiv.org/abs/2105.04663
  • DeepSeek-AI et al., DeepSeek-V3 Technical Report, 2024 — https://arxiv.org/abs/2412.19437
  • Austin et al., How To Scale Your Model (The Scaling Book) — https://jax-ml.github.io/scaling-book/

Supporting references used in the explanations above

  • Hinton, Vinyals & Dean, Distilling the Knowledge in a Neural Network, 2015 — https://arxiv.org/abs/1503.02531
  • Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer, 2017 — https://arxiv.org/abs/1701.06538
  • Fedus, Zoph & Shazeer, Switch Transformers, 2021 — https://arxiv.org/abs/2101.03961
  • Dao et al., FlashAttention, 2022 — https://arxiv.org/abs/2205.14135
  • Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models, 2023 — https://arxiv.org/abs/2305.13245
  • Horowitz, Computing's Energy Problem (and what we can do about it), ISSCC 2014 — the canonical energy-per-operation table
  • Jacob Steinhardt, Research as a Stochastic Decision Process — https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
  • Grattafiori et al., The Llama 3 Herd of Models, 2024 — https://arxiv.org/abs/2407.21783