Phase 01 — Scaling Laws I: Kaplan → Chinchilla → IsoFLOPs

The phase where you learn to predict the result of an experiment you cannot afford to run twice. Feinberg's framing of why this discipline exists is the sharpest available: old ML iterated on CIFAR-10 before ImageNet and did LR searches "by doing multiple final runs" — "the last data point is our test set!" — whereas now "every single time you go up for a pre-training run, you're about to put in more FLOPs into this run than you've ever done before." Every next run requires extrapolation.

Why this phase exists

This is the intellectual centre of a pre-training lead's job. Everything else — MoE, sharding, distillation, quantization — is machinery in service of one decision: given C, what should we build? And that decision is made before the money is spent, on the strength of a fitted curve.

Three reasons this is genuinely hard, and not just curve-fitting:

1. You are extrapolating three orders of magnitude beyond your largest data point. Any functional form fits your ladder; they diverge wildly at the target.
2. The law describes your recipe, not nature. His slide is explicit: "Analysis made in the context of a parameterized LLM training recipe! Must already have architecture scaling, schedule defined for N, D." Change how depth scales with width, or how LR decays, and you must refit.
3. The most consequential result in the field came from fixing an experimental-design bug. Chinchilla beat Kaplan not with a better idea but by noticing that reading loss mid-run is a biased estimator. That is the lesson: methodology is the frontier.

And the payoff line from his slides, which is the whole reason to care: "Loss forecast implies model/recipe selection capability!" Forecasting is not reporting. It is the decision procedure.

Concept map

   ┌───────────────────────────────────────────────────────────────────┐
   │  A PARAMETERIZED RECIPE                                           │
   │  how depth/width/LR/batch/warmup all scale with N and D           │
   │  (without this fixed, a "scaling law" means nothing)              │
   └────────────────────────────┬──────────────────────────────────────┘
                                │
                                ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │  THE LADDER — many cheap runs                                     │
   │                                                                    │
   │   IsoFLOPs                      or        Parametric fit           │
   │   ─────────                                ──────────────          │
   │   fix C, sweep N                           collect (N, D, L)       │
   │   loss vs log N is a U                     fit L = E + A/N^a + B/D^b│
   │   fit a parabola, take the vertex          (Huber on log L)        │
   │   repeat for several C                                             │
   │   fit power laws  N*(C), D*(C)                                     │
   └────────────────────────────┬──────────────────────────────────────┘
                                │
                                ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │  EXTRAPOLATE to the flagship C, with a confidence interval        │
   └────────────────────────────┬──────────────────────────────────────┘
                                │
                                ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │  DECIDE:  compare baseline law vs candidate law AT THE TARGET C   │
   │  report the crossover point, not just the winner                   │
   └───────────────────────────────────────────────────────────────────┘

What you will be able to do

1. State the power-law form L = E + A/N^α + B/D^β and say what each of the five parameters means physically (E = irreducible entropy of the data; A/N^α = capacity limit; B/D^β = data limit).
2. Run an IsoFLOPs sweep: fix C, vary N, fit a parabola in log N, extract the vertex.
3. Fit power laws N_opt ∝ C^a, D_opt ∝ C^b and check a + b ≈ 1 (which it must, since C = 6ND).
4. Explain precisely why Kaplan got a ≈ 0.73 and Chinchilla got a ≈ 0.5 — the LR-decay bias — and demonstrate the bias numerically.
5. Fit the parametric form with a robust loss on log L, and show that least-squares on raw L gives different (worse) exponents.
6. Produce a forecast with a confidence interval, via bootstrap over the ladder points.
7. Compare a baseline recipe against a candidate recipe as fitted laws at the target C, and report the crossover.
8. Choose ladder design points to minimize extrapolation variance instead of filling a grid.

The lab

Success criteria. LAB_MODULE=solution pytest test_lab.py -v green; python solution.py reproduces the Kaplan→Chinchilla story end to end from synthetic data, including the moment where correcting the bias moves the recommended N by ~2×.

Deliverables checklist

• I can write L = E + A/N^α + B/D^β and explain all five parameters.
• I have run an IsoFLOPs sweep and extracted a vertex from a parabola fit.
• I can explain the Kaplan bug in one paragraph, and I have reproduced it numerically.
• I know why a + b ≈ 1 is a consistency check I should always run.
• I can produce a forecast with error bars and say what drives the width.
• I have compared two recipes at a target C and reported the crossover.
• I can justify where to place my next ladder run in terms of information gain.

Key takeaways

1. A scaling law is a property of a recipe. Fix the recipe first; the law describes that.
2. Loss forecasting is recipe selection. Compare fitted curves at the target, not runs at small scale.
3. The U-curve at fixed C is real and its bottom is flat. Flatness is a gift (you can move off the optimum for serving reasons cheaply) and a trap (noise moves the fitted vertex a lot).
4. a + b ≈ 1 falls out of C = 6ND. If your fitted exponents violate it, you have a bug.
5. Reading loss mid-run is biased, systematically against training on more data. This one methodological error set the industry's strategy for two years.
6. The estimator matters. Least-squares on L, least-squares on log L, Huber on log L, and MLE give different exponents on identical data — and therefore different flagship recommendations. Feinberg's own listed open problem: "Formalize."
7. Where you place ladder points matters more than how many you run. Spread beats density.