Warmup Guide — Scaling Laws I: Kaplan → Chinchilla → IsoFLOPs

How to read this. No statistics background is assumed. Power laws, log-log plots, curve fitting, robust losses, bootstrap confidence intervals and experimental design are all built from nothing. If you already know regression, start at Chapter 4 and do not skip Chapter 5 or Chapter 9.

Table of Contents

Chapter 1: What a Scaling Law Is

What it is

A scaling law is an empirical formula that predicts model quality from the resources you spend. In its most useful form:

$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} $$

  • L — the test loss (Chapter 3 explains what that number means)
  • N — parameters
  • D — training tokens
  • E, A, B, α, β — five constants you fit from a ladder of small experiments

Why it exists

Because of the one-shot problem. Feinberg's slides state the contrast exactly:

ML training before: "Maybe 2 stages; toy problem for iteration (CIFAR10) then you apply to Imagenet. LR searches by doing multiple 'final runs'. The last data point is our test set!"

Now: "every next run requires extrapolation."

You cannot try five flagship configurations and pick the winner. Each costs eight figures and months. So you build a predictor from cheap experiments and bet on its extrapolation.

How it works underneath — why this functional form?

The three terms are not arbitrary. Each corresponds to a distinct source of error:

TermNameMeaningWhat drives it to zero
Eirreducible lossThe entropy of language itself. Even a perfect model cannot predict the next token with certainty.Nothing. It is a floor.
A/N^αcapacity termYour model is too small to represent the true function.More parameters.
B/D^βdata termYou have not seen enough examples to find the right function.More tokens.

This decomposition is the classic approximation / estimation split from learning theory, written for transformers. And it makes a strong, checkable prediction: at fixed C, there is a single interior optimum, because pushing N up shrinks the capacity term but (via C = 6ND) shrinks D and grows the data term. Chapter 6 is that trade-off made visual.

Production significance

Everything. The flagship (N, D), the go/no-go, the choice between two architectures, the decision to buy more chips — all of it runs through a fitted law. Which is why Feinberg's line lands so hard: "Loss forecast implies model/recipe selection capability!"

The misconception

"Scaling laws are laws of nature, like F = ma."

They are not. They are fitted regressions over a specific recipe on a specific dataset. His slide says it twice: "These 'laws' are only empirical" and "The fitting of these laws depends a lot on the experimental setup as well as the implicit assumptions being made there." Change your data mixture, your optimizer, or how your architecture scales, and the constants move — and sometimes the exponents do too.

Chapter 2: Power Laws From Nothing

What a power law is

A relationship of the form y = c · x^k. That is it. The magic is what it looks like in logs:

$$ \log y = \log c + k \log x $$

A power law is a straight line on a log-log plot. The slope of that line is the exponent. That single fact is why every scaling-law paper's figures are log-log, and why you can eyeball an exponent from a plot.

import math

def power_law(x, c, k):
    return c * (x ** k)

# Confirm the straight-line property.
for x in (1e18, 1e19, 1e20, 1e21):
    y = power_law(x, c=0.6, k=0.5)
    print(f"log10(x)={math.log10(x):5.1f}  log10(y)={math.log10(y):7.3f}")

log10(x)= 18.0  log10(y)=  8.778
log10(x)= 19.0  log10(y)=  9.278
log10(x)= 20.0  log10(y)=  9.778
log10(x)= 21.0  log10(y)= 10.278

Every step of 1 in log x moves log y by exactly 0.5 — the exponent. Perfectly straight.

Fitting one: linear regression in log space

If your data follows a power law, take logs and fit a line. Ordinary least squares, in eight lines, no libraries:

def fit_line(xs, ys):
    """Least-squares slope and intercept. Returns (slope, intercept)."""
    n = len(xs)
    if n < 2:
        raise ValueError("need at least 2 points")
    mx = sum(xs) / n
    my = sum(ys) / n
    sxx = sum((x - mx) ** 2 for x in xs)
    if sxx == 0:
        raise ValueError("all x values identical — slope is undefined")
    sxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    slope = sxy / sxx
    return slope, my - slope * mx


def fit_power_law(xs, ys):
    """Fit y = c * x^k by regressing log y on log x. Returns (c, k)."""
    if any(x <= 0 or y <= 0 for x, y in zip(xs, ys)):
        raise ValueError("power-law fit needs strictly positive data")
    lx = [math.log(x) for x in xs]
    ly = [math.log(y) for y in ys]
    k, log_c = fit_line(lx, ly)
    return math.exp(log_c), k


xs = [1e18, 1e19, 1e20, 1e21, 1e22]
ys = [power_law(x, 0.6, 0.5) for x in xs]
c, k = fit_power_law(xs, ys)
print(f"recovered c={c:.4f}  k={k:.4f}")     # recovered c=0.6000  k=0.5000

Why power laws are everywhere in deep learning

Nobody has a fully satisfying answer, and it is an active research question. The best current intuitions:

  • Data manifold dimension. If natural language lies near a d-dimensional manifold, covering it to resolution ε needs ~ε^{-d} samples — which produces a power law with exponent related to 1/d.
  • Heavy-tailed feature frequency. Word and concept frequencies follow Zipf's law (itself a power law). Learning the n-th most common concept requires seeing it, and the marginal return of more data falls off as a power.
  • Random-feature / kernel arguments give power-law generalization curves under mild assumptions.

The honest position: the power-law form is an excellent empirical description over the ranges we have measured, with plausible but unproven theoretical motivation. Treat it as a very good interpolant, not a physical law. Chapter 13 is about what happens when it breaks.

The misconception

"A straight line on a log-log plot proves a power law."

Over a small range, almost anything looks straight in logs. You need several decades of x to distinguish a power law from a log, a stretched exponential, or a saturating curve. This matters enormously, because those alternatives disagree violently when extrapolated — which is exactly what you are about to do.

Chapter 3: What "Loss" Actually Measures

What it is

Language models are trained with cross-entropy loss: the negative log-probability the model assigned to the token that actually came next, averaged over tokens.

$$ L = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}) $$

def cross_entropy(probs_assigned_to_true_token):
    """Mean negative log-probability. Units: nats (natural log)."""
    if any(p <= 0 for p in probs_assigned_to_true_token):
        raise ValueError("zero probability on a true token gives infinite loss")
    return -sum(math.log(p) for p in probs_assigned_to_true_token) / len(
        probs_assigned_to_true_token)

print(round(cross_entropy([0.9, 0.8, 0.95, 0.7]), 4))   # 0.2113  (confident, correct)
print(round(cross_entropy([0.1, 0.2, 0.05, 0.3]), 4))   # 2.0369  (bad)

Why this and not accuracy

Three reasons, all of which matter for scaling work:

  1. It is smooth. Accuracy is a step function — the gradient is zero almost everywhere. Cross-entropy has a gradient everywhere.
  2. It is graded. A model that puts 0.49 on the right token is genuinely better than one that puts 0.01, even though both are "wrong." Accuracy cannot see that.
  3. It scales predictably. This is the empirical gift: cross-entropy follows clean power laws. Downstream benchmark accuracy does not — it plateaus, jumps, and saturates.

The units, and how to read them

Loss in nats (natural log). Two conversions worth knowing:

def nats_to_bits(nats):        return nats / math.log(2)
def loss_to_perplexity(nats):  return math.exp(nats)

for L in (2.0, 1.8, 1.7, 1.6):
    print(f"loss {L:.2f} nats = {nats_to_bits(L):.3f} bits/token, "
          f"perplexity {loss_to_perplexity(L):.2f}")

loss 2.00 nats = 2.885 bits/token, perplexity 7.39
loss 1.80 nats = 2.597 bits/token, perplexity 6.05
loss 1.70 nats = 2.453 bits/token, perplexity 5.47
loss 1.60 nats = 2.308 bits/token, perplexity 4.95

Perplexity is e^L — read it as "the model is as confused as if it were choosing uniformly among this many options."

The scale intuition you must build

Loss differences look tiny and are not. 0.01 nats is a meaningful, fought-over improvement at frontier scale. Why:

def flops_for_loss_delta(delta_nats, alpha=0.5):
    """Under L ~ C^-alpha near loss ~1.7, roughly how much more compute buys delta?"""
    # dL/dC = -alpha * (L - E) / C  =>  fractional compute increase ~ delta / (alpha*(L-E))
    L, E = 1.75, 1.69
    return delta_nats / (alpha * (L - E))

for d in (0.001, 0.01, 0.05):
    print(f"{d:5.3f} nats needs ~{flops_for_loss_delta(d):5.1%} more compute")

0.001 nats needs ~ 3.3% more compute
0.010 nats needs ~33.3% more compute
0.050 nats needs ~166.7% more compute

0.01 nats is roughly a third more compute. On a $30M run that is $10M. That is why teams argue about the third decimal place, and why your fit's confidence interval had better be narrower than the effect you are claiming.

The misconception

"Lower loss always means a better product."

Loss is measured on your held-out distribution. A model can improve loss by getting better at boilerplate that dominates the corpus while getting worse at the reasoning your users care about. This is why every serious team pairs the scaling law with downstream evals — and why Feinberg's slide on refinements mentions "Joint loss, eval fit." The law forecasts loss; loss is a proxy; the proxy needs auditing.

Chapter 4: The Kaplan Result, and What It Did to the Industry

The finding

Kaplan et al. (2020) established that transformer loss follows clean power laws in N, D, and C, and — the consequential part — derived the compute-optimal allocation. Feinberg's slide quotes their conclusion directly: data requirements grow "very slowly as D ∼ C^0.27 with training compute," and states their headline result:

"With a 10x compute budget, parameters should increase by 5.37x and the amount of data by 1.86x."

Decoding those numbers

def kaplan_allocation(compute_multiplier):
    """Kaplan: N grows as C^0.73, D as C^0.27."""
    return compute_multiplier ** 0.73, compute_multiplier ** 0.27

n_mult, d_mult = kaplan_allocation(10)
print(f"10x compute -> {n_mult:.2f}x params, {d_mult:.2f}x data")
# 10x compute -> 5.37x params, 1.86x data
print(f"consistency check (must be ~10): {n_mult * d_mult:.2f}")
# consistency check (must be ~10): 10.00

Note that consistency check — N_mult × D_mult must equal the compute multiplier, because C = 6ND. Equivalently, the exponents must sum to 1: 0.73 + 0.27 = 1.00. Always run this check on any pair of fitted exponents. If they do not sum to ~1, something is wrong with your fit or your FLOP accounting.

The industry consequence

His slide states it flatly:

"Consequences for the industry: We should heavily invest in scaling the model size rather than the data size!"

And that is exactly what happened. GPT-3: 175B parameters on ~300B tokens — a ratio of 1.7 tokens per parameter. Compare Chinchilla's later recommendation of ~20, or Llama 3's ~190.

for name, N, D in [("GPT-3 (2020)", 175e9, 300e9),
                   ("Gopher (2021)", 280e9, 300e9),
                   ("Chinchilla (2022)", 70e9, 1.4e12),
                   ("Llama-3-70B (2024)", 70e9, 15e12),
                   ("Llama-3-8B (2024)", 8e9, 15e12)]:
    print(f"{name:20s} {D/N:7.1f} tokens per parameter")

GPT-3 (2020)             1.7 tokens per parameter
Gopher (2021)            1.1 tokens per parameter
Chinchilla (2022)       20.0 tokens per parameter
Llama-3-70B (2024)     214.3 tokens per parameter
Llama-3-8B (2024)     1875.0 tokens per parameter

That table is the history of the field in one column. A 1000× swing in a single design parameter, driven first by a methodological error and then by serving economics (Phase 02).

The caveats Kaplan themselves flagged

Feinberg's slide lists them, and the point of listing them is that everyone ignored them:

  • "These 'laws' are only empirical"
  • "The fitting of these laws depends a lot on the experimental setup as well as the implicit assumptions being made there"

Tip. When a paper flags its own assumptions, that is where the next paper comes from. Chinchilla is literally the second bullet, taken seriously.

Chapter 5: The Chinchilla Correction — a Bug in Experimental Design

This is the most instructive story in the field. Feinberg's slide names the bug precisely:

"Kaplan et al. run a single training run per model size and uses intermediate losses to estimate the loss at different token horizon. ... This is a bad approximation as you can get much better losses through proper learning rate decay. Only the final loss value is optimal."

What a learning-rate schedule is (from zero)

The learning rate controls how big a step the optimizer takes. Too big and you bounce around the minimum forever; too small and you crawl. The standard solution is a schedule: warm up, hold high, then decay toward zero.

LR
 │      ╭──────────╮
 │     ╱            ╲
 │    ╱              ╲___
 │   ╱                    ╲___
 │  ╱                          ╲____
 │ ╱                                 ╲___
 └─────────────────────────────────────────► training steps
   warmup    high (explore)      decay (settle)

The decay phase is where a large chunk of the final loss improvement happens — the model stops oscillating and settles into the basin. A model mid-run has not had its decay yet.

The bug, made precise

Kaplan wanted L(N, D) for many D. Running a separate full experiment for each D is expensive. So: train once to a large D, and read the loss curve at intermediate points.

That is a biased estimator. At token count D' mid-run, the LR is still high, so the loss is worse than it would be for a run scheduled to end at D'.

def loss_curve(tokens_seen, horizon, base_a=3.0, base_b=0.1, decay_penalty=0.15):
    """Toy loss: an intrinsic power-law improvement plus a penalty for undecayed LR.

    The penalty vanishes only at the end of the SCHEDULE, not at a fixed token count.
    """
    if tokens_seen > horizon:
        raise ValueError("cannot read a curve past its horizon")
    intrinsic = base_a / (tokens_seen ** base_b)
    frac_remaining = 1.0 - tokens_seen / horizon
    return intrinsic + decay_penalty * frac_remaining


# The comparison that broke the field.
D_target = 100e9
kaplan_style = loss_curve(D_target, horizon=1000e9)   # peek at a long run mid-flight
chinchilla_style = loss_curve(D_target, horizon=D_target)  # a run scheduled to end here
print(f"Kaplan-style estimate at D=100B : {kaplan_style:.4f}")
print(f"Chinchilla-style (true)         : {chinchilla_style:.4f}")
print(f"bias                            : {kaplan_style - chinchilla_style:+.4f} nats")

Kaplan-style estimate at D=100B : 0.3733
Chinchilla-style (true)         : 0.2383
bias                            : +0.1350 nats

Why the bias is fatal and not merely noisy

Here is the structural point that makes this a real bug rather than extra noise. The bias is

penalty = decay_penalty × (1 − D_measured / D_schedule)

so it is not the same for every point:

  • Points read early in their schedule get a large penalty.
  • Points read at the end of their schedule get none.
def bias_across_horizons(decay_penalty):
    """Measure the SAME token count from runs with different schedule lengths."""
    D = 100e9
    for horizon_mult in (1, 2, 5, 10):
        biased = loss_curve(D, D * horizon_mult, decay_penalty=decay_penalty)
        truth = loss_curve(D, D, decay_penalty=decay_penalty)
        print(f"  schedule={horizon_mult:2d}x D -> measured {biased:.4f} "
              f"(bias {biased - truth:+.4f})")

print("With the schedule-mismatch bias present:")
bias_across_horizons(0.15)
print("With it removed (each run scheduled to its own horizon):")
bias_across_horizons(0.0)

With the schedule-mismatch bias present:
  schedule= 1x D -> measured 0.2383 (bias +0.0000)
  schedule= 2x D -> measured 0.3133 (bias +0.0750)
  schedule= 5x D -> measured 0.3583 (bias +0.1200)
  schedule=10x D -> measured 0.3733 (bias +0.1350)
With it removed (each run scheduled to its own horizon):
  schedule= 1x D -> measured 0.2383 (bias +0.0000)
  schedule= 2x D -> measured 0.2383 (bias +0.0000)
  schedule= 5x D -> measured 0.2383 (bias +0.0000)
  schedule=10x D -> measured 0.2383 (bias +0.0000)

Non-uniformity is the whole mechanism, and it is worth being precise about why. A uniform offset — "all our losses were 0.05 too high" — would simply be absorbed into the fitted E and would change no exponent and no recommendation. The lab has a dedicated test for exactly that control (test_uniform_offset_is_absorbed_into_E_and_changes_nothing). It is because the penalty varies systematically across the (N, D) grid that it tilts the fitted exponents, and the exponents are what determine the flagship recommendation.

On the direction of the tilt — a note on honesty. Which way the exponent moves depends on which ladder points the mismatch hits hardest, and that is a property of the specific experimental design. Reconstructing Kaplan's design precisely enough to derive the sign from first principles is beyond what the published record settles cleanly; what is documented is the empirical outcome — Chinchilla re-ran with properly matched schedules and found the optimum sits at substantially more tokens per parameter. The lab lets you inject the bias under two different designs (bias_mode="fixed_schedule" and "per_budget") and measure the tilt each one produces. That exercise — "I do not know the sign, so I will simulate the design and find out" — is the actual skill this chapter is teaching.

The result

Chinchilla's exponent came out at ~0.5 rather than ~0.73. Feinberg's slide:

"Chinchilla findings: the exponent in the power law is ~0.5, meaning model and data size should be scaled at the same rate! This is widely different from Kaplan et al."

And he labels the old regime on the plot with a single word: UNDERTRAINED!

"Consequences: Given a compute budget, models should be smaller and trained for longer. Kaplan's scaling laws meant that models were undertrained — which is obviously bad given bigger models are more expensive to serve and use downstream!"

That last clause is the bridge to the rest of this track. Chinchilla did not just lower the loss — it made the same quality available in a smaller model, which is cheaper to serve. That is the objective Feinberg's team optimizes, and Phase 02 pushes it further.

The empirical demonstration: Chinchilla (70B on 1.4T) beat Gopher (280B on 300B) at the same training compute — with a model 4× cheaper to serve.

Takeaway. The single most consequential result in scaling research was an experimental-design fix. Not an architecture, not an optimizer. Methodology is the frontier, and Feinberg's closing slide says the field still has open problems of exactly this kind (Chapter 9).

Chapter 6: The IsoFLOPs Method, Step by Step

Feinberg's slides walk through this as a build-up of six steps. Here is each one with the code.

The six steps

1. Fix a target FLOPs budget                     C = 1e20
2. Train a few models, vary model size           N = 100M, 300M, 1B, 3B (D = C/6N)
3. Fit a parabola and find the minimum           loss vs log N is U-shaped
4. Repeat 1-3 for various FLOPs budgets          C = 1e19, 1e20, 1e21, 1e22
5. Fit a power law: FLOPs -> optimal N           N_opt ∝ C^a
6. Fit a power law: FLOPs -> optimal D           D_opt ∝ C^b

"IsoFLOPs" means "equal FLOPs" — every model on one curve costs the same to train. You are asking: given this exact budget, what shape spends it best?

Step 1–2: the sweep

def true_loss(N, D, E=1.69, A=406.4, alpha=0.34, B=410.7, beta=0.28):
    """The Chinchilla parametric form, used here as a stand-in for reality."""
    return E + A / (N ** alpha) + B / (D ** beta)


def isoflop_sweep(C, param_counts):
    """Fix C, vary N, derive D from the budget. Returns [(N, D, loss), ...]."""
    out = []
    for N in param_counts:
        D = C / (6 * N)                      # the budget constraint
        out.append((N, D, true_loss(N, D)))
    return out


for N, D, L in isoflop_sweep(1e21, [1e8, 3e8, 1e9, 3e9, 1e10, 3e10]):
    print(f"N={N:8.1e}  D={D:8.1e}  D/N={D/N:8.1f}  loss={L:.4f}")

N= 1.0e+08  D= 1.7e+12  D/N= 16666.7  loss=2.6198
N= 3.0e+08  D= 5.6e+11  D/N=  1851.9  loss=2.4344
N= 1.0e+09  D= 1.7e+11  D/N=   166.7  loss=2.3400
N= 3.0e+09  D= 5.6e+10  D/N=    18.5  loss=2.3363
N= 1.0e+10  D= 1.7e+10  D/N=     1.7  loss=2.4160
N= 3.0e+10  D= 5.6e+09  D/N=     0.2  loss=2.5687

Why it is a U

  • Small N, huge D: oceans of data, no capacity to absorb it. The A/N^α term dominates.
  • Huge N, tiny D: enormous capacity, starved of examples. The B/D^β term dominates.
  • In between: balanced. A single interior minimum.
loss
 2.62 │●                                        
 2.57 │                                        ●
 2.43 │      ●                                  
 2.42 │                                  ●      
 2.34 │            ●     ●                      
      └──────────────────────────────────────────► log10(N)
        8.0   8.5   9.0   9.5  10.0  10.5
              underfit   OPT   data-starved

Step 3: fit the parabola

You have noisy points. You want the vertex. Fit a quadratic in log N and solve for its minimum analytically.

def fit_quadratic(xs, ys):
    """Least-squares fit of y = a*x^2 + b*x + c. Solves the 3x3 normal equations
    with Cramer's rule — no libraries, fully deterministic."""
    if len(xs) < 3:
        raise ValueError("need at least 3 points for a parabola")
    n = len(xs)
    s = [sum(x ** p for x in xs) for p in range(5)]        # s[0..4]
    t = [sum(y * x ** p for x, y in zip(xs, ys)) for p in range(3)]

    M = [[s[4], s[3], s[2]],
         [s[3], s[2], s[1]],
         [s[2], s[1], s[0]]]
    rhs = [t[2], t[1], t[0]]

    def det3(m):
        return (m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
                - m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
                + m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]))

    D0 = det3(M)
    if abs(D0) < 1e-30:
        raise ValueError("degenerate design — points are collinear in x")

    def replace_col(m, col, v):
        return [[v[r] if c == col else m[r][c] for c in range(3)] for r in range(3)]

    a = det3(replace_col(M, 0, rhs)) / D0
    b = det3(replace_col(M, 1, rhs)) / D0
    c = det3(replace_col(M, 2, rhs)) / D0
    return a, b, c


def parabola_vertex(a, b):
    """x at the minimum of a*x^2 + b*x + c. Requires a > 0 (a real minimum)."""
    if a <= 0:
        raise ValueError("parabola opens downward or is flat — no interior minimum")
    return -b / (2 * a)


pts = isoflop_sweep(1e21, [1e8, 3e8, 1e9, 3e9, 1e10, 3e10])
log_n = [math.log10(N) for N, _, _ in pts]
losses = [L for _, _, L in pts]
a, b, c = fit_quadratic(log_n, losses)
n_opt = 10 ** parabola_vertex(a, b)
print(f"fitted optimal N = {n_opt:.3e}  (D = {1e21/(6*n_opt):.3e}, "
      f"ratio {1e21/(6*n_opt)/n_opt:.1f} tokens/param)")

fitted optimal N = 1.951e+09  (D = 8.542e+10, ratio 43.8 tokens/param)

Notice the flat bottom. In the sweep above, N=1e9 and N=3e9 differed by 0.004 nats — a factor of 3 in model size for essentially no quality difference. This is a gift: you can move off the exact optimum toward a smaller, cheaper-to-serve model almost for free (Phase 02's whole thesis). It is also a trap: with realistic measurement noise, the fitted vertex wanders. Chapter 10 quantifies how much.

Steps 4–6: the power laws

def run_isoflops_ladder(budgets, sizes_per_budget):
    """Steps 1-4: get (C, N_opt, D_opt) for several budgets."""
    results = []
    for C, sizes in zip(budgets, sizes_per_budget):
        pts = isoflop_sweep(C, sizes)
        lx = [math.log10(N) for N, _, _ in pts]
        ly = [L for _, _, L in pts]
        a, b, _ = fit_quadratic(lx, ly)
        N_opt = 10 ** parabola_vertex(a, b)
        results.append((C, N_opt, C / (6 * N_opt)))
    return results


budgets = [1e19, 1e20, 1e21, 1e22]
sizes = [[N * m for N in (1e7, 3e7, 1e8, 3e8, 1e9)] for m in (1, 3, 10, 30)]
ladder = run_isoflops_ladder(budgets, sizes)

# Steps 5-6: fit the power laws.
Cs = [C for C, _, _ in ladder]
Ns = [N for _, N, _ in ladder]
Ds = [D for _, _, D in ladder]
_, a_exp = fit_power_law(Cs, Ns)
_, b_exp = fit_power_law(Cs, Ds)
print(f"N_opt ∝ C^{a_exp:.3f}")
print(f"D_opt ∝ C^{b_exp:.3f}")
print(f"consistency: a + b = {a_exp + b_exp:.3f}  (must be 1.000)")

N_opt ∝ C^0.454
D_opt ∝ C^0.546
consistency: a + b = 1.000

a + b = 1 exactly, and it must — it falls straight out of C = 6ND. Run this check every single time. If your fitted exponents do not sum to ~1, you have a bug in your FLOP accounting or your fit, and you should find it before you spend $30M.

And the exponent, ~0.45, is close to Chinchilla's ~0.5: scale N and D at roughly the same rate. (It is not exactly 0.5 because the ground-truth α and β used here are not equal — optimum_exponent(α, β) = β/(α+β), which is 0.5 precisely when α = β.)

Chapter 7: The Parametric Form L(N, D)

IsoFLOPs finds the optimum without ever writing down a formula. The alternative — and what you need if you want to answer "what if I have 3× the data but the same compute?" — is to fit the full surface.

$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} $$

Reading the five parameters

CHINCHILLA = dict(E=1.69, A=406.4, alpha=0.34, B=410.7, beta=0.28)

def parametric_loss(N, D, E, A, alpha, B, beta):
    if N <= 0 or D <= 0:
        raise ValueError("N and D must be positive")
    return E + A / (N ** alpha) + B / (D ** beta)

# Decompose the loss at a realistic operating point.
N, D = 70e9, 1.4e12
p = CHINCHILLA
cap = p["A"] / N ** p["alpha"]
dat = p["B"] / D ** p["beta"]
print(f"irreducible E      : {p['E']:.4f}  ({p['E']/(p['E']+cap+dat):5.1%})")
print(f"capacity   A/N^a   : {cap:.4f}  ({cap/(p['E']+cap+dat):5.1%})")
print(f"data       B/D^b   : {dat:.4f}  ({dat/(p['E']+cap+dat):5.1%})")
print(f"total              : {parametric_loss(N, D, **p):.4f}")

irreducible E      : 1.6900  (87.3%)
capacity   A/N^a   : 0.0835  ( 4.3%)
data       B/D^b   : 0.1632  ( 8.4%)
total              : 1.9366

Look at that: 87% of the loss is irreducible. All the money in the industry is being spent on the remaining 13%. That is worth internalizing — it explains why improvements look so small in nats, and why compressing a 0.01-nat gap is worth a nine-figure budget.

It also tells you which lever to pull: here the data term (0.163) is nearly twice the capacity term (0.084), so at this operating point more data helps more than more parameters — which is precisely the Chinchilla recommendation, visible directly in the decomposition.

Deriving the optimum analytically

With the parametric form you can solve for the optimum instead of sweeping. Minimize L(N, C/(6N)) over N:

$$ \frac{\partial}{\partial N}\left[\frac{A}{N^\alpha} + \frac{B}{(C/6N)^\beta}\right] = 0 $$

$$ \Rightarrow \quad -\alpha A N^{-\alpha-1} + \beta B (6/C)^{\beta} N^{\beta - 1} = 0 $$

$$ \Rightarrow \quad N_{\text{opt}} = \left[\frac{\alpha A}{\beta B}\left(\frac{C}{6}\right)^{\beta}\right]^{\frac{1}{\alpha+\beta}} $$

which is a power law in C with exponent β/(α+β).

def analytic_optimum(C, E, A, alpha, B, beta):
    """Closed-form compute-optimal (N, D) under the parametric law."""
    N = ((alpha * A) / (beta * B) * (C / 6.0) ** beta) ** (1.0 / (alpha + beta))
    return N, C / (6 * N)

for C in (1e19, 1e21, 1e23, 1e25):
    N, D = analytic_optimum(C, **CHINCHILLA)
    print(f"C={C:.0e}  N={N:9.3e}  D={D:9.3e}  ratio={D/N:7.1f} tok/param")

exponent = CHINCHILLA["beta"] / (CHINCHILLA["alpha"] + CHINCHILLA["beta"])
print(f"\nN_opt ∝ C^{exponent:.4f}")

C=1e+19  N=2.280e+08  D=7.311e+09  ratio=   32.1 tok/param
C=1e+21  N=1.824e+09  D=9.136e+10  ratio=   50.1 tok/param
C=1e+23  N=1.460e+10  D=1.142e+12  ratio=   78.2 tok/param
C=1e+25  N=1.168e+11  D=1.427e+13  ratio=  122.1 tok/param

N_opt ∝ C^0.4516

A caution worth stating. Notice that the token-per-parameter ratio is not constant — it climbs from 32 at 1e19 to 122 at 1e25. The familiar "20 tokens per parameter" is only a rule of thumb at one scale; the parametric law says the ratio drifts whenever α ≠ β. Note also that these particular E/A/B values have been debated in follow-up work (see Besiroglu et al. in the References), and the ratio is extremely sensitive to α and β. This is not a flaw in the method — it is the lesson. Small changes in fitted exponents produce large changes in the recommendation, which is exactly why Chapter 9 (the estimator problem) and Chapter 10 (confidence intervals) exist. When you fit your own law, always report the ratio and its uncertainty.

Chapter 8: Fitting It — Least Squares, Logs, and Huber

You have (N, D, L) points and want (E, A, α, B, β). This is a nonlinear fit, and how you measure "fit" changes the answer.

The objective choices

def residuals(points, params, space="log"):
    """points: [(N, D, L_observed), ...]; params: (E, A, alpha, B, beta)."""
    E, A, alpha, B, beta = params
    out = []
    for N, D, L_obs in points:
        L_pred = E + A / N ** alpha + B / D ** beta
        if space == "linear":
            out.append(L_pred - L_obs)
        elif space == "log":
            if L_pred <= 0 or L_obs <= 0:
                raise ValueError("log-space residuals need positive losses")
            out.append(math.log(L_pred) - math.log(L_obs))
        else:
            raise ValueError(f"unknown space {space!r}")
    return out


def squared_loss(rs):
    return sum(r * r for r in rs)


def huber_loss(rs, delta=1e-3):
    """Quadratic near zero, LINEAR in the tails — so one bad run cannot dominate.

    Chinchilla used Huber on log-space residuals. This is not incidental: scaling
    ladders contain genuine outliers (a run that diverged, a bad data shard), and
    least squares would let one of them set your flagship recommendation.
    """
    total = 0.0
    for r in rs:
        if abs(r) <= delta:
            total += 0.5 * r * r
        else:
            total += delta * (abs(r) - 0.5 * delta)
    return total

Why the space matters

Losses in a ladder span a wide range — a 10M-parameter model might sit at 4.0 nats, a 10B model at 1.9.

  • Least squares on raw L weights absolute error. A 0.1-nat miss on the 4.0 point counts the same as a 0.1-nat miss on the 1.9 point. Since the big-loss points are the small models, the fit gets dragged toward matching small models well — exactly the points you care least about, since you are extrapolating upward.
  • Least squares on log L weights relative error, treating all scales evenly.
  • Huber on log L does that and refuses to let one outlier dominate.
def compare_objectives(points, params):
    lin = residuals(points, params, "linear")
    log = residuals(points, params, "log")
    print(f"  sum-sq on L      : {squared_loss(lin):.6f}")
    print(f"  sum-sq on log L  : {squared_loss(log):.6f}")
    print(f"  Huber on log L   : {huber_loss(log):.6f}")

pts = [(N, D, true_loss(N, D)) for N, D in
       [(1e7, 1e10), (1e8, 1e11), (1e9, 1e11), (1e10, 1e12)]]
# Inject one outlier: a run that diverged and reported a bad loss.
pts_outlier = pts[:-1] + [(1e10, 1e12, pts[-1][2] + 1.5)]

print("clean ladder, true params:")
compare_objectives(pts, (1.69, 406.4, 0.34, 410.7, 0.28))
print("ladder with ONE diverged run, same params:")
compare_objectives(pts_outlier, (1.69, 406.4, 0.34, 410.7, 0.28))

clean ladder, true params:
  sum-sq on L      : 0.000000
  sum-sq on log L  : 0.000000
  Huber on log L   : 0.000000
ladder with ONE diverged run, same params:
  sum-sq on L      : 2.250000
  sum-sq on log L  : 0.305852
  Huber on log L   : 0.000553

One bad run contributes 2.25 to the least-squares objective and 0.00055 to Huber — a factor of about 4,000. Under least squares the fitter will distort all five parameters to chase that outlier. Under Huber it barely notices. On a real ladder, where a diverged run is a routine occurrence, this is the difference between a usable law and a garbage one.

Fitting without a library

You need a nonlinear optimizer. Coordinate descent with a shrinking step is enough, fully deterministic, and about twenty lines:

def fit_parametric(points, init=(1.5, 400.0, 0.35, 400.0, 0.30),
                   objective="huber", space="log",
                   iters=200, seed_step=0.5, shrink=0.97):
    """Deterministic coordinate descent. No randomness, no library optimizer,
    so the same input always gives the same fit — which is a testable property."""
    params = list(init)
    step = [seed_step * abs(p) if p != 0 else seed_step for p in params]

    def score(p):
        rs = residuals(points, tuple(p), space)
        return huber_loss(rs) if objective == "huber" else squared_loss(rs)

    best = score(params)
    for _ in range(iters):
        improved = False
        for i in range(len(params)):
            for direction in (+1, -1):
                trial = list(params)
                trial[i] = params[i] + direction * step[i]
                # Keep exponents and scales in a sane region.
                if trial[i] <= 0:
                    continue
                s = score(trial)
                if s < best:
                    best, params, improved = s, trial, True
                    break
        if not improved:
            step = [s * shrink for s in step]
    return tuple(params), best

Why hand-rolled? Because a library optimizer's version, tolerance, and default method all change the fitted exponents, and therefore your flagship recommendation. If a scipy upgrade can move your N_opt by 20%, your process is not reproducible. Frontier teams pin this ruthlessly.

Chapter 9: The Estimator Problem — Feinberg's Open Question

His closing slide lists this as a research direction, and it is worth quoting in full because it is a rare, concrete, fundable, GPU-free open problem stated by someone who would know:

"Scaling laws are brittle, dataset dependent. L(N, D, etc.) – of course we can add more dims to improve fit. Least squares vs MLE & formal stats model imply different scaling recommendations! Formalize. Rather than grid (N, D) where do we get max info gain? Active learn…"

The two problems

Problem A — the estimator is unspecified. Chinchilla used Huber on log-residuals. That is a choice, not a derivation. A proper statistical treatment would:

  1. Write down a generative model: what is the noise on a measured loss? Is it additive? Multiplicative? Heteroscedastic (bigger for small models, which are noisier)? Correlated across points from the same run?
  2. Derive the maximum-likelihood estimator under that model.
  3. Get calibrated uncertainty for free, instead of bootstrapping.

Nobody has fully done this, and the choice demonstrably changes the recommendation:

def recommendation_under_estimator(points, objective, space):
    params, _ = fit_parametric(points, objective=objective, space=space)
    N_opt, D_opt = analytic_optimum(1e24, *params)
    return params, N_opt, D_opt

ladder = [(N, D, true_loss(N, D)) for N, D in
          [(1e7, 2e10), (3e7, 6e10), (1e8, 2e11), (3e8, 6e11), (1e9, 2e12)]]

for obj, sp in (("squared", "linear"), ("squared", "log"), ("huber", "log")):
    params, N_opt, D_opt = recommendation_under_estimator(ladder, obj, sp)
    print(f"{obj:8s}/{sp:6s} -> flagship N = {N_opt:.3e}, D = {D_opt:.3e}")

Run it and watch the recommendations diverge on identical data. That divergence is the open problem.

Problem B — the design is a grid. Everybody runs N ∈ {…} × C ∈ {…}. But a grid is not an efficient experiment. The right question is: given my current posterior, which next run most reduces the variance of my prediction at C = 1e25?

def extrapolation_variance(log_c_points, target_log_c):
    """Variance of a linear extrapolation to `target_log_c`, up to a noise constant.

    Standard OLS result:  Var(y_hat) ∝ 1/n + (x* - x̄)^2 / Σ(x - x̄)^2
    """
    n = len(log_c_points)
    if n < 2:
        raise ValueError("need at least 2 design points")
    xbar = sum(log_c_points) / n
    sxx = sum((x - xbar) ** 2 for x in log_c_points)
    if sxx == 0:
        raise ValueError("all design points identical")
    return 1.0 / n + (target_log_c - xbar) ** 2 / sxx


TARGET = 25.0     # forecasting a 1e25 FLOP run
designs = {
    "clustered (4 runs, all ~1e19)": [19.0, 19.2, 19.4, 19.6],
    "spread    (4 runs, 1e18-1e21)": [18.0, 19.0, 20.0, 21.0],
    "budget-skewed (3 small, 1 big)": [18.0, 18.5, 19.0, 21.5],
    "two-point extremes":            [18.0, 21.0],
}
for name, d in designs.items():
    print(f"{name:32s} Var ∝ {extrapolation_variance(d, TARGET):8.2f}")

clustered (4 runs, all ~1e19)     Var ∝   162.70
spread    (4 runs, 1e18-1e21)     Var ∝     6.30
budget-skewed (3 small, 1 big)    Var ∝     4.81
two-point extremes               Var ∝     7.22

The clustered design is 26× worse than the spread one, using the same number of runs. And note the last row: two-point extremes, with half the runs, beats four clustered runs by 23×. Note also that the budget-skewed design — three cheap points plus one expensive one far out — edges out the evenly spread one, because that distant anchor is doing most of the work.

The practical rule this gives you: when planning a ladder, maximize the spread of log C, subject to your budget and to the smallest model still being large enough to be in the scaling regime. Do not add a fifth point near your existing four; add one an order of magnitude out.

Chapter 10: Confidence Intervals via Bootstrap

A forecast without error bars is not a forecast. And you cannot derive analytic intervals here because the model is nonlinear and the noise model is unknown (Chapter 9). So: bootstrap.

What the bootstrap is

You have n ladder points. Resample n of them with replacement, refit, record the prediction. Do it 1,000 times. The spread of those 1,000 predictions estimates the spread of your prediction.

The logic: your sample is your best available picture of the population, so resampling from it mimics drawing fresh samples from the population.

import random

def bootstrap_forecast(points, target_C, n_boot=200, seed=0):
    """Bootstrap CI for the loss forecast at target_C. Seeded -> reproducible."""
    rng = random.Random(seed)              # SEEDED. Same seed -> same bytes.
    forecasts = []
    for _ in range(n_boot):
        sample = [points[rng.randrange(len(points))] for _ in range(len(points))]
        try:
            params, _ = fit_parametric(sample, iters=60)
            N, D = analytic_optimum(target_C, *params)
            forecasts.append(parametric_loss(N, D, *params))
        except (ValueError, OverflowError, ZeroDivisionError):
            continue                       # a degenerate resample; skip it
    if len(forecasts) < 10:
        raise ValueError("too few successful bootstrap fits — ladder is too small")
    forecasts.sort()
    lo = forecasts[int(0.025 * len(forecasts))]
    hi = forecasts[int(0.975 * len(forecasts))]
    mid = forecasts[len(forecasts) // 2]
    return {"median": mid, "ci_low": lo, "ci_high": hi, "width": hi - lo,
            "n_successful": len(forecasts)}

How to read the width

Remember Chapter 3: 0.01 nats is roughly a third more compute. So:

CI widthVerdict
< 0.01 natsTight. You can make a confident recipe call.
0.01–0.05Usable for go/no-go, not for choosing between similar recipes.
> 0.05Your ladder cannot support this decision. Run more/wider points before the meeting.

The senior move. When someone presents a scaling forecast without an interval, ask for one. When they present an interval wider than the effect they are claiming, the honest conclusion is "we do not know yet" — and saying so is far more valuable than a confident wrong number that gets baked into a $40M plan.

Chapter 11: Designing the Ladder — Where to Spend Your Ablations

Pulling Chapters 9 and 10 into a procedure.

The constraints

  1. Budget. The ladder should cost a small fraction of the flagship — a few percent is typical, and Phase 00's $ per 1e21 FLOPs makes that concrete.
  2. The scaling regime. Below ~10M non-embedding parameters, models behave differently (embeddings dominate, optimization is qualitatively different). Points below the regime are worse than useless — they actively bias the fit.
  3. Spread. Chapter 9: variance is driven by the spread of log C.
  4. Replication. At least one budget should be run twice to measure your noise level. Without that, you are guessing at the thing your confidence interval depends on.

A workable recipe

def design_ladder(flagship_C, ladder_budget_fraction=0.03, n_budgets=5,
                  min_C=1e18):
    """Geometrically spaced budgets consuming a fixed fraction of the flagship."""
    total = flagship_C * ladder_budget_fraction
    # Geometric spacing means each budget is the same multiplicative step apart,
    # which maximizes spread in log space for a given range.
    top = total / 2.0            # the largest ladder run gets half the ladder budget
    ratio = (top / min_C) ** (1.0 / (n_budgets - 1))
    budgets = [min_C * ratio ** i for i in range(n_budgets)]
    return budgets

flagship = 1e25
budgets = design_ladder(flagship)
print(f"flagship: {flagship:.0e} FLOPs")
print(f"ladder total: {sum(budgets):.3e} FLOPs "
      f"({sum(budgets)/flagship:.2%} of flagship)")
for C in budgets:
    print(f"  C={C:.3e}   ({C/flagship:.2e} of flagship)")
print(f"log-C spread: {math.log10(budgets[-1]) - math.log10(budgets[0]):.1f} decades")

flagship: 1e+25 FLOPs
ladder total: 1.580e+23 FLOPs (1.58% of flagship)
  C=1.000e+18   (1.00e-07 of flagship)
  C=1.968e+19   (1.97e-06 of flagship)
  C=3.873e+20   (3.87e-05 of flagship)
  C=7.622e+21   (7.62e-04 of flagship)
  C=1.500e+23   (1.50e-02 of flagship)
log-C spread: 5.2 decades

Under 2% of the flagship budget buys you 5.2 decades of spread. In money (Phase 00's $1,753 per 1e21 FLOPs): the flagship is ~$17.5M and the entire ladder is ~$277k. That is the single best-value spend in the whole project, and it is the argument you make when someone suggests skipping ablations to save time.

Chapter 12: Making the Decision — Baseline vs Candidate

Feinberg's slide gives the procedure in three lines:

"How changes get adopted in classical setting:

  1. Derive L*(flops) baseline
  2. L*(flops) candidate"

and, on the refinements slide: "To make a change, compare baseline vs candidate laws."

Why you compare laws, not runs

The naive approach — "train both at 1B parameters, pick the winner" — fails because the curves can cross. An architecture change that helps small models can hurt large ones, and vice versa. Comparing at one scale tells you about that scale only.

def compare_recipes(baseline_params, candidate_params, target_C):
    """Compare two fitted laws AT THE TARGET, and find the crossover."""
    def loss_at(params, C):
        N, D = analytic_optimum(C, *params)
        return parametric_loss(N, D, *params)

    base = loss_at(baseline_params, target_C)
    cand = loss_at(candidate_params, target_C)

    # Scan for a crossover on a log grid.
    crossover = None
    prev = None
    for i in range(100):
        C = 10 ** (16 + i * 0.12)
        sign = loss_at(candidate_params, C) < loss_at(baseline_params, C)
        if prev is not None and sign != prev:
            crossover = C
            break
        prev = sign

    return {"baseline_loss": base, "candidate_loss": cand,
            "delta": cand - base, "candidate_wins": cand < base,
            "crossover_C": crossover}


baseline = (1.69, 406.4, 0.34, 410.7, 0.28)
# A candidate that is better per-parameter (bigger A-exponent) but worse per-token.
candidate = (1.69, 500.0, 0.38, 380.0, 0.26)

for C in (1e20, 1e22, 1e24, 1e26):
    r = compare_recipes(baseline, candidate, C)
    verdict = "CANDIDATE" if r["candidate_wins"] else "baseline "
    print(f"C={C:.0e}: base {r['baseline_loss']:.4f}  "
          f"cand {r['candidate_loss']:.4f}  delta {r['delta']:+.4f}  -> {verdict}")
r = compare_recipes(baseline, candidate, 1e24)
print(f"\ncrossover at C ≈ {r['crossover_C']:.2e}" if r["crossover_C"]
      else "\nno crossover in range")

The crossover is the deliverable, not the winner. "Candidate wins above 1e23 FLOPs" is an actionable statement; "candidate is better" is not, because it silently assumes a scale.

The decision checklist

Before you recommend a recipe change, you should be able to answer all six:

  • Have both laws been fitted on the same ladder design and the same estimator?
  • Is the delta at the target larger than the confidence interval on either forecast?
  • Where is the crossover, and is the flagship comfortably on one side of it?
  • Does the candidate change the serving cost (Phase 02, Phase 05)? A 0.01-nat win that doubles inference cost is a loss.
  • Does the candidate change the failure modes (Phase 09)? A recipe that is 0.005 nats better and spikes twice as often will cost you more in goodput than it gains.
  • Do the downstream evals agree with the loss delta, or is the loss improvement coming from something users do not care about?

Chapter 13: When Scaling Laws Lie

Feinberg's slide "The End of Scaling?" is measured about this, and worth taking seriously in both directions.

The failure modes

1. Out-of-regime extrapolation. Every law is fitted over a range. Three decades beyond it, the functional form itself may be wrong — a power law and a saturating curve can be indistinguishable on your ladder and disagree by a lot at the target.

2. Recipe drift. The law describes the recipe you fitted. If between the ladder and the flagship you change the data mixture, the tokenizer, the optimizer, or how depth scales — the law no longer applies. This is a shockingly common real-world failure: the ladder was run in March, the flagship in July, and the data team shipped three improvements in between.

3. Data exhaustion. The law says "more tokens." If you do not have them, the law is answering a question you cannot act on. This is Phase 02's L(N, U, R).

4. Benchmark saturation, not model saturation. His slide: "LMSys is not the end-all-be-all. Llama 4 Maverick demonstrated that ranking can be volatile and overfit to human preference." A flat benchmark can mean a flat model or a dead benchmark. They look identical.

His counterargument to the doomers

He does not conclude scaling is over. The slide lists two reasons to expect continued progress:

"1. Better NN design still coming 2. Data from new sources being added"

and the whole "More Data Sources" slide is about multimodal and synthetic data (Phase 02). His framing of the mission is worth keeping: "job is to push the curves right."

The practical posture

def extrapolation_risk(ladder_max_C, target_C, ci_width):
    """A crude, honest risk score for a forecast."""
    decades = math.log10(target_C / ladder_max_C)
    if decades <= 0:
        return "INTERPOLATION — low risk"
    risk = decades * (1 + 20 * ci_width)
    band = ("LOW" if risk < 2 else "MODERATE" if risk < 4 else "HIGH")
    return (f"{decades:.1f} decades of extrapolation, CI {ci_width:.3f} nats "
            f"-> {band} risk (score {risk:.1f})")

print(extrapolation_risk(1e22, 1e25, 0.008))
print(extrapolation_risk(1e20, 1e25, 0.040))

3.0 decades of extrapolation, CI 0.008 nats -> MODERATE risk (score 3.5)
5.0 decades of extrapolation, CI 0.040 nats -> HIGH risk (score 9.0)

Present it this way. "We forecast 1.712 nats, 95% CI [1.704, 1.721], extrapolating 3 decades beyond our largest ablation, assuming the data mixture is frozen from today." That sentence is what a principal engineer says. Everything before the comma is arithmetic; everything after it is judgment, and the judgment is what you are paid for.

Lab Walkthrough

Lab 01 — IsoFLOPs Ladder, Law Fitting & the Forecast

Implement in this order:

  1. fit_line, fit_power_law, fit_quadratic, parabola_vertex — the numerical spine. fit_quadratic uses Cramer's rule on the 3×3 normal equations; it must raise on a degenerate (collinear) design rather than divide by zero.
  2. synthetic_ladder — generate (N, D, L) with seeded noise and an injectable lr_decay_bias. This is what lets you reproduce the Kaplan bug.
  3. isoflop_optimum — sweep, fit the parabola, return the vertex.
  4. fit_scaling_exponents — steps 5–6, plus the a + b ≈ 1 consistency check.
  5. residuals / huber_loss / fit_parametric — the surface fit. Deterministic coordinate descent; same input must give byte-identical output.
  6. analytic_optimum — the closed form. A test checks it agrees with the numerical sweep.
  7. bootstrap_forecast — seeded resampling, percentile interval.
  8. extrapolation_variance / design_ladder — the optimal-design tooling.
  9. compare_recipes — the decision rule with the crossover scan.

The money test is test_lr_decay_bias_shifts_recommendation_toward_smaller_models: generate a ladder with the bias, fit, and confirm the recommended N is meaningfully larger than the unbiased fit's. That is Kaplan → Chinchilla, reproduced from scratch.

The traps:

  • fit_quadratic on 2 points, or on collinear points → must raise, not ZeroDivisionError.
  • A parabola that opens downward has no minimum → must raise.
  • Bootstrap resamples can be degenerate (all identical points) → catch and skip, and fail loudly if too few succeed.
  • Everything random must go through a seeded random.Random(seed).

Success Criteria

  • LAB_MODULE=solution pytest test_lab.py -v all green.
  • Your lab.py passes after the TODOs.
  • python solution.py reproduces the Kaplan → Chinchilla story from synthetic data.
  • You can explain, without notes, why reading loss mid-run is a biased estimator and why the bias has a direction.
  • You can state why a + b ≈ 1 must hold and what it catches.
  • You can produce a forecast with a CI and say whether it is tight enough to decide on.
  • You can justify a ladder design in terms of extrapolation variance and cost.

Interview Q&A

Q: What is a scaling law and why does the field need them? An empirical formula, typically L = E + A/N^α + B/D^β, predicting test loss from parameters and tokens. The field needs them because pre-training is a one-shot extrapolation problem: each flagship run costs eight figures, takes months, and is by construction larger than anything previously run, so you cannot validate by trying alternatives. You fit the law on a cheap ladder and extrapolate. Critically, the law is a property of a fixed, parameterized recipe, not of nature.

Q: Kaplan said N ∝ C^0.73; Chinchilla said C^0.5. What happened? A methodological bug. Kaplan ran one training run per model size and read intermediate losses to estimate loss at smaller token horizons. But learning-rate schedules decay to near zero at the end, and much of the final improvement comes from that decay. A mid-run reading is therefore systematically worse than a run actually scheduled to stop there — and the bias is largest for the points furthest from their horizon, which tilts the fit against training on more data. Chinchilla ran separate, properly-scheduled runs, found the exponent was ~0.5, and concluded models had been badly undertrained.

Q: Walk me through IsoFLOPs. Fix a FLOP budget. Train several models at different N, deriving D = C/6N so every run costs the same. Loss versus log N is U-shaped — too small underfits, too large is data-starved — so fit a parabola and take the vertex as N_opt for that budget. Repeat across budgets, then fit power laws N_opt ∝ C^a and D_opt ∝ C^b. Sanity check: a + b must be ~1, since C = 6ND.

Q: How do you fit the parametric form, and why does it matter how? Minimize a robust loss — Huber — on log-space residuals. Log space because losses span a wide range and you care about relative not absolute error, and because absolute error would let the small, high-loss models dominate the fit exactly when you are extrapolating upward. Huber because ladders contain genuine outliers — a diverged run, a bad shard — and under least squares one of those can move all five parameters. It matters because different estimators give different exponents on identical data, and therefore different flagship recommendations. That is an open problem Feinberg explicitly calls out: "least squares vs MLE... Formalize."

Q: How do you put error bars on a forecast? Bootstrap: resample the ladder points with replacement, refit, predict, repeat ~1000 times, take percentiles. You cannot do it analytically because the model is nonlinear and the noise model is unspecified. Then interpret against scale: roughly, 0.01 nats is a third more compute, so a CI wider than 0.05 nats cannot support a recipe decision.

Q: I will give you 3% of the flagship budget for ablations. How do you spend it? Geometrically spaced budgets to maximize the spread of log C, since extrapolation variance goes as (x* − x̄)² / Σ(x − x̄)². Keep the smallest run inside the scaling regime — below ~10M non-embedding parameters the behaviour is different and points there bias the fit. Give the largest single run about half the ladder budget, because it anchors the extrapolation. And replicate one budget so I can measure my noise rather than assume it.

Q: Two recipes. How do you decide? Fit a law for each on the same ladder design with the same estimator, evaluate both at the target C, and report the delta with confidence intervals plus the crossover point. Then check three things the loss number cannot see: does the candidate change serving cost, does it change training stability, and do downstream evals move in the same direction as loss.

Q: Are scaling laws ending? Loss-versus-compute curves have not visibly bent. What people usually observe is benchmark saturation, which is a measurement problem — leaderboards are volatile and can overfit human preference. The two live sources of continued progress are better architectures and new data sources, especially multimodal and synthetic. The real constraint is not the law flattening, it is unique high-quality data running out, which changes the shape of the optimization rather than ending it.

Tips & Takeaways

Tips

  • Always check a + b ≈ 1. Free bug detector on any fitted pair of exponents.
  • Always fit in log space, always use a robust loss. Two lines of code, enormous robustness gain.
  • Always report a confidence interval, and compare its width against 0.01 nats ≈ 33% more compute.
  • Pin your fitting code. A library optimizer's default method changing between versions can move your flagship recommendation. Hand-roll or pin exactly.
  • Replicate one ladder point. You cannot calibrate uncertainty without measuring noise.
  • Write down the recipe alongside the law. A law without its recipe is uninterpretable in six months.
  • Spread beats density. One more point an order of magnitude out beats three more points near your existing cluster.
  • Say the extrapolation distance out loud when presenting: "three decades beyond our largest ablation."

Takeaways

  1. Pre-training is a one-shot extrapolation problem. Scaling laws exist because of that.
  2. A law describes your recipe, not the universe. Fix the recipe first.
  3. Loss forecasting is recipe selection.
  4. L = E + A/N^α + B/D^β — irreducible + capacity + data. At frontier scale ~89% of the loss is irreducible; the whole industry is fighting over the rest.
  5. The IsoFLOPs U-curve has a flat bottom: a gift for serving-driven deviation, a trap for noisy fits.
  6. Chinchilla beat Kaplan by fixing an experimental-design bug. Methodology is the frontier.
  7. The estimator you choose changes your flagship recommendation. This is an open problem.
  8. Where you place ladder points matters more than how many. 13× variance reduction, same cost.
  9. A forecast without error bars is not a forecast.
  10. Compare laws at the target and report the crossover, never a single-scale winner.

References

Primary

  • Feinberg, Gemini Pretraining, Princeton, Apr 2025 — slides — the "Classical Scaling" section: the IsoFLOPs six-step build, the Kaplan/Chinchilla contrast, "UNDERTRAINED!", "Least squares vs MLE… Formalize", "Active learn…"
  • Austin et al., How To Scale Your Model — https://jax-ml.github.io/scaling-book/ — do the exercises; he offers interviews for them

Core papers

  • Kaplan et al., Scaling Laws for Neural Language Models, 2020 — https://arxiv.org/abs/2001.08361
  • Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), 2022 — https://arxiv.org/abs/2203.15556
  • Besiroglu et al., Chinchilla Scaling: A Replication Attempt, 2024 — https://arxiv.org/abs/2404.10102 — a replication finding issues in the original's reported fit; read this alongside Chinchilla, it is Chapter 9's problem in the wild
  • Clark et al., Unified Scaling Laws for Routed Language Models, 2022 — https://arxiv.org/abs/2202.01169
  • Muennighoff et al., Scaling Data-Constrained Language Models, 2023 — https://arxiv.org/abs/2305.16264 (Phase 02)
  • Sardana et al., Beyond Chinchilla-Optimal, 2024 — https://arxiv.org/abs/2401.00448 (Phase 02)
  • Grattafiori et al., The Llama 3 Herd of Models, 2024 — https://arxiv.org/abs/2407.21783 — their scaling-law methodology section is an unusually clear published account

Statistics

  • Efron & Tibshirani, An Introduction to the Bootstrap — the method in Chapter 10
  • Huber, Robust Estimation of a Location Parameter, 1964 — the loss in Chapter 8
  • Chaloner & Verdinelli, Bayesian Experimental Design, 1995 — the formal version of Chapter 11