Lab 01 — IsoFLOPs Ladder, Law Fitting & the Forecast

Build the machine that turns a few cheap experiments into a defensible prediction about an experiment you can only run once.

The problem

You have a flagship budget of 1e25 FLOPs and one shot. Before spending it you must state what you will train and what loss you expect, and defend both. The only tool is a ladder of small runs and a fitted curve — so the whole job reduces to: fit well, extrapolate honestly, and know how wrong you might be.

This lab builds that machine end to end, and along the way reproduces the Kaplan → Chinchilla correction from scratch — because the lesson there is not a fact to memorize, it is a failure mode to recognize in your own work.

What you build

Key concepts

Files

Run

pytest test_lab.py -v                       # your lab.py — red until you implement
LAB_MODULE=solution pytest test_lab.py -v   # the reference — must be green (64 passed)
python solution.py                          # the worked example

Where to start

Work top to bottom; each group depends on the one above.

1. The spine. fit_line → fit_power_law → fit_quadratic → parabola_vertex. Get test_fit_power_law_recovers_exponent and test_fit_quadratic_recovers_exact_parabola passing before anything else.
2. parametric_loss and synthetic_ladder. Note the budget constraint: every point on an IsoFLOPs curve satisfies C = 6ND, and a test checks it.
3. The IsoFLOPs method, ending at test_exponents_sum_to_one.
4. fit_parametric. Deterministic multi-start coordinate descent. test_fit_parametric_is_deterministic is not optional garnish — a fit that moves between runs cannot support a nine-figure decision.
5. The rest.

The money test is test_nonuniform_bias_tilts_the_fitted_exponent, paired with its control test_uniform_offset_is_absorbed_into_E_and_changes_nothing. Together they are the Kaplan → Chinchilla lesson: a uniform measurement error changes nothing, a non-uniform one changes your flagship recommendation.

The traps:

• fit_quadratic on collinear x values → must raise ValueError, not ZeroDivisionError.
• A downward-opening parabola has a maximum → must raise, because it means your sweep is wrong.
• Bootstrap resamples can be degenerate → skip them, and fail loudly if too few succeed.
• Do not recompute C = 6ND to regroup a ladder. Floating-point rounding will split one budget into two groups. Carry the budget label. (A test comment calls this out.)
• Everything random goes through a seeded random.Random(seed).

Success criteria

• LAB_MODULE=solution pytest test_lab.py -v → 64 passed.
• Your lab.py reaches 64 passed.
• python solution.py runs, and you can explain all eleven sections.
• You can explain why a uniform loss offset changes nothing and a non-uniform one changes everything.
• You can state why a + b ≈ 1 must hold and what it catches.
• You can produce a forecast with a CI and say whether it is tight enough to decide on.
• You can justify a ladder design in terms of extrapolation variance and dollars.

How this maps to the real stack

What is not a lie: the IsoFLOPs procedure, the a + b = 1 identity, the analytic optimum derivation, and the extrapolation-variance formula. Those are exact, and they are what you will be asked about.

Extensions

1. Fit against published data. Pull the (N, D, loss) table from any open model report (Llama 3's scaling section is unusually complete) and fit your law to it. Compare your extrapolation to what they actually shipped.
2. Add a hierarchical noise model. Points within one budget share a run's seed and infrastructure; model that correlation and see how the confidence interval widens. This is Feinberg's "formal stats model" suggestion, at a tractable scale.
3. Implement true optimal design. Instead of scoring fixed candidate designs, search for the next C that minimizes posterior predictive variance at the flagship. That is the "active learn" bullet on his slide, and it is a publishable exercise.
4. Add a third axis. Extend to L(N, D, U) with unique tokens, then compare against Phase 02's data-constrained law.
5. Compare estimators properly. Run a simulation study: generate 1,000 ladders from known truth, fit with each estimator, and report bias and variance of the recovered exponents. That is a real, small, honest paper.

Interview / resume bullets

• "Implemented the full IsoFLOPs scaling-law methodology from scratch — parabola fitting for per-budget optima, power-law regression for N_opt(C) and D_opt(C) with an a + b = 1 consistency check, and robust (Huber-on-log) fitting of the parametric L(N, D) surface — producing flagship forecasts with bootstrap confidence intervals."
• "Reproduced the Kaplan→Chinchilla correction from first principles by simulating a schedule-mismatch measurement bias, demonstrating that a uniform loss offset is absorbed into the irreducible term while a non-uniform one tilts the fitted exponents and shifts the compute-optimal recommendation."
• "Built optimal-experimental-design tooling for scaling ladders, showing ~26× lower extrapolation variance from point placement at identical cost, and sized a ladder at under 2% of flagship budget for 5.2 decades of coverage."
• Interview-ready: "I would not train both and compare. I would fit a law for each recipe over a shared ladder, evaluate both at the target FLOP count, and report the delta with confidence intervals plus the crossover — because curves cross, and 'better' silently assumes a scale."