Lab Standard — Frontier Pre-Training Lead

Every lab in this track builds a runnable, test-verified miniature of a real pre-training decision procedure — the FLOP counter, the IsoFLOPs fitter, the MoE router with capacity, the collective cost model, the roofline, the pipeline scheduler, the teacher-logit store, the 4-bit quantizer, the goodput model, the PPO step, the tile-DSL interpreter, the research planner.

You do not "call scaling_laws.fit()". You implement the parabola fit, the power-law regression, and the extrapolation, and then you report the confidence interval. That is what makes the knowledge defensible when a director asks you to justify a $40M run.

Why miniatures and not real training runs

A real pre-training run costs eight figures, needs a datacenter and clearances, is non-deterministic, and hides every mechanism behind XLA and a scheduler. A lab that reimplements the decision algebra — 6ND, the IsoFLOPs parabola, the load-balance loss, the all-to-all byte count, the arithmetic-intensity ridge point, the pipeline bubble fraction, the quantization error bound, the goodput integral — is offline, deterministic, free, and teaches exactly what a frontier interview probes.

Every lab README ends with a "How this maps to the real stack" section connecting the miniature to JAX/XLA, GSPMD, MaxText, Megatron-LM, DeepSpeed, vLLM, or the relevant paper — including where the miniature lies.

Required files (per lab)

The runnable core is Python (stdlib + pytest), offline, deterministic. No GPU, no CUDA, no network, no model downloads, no pip install torch, no unseeded randomness. Where a lab needs vectors, matrices, or a least-squares fit, implement it with stdlib lists/math so the mechanism stays visible — NumPy only if a lab explicitly declares it in its requirements.txt, and even then the algorithm must be the thing you write.

Determinism rules

• Any randomness goes through an explicit seeded random.Random(seed); same seed → same bytes. Tests assert this.
• Float comparisons in tests use pytest.approx or an explicit tolerance — never ==.
• softmax, log-sum-exp, KL, and any loss use the max-subtraction / log-space trick, so tests can include large logits without overflow. That numerical detail is part of the lesson.
• Curve fits use closed-form or iterative solvers you write, with a fixed iteration count and a fixed tolerance — no library optimizer whose version could change the answer.

Units discipline (specific to this track)

Half of frontier interview failures are unit errors. So:

• Every function that returns a physical quantity names the unit in the identifier or the docstring: flops, bytes_moved, seconds, joules, dollars_per_million_tokens.
• Tests include at least one dimensional-analysis assertion — e.g. doubling parameters doubles FLOPs; halving bandwidth doubles memory-bound time; 2N forward and 4N backward sum to 6N.
• Any constant taken from hardware (peak FLOP/s, HBM bandwidth, pJ per bit) lives in a single named table at the top of the module with a source comment, never inline.

The test contract

import importlib, os
lab = importlib.import_module(os.environ.get("LAB_MODULE", "lab"))

Run both ways; the reference must pass, and your lab.py passes once the TODOs are filled:

pytest test_lab.py -v                       # against your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # against the reference (must be green)
python solution.py                          # the worked example

Test taxonomy every lab includes:

• happy path — the textbook case
• malformed / out-of-range input — raises ValueError (negative parameter counts, zero bandwidth, top_k > n_experts, capacity factor below 1/k)
• boundary cases — the off-by-ones interviewers probe: one chip, one expert, one token, k = n_experts (MoE degenerates to dense), zero repeats, a single pipeline stage, a quantization group where every value is identical
• numerical-stability cases — large logits do not overflow; log(0) is guarded; a degenerate parabola fit is detected rather than dividing by zero
• invariants — softmax sums to 1; gate weights over the chosen top-k sum to 1; forward + backward = 3 × forward FLOPs; dequant(quant(x)) is within the group's error bound; pipeline utilization is M/(M+S−1); goodput ≤ 1
• deterministic output — same seed → same bytes

The two teaching docs (per phase)

Doc conventions (mdBook-compatible)

• WARMUP.md opens with a Table of Contents of working anchor links, kept in sync with the headings.
• MathJax for math: \( … \) inline, $$ … $$ block.
• No bare angle brackets in prose — wrap <like-this> in backticks so mdBook does not eat them as HTML.
• File references use relative links. Code fences are language-tagged.
• Every quantitative claim taken from a paper or talk carries a citation in the phase's References section.

Definition of done

A lab is complete only when the reference suite passes (LAB_MODULE=solution pytest), the learner lab.py passes once the TODOs are filled, python solution.py prints a sensible worked example, the README's success criteria are testable, and the "How this maps to the real stack" section is honest about where the miniature diverges from production.

A phase is complete when both teaching docs exist, the WARMUP's ToC anchors resolve, and every lab in it is done.