Frontier Pre-Training Lead

← All Roles

The track that teaches you the job Vlad Feinberg does: turn a fixed pile of compute into the best model that can actually be served — forecast it before you spend the money, build it with an architecture the hardware likes, compress it until it fits a real latency budget, and keep the run alive for forty days while it happens.

Start here

Document	What it gives you
The Role	Who Vlad Feinberg is, the exact career path, what a pre-training area lead owns, the synthesized job description, the skill matrix, and his own hiring advice
The Transcript, Dissected	Every claim from his interview and his Princeton talk, decoded from zero background — MoE, scaling laws, MFU, distillation, quantization, pipeline prefill, the Flash 2.0 war story — with runnable code for each
Lab Standard	What every lab in this track guarantees
Glossary	Every term, one line each
Cheat Sheet	The numbers and formulas to have memorized

Then work the phases in order. Phase 00 gates everything — if C = 6ND is not something you can derive on a napkin, nothing downstream will land.

Every other track in this hub teaches you to use models — serve them, fine-tune them, chain them into agents. This one teaches you to decide what model should exist in the first place, which is a different discipline with different math and a different failure mode.

The defining constraint: you get one shot. A flagship pre-training run costs eight or nine figures, takes one to three months, and is by construction larger than anything you have ever run. You cannot A/B it. You cannot roll it back. So the entire field is organized around one question — how do you make a defensible prediction about an experiment you have never run? — and everything else (scaling laws, IsoFLOPs ladders, ablations, forecasting) is machinery for answering it.

Then a second constraint lands on top: the model has to be servable. Feinberg's own framing is that Chinchilla-optimal is the wrong objective for a model that will serve trillions of tokens, because model size is a serving cost paid forever. So the objective becomes quality per served token per watt — and that pulls distillation, quantization, MoE, sharding, and roofline analysis into what would otherwise be a pure modelling job.

Nobody teaches this. There is no course. The knowledge lives in about a dozen papers, a handful of talks, and the heads of maybe a few hundred people. This track assembles it.

The roadmap

   BUILT ─────────────────────────────────────────────────────────┐
                    ┌─────────────────────────────────────────┐   │
   FOUNDATIONS      │ 00  FLOPs & memory algebra   C = 6ND    │   │
                    │ 01  Scaling laws  Kaplan → Chinchilla   │   │
                    └────────────────┬────────────────────────┘   │
                                     │                            │
                    ┌────────────────▼────────────────────────┐   │
   ARCHITECTURE     │ 02  Mixture of Experts from scratch     │   │
   & HARDWARE       │ 03  Roofline, MFU & inference co-design │   │
                    └────────────────┬────────────────────────┘   │
   ───────────────────────────────── │ ───────────────────────────┘
   PLANNED                           │
                    ┌────────────────▼────────────────────────┐
   SYSTEMS          │ Sharding & parallelism  DP/TP/PP/EP     │
                    │ Pipeline prefill  (the Flash 2.0 fix)   │
                    └────────────────┬────────────────────────┘
                                     │
                    ┌────────────────▼────────────────────────┐
   COMPRESSION      │ Scaling laws II   U/R, inference-aware  │
                    │ Distillation at trillion-token scale    │
                    │ Quantization & power economics          │
                    └────────────────┬────────────────────────┘
                                     │
                    ┌────────────────▼────────────────────────┐
   OPERATIONS       │ Training stability & the 40-day run     │
   & CRAFT          │ Post-training: SFT, RLHF, PPO           │
                    │ Kernels & tile DSLs                     │
                    │ Research as an MDP  (taste)             │
                    └────────────────┬────────────────────────┘
                                     │
                    ┌────────────────▼────────────────────────┐
   CAPSTONE         │ Mini frontier lab: budget → recipe →    │
                    │ forecast → compress → serve → the memo  │
                    └─────────────────────────────────────────┘

Phase index

Built and test-verified

#	Phase	What you build	Maps to
00	FLOPs & Memory Algebra	A transformer FLOP/memory/time calculator; the `6ND` derivation; MoE active-vs-total; KV-cache sizing; budget → `(N, D)` → dollars	"`C = 6ND` is a very good approximation" · 75 tests
01	Scaling Laws I — Kaplan → Chinchilla	IsoFLOPs ladder, parabola/power-law/parametric fitting, bootstrap forecasts, optimal ladder design, the baseline-vs-candidate crossover	The six-step IsoFLOPs slide; "UNDERTRAINED!" · 64 tests
02	Mixture of Experts From Scratch	Router, top-k gating, load-balance & z-loss, capacity/drop/pad, shared experts, total-vs-active, the all-to-all wall, a router-collapse simulator	"MoE… uses a lot more parameters" · 53 tests
03	Roofline, MFU & Inference Co-Design	Roofline with ridge points, MFU/HFU accounting and budget decomposition, prefill/decode latency models, the chip-count solver, GQA and tile levers	"shapes… that fully saturate all of those hardware units" · 50 tests

Roadmap — the remaining phases

These are specified but not yet built. Each will follow the same contract: README.md + WARMUP.md + a runnable, test-verified lab. The material for all of them is already covered conceptually — with code — in The Transcript, Dissected, which is the fastest way to get the ideas now.

Phase	What it will build	Covered today in
Scaling Laws II — beyond `L(N, D)`	`L(N, U, R)` data-constrained optimizer, inference-aware lifetime cost, routed & distillation laws	Claims 15–17
Sharding & Parallelism	Collective cost models, DP/FSDP/TP/PP/EP planner, the `4×4` mesh question	Claim 12
Pipeline Prefill & Serving	All-to-all vs pipelined-prefill simulator, bubble analysis, prefill/decode disaggregation	Claim 12
Distillation at Scale	KL/temperature losses, top-k teacher-logit store, teacher-compute allocation	Claims 3, 17
Quantization & Power Economics	Affine/group/vector quantizers, outlier handling, the energy & TCO model	Claim 5
Training Stability & SRE	Goodput model, spike detector, checkpoint-policy optimizer	Claim 13
Post-Training: SFT, RLHF, PPO	SFT masking, Bradley-Terry reward model, PPO step with KL control	Claim 1
Kernels & Tile DSLs	A tiny tile-DSL interpreter, online softmax, memory-traffic accounting	Claims 1, 19
Research as an MDP	Portfolio planner, value of information, calibration scoring	Claim 2
Capstone — Mini Frontier Lab	Budget → recipe → forecast → compress → serve → the go/no-go memo	the whole track

What every phase contains

File	Voice	What it gives you
`README.md`	the syllabus	why the phase exists, the concept map, the lab spec, deliverables, key takeaways
`WARMUP.md`	the professor	zero background → principal level. Every term built from nothing: what it is → why it exists → how it works underneath (mechanism, diagrams, math, runnable code) → what it costs in production → the common misconceptions. Then a lab walkthrough, interview Q&A, tips & takeaways, and primary-source references
`lab-*/`	the workbench	a runnable, test-verified miniature: `README.md`, `lab.py` (TODOs), `solution.py`, `test_lab.py`, `requirements.txt`

Labs are pure Python stdlib, offline, deterministic, free. No GPU, no network, no model downloads. You implement the mechanism — the FLOP counter, the parabola fit, the router, the collective cost model, the pipeline scheduler, the quantizer, the PPO step — because that is what makes the knowledge defensible in an interview and useful at 2 a.m. See LAB-STANDARD.md.

Suggested schedule

Assumes ~10 focused hours a week. Compress aggressively if you already have the background.

Weeks	Work	Milestone
1	Read jd.md + TRANSCRIPT-DISSECTED.md	You can explain MoE, MFU, `6ND`, distillation and the Flash 2.0 fix to a colleague
2	Phase 00	You can size any model on any cluster on a napkin
3–4	Phase 01	You can fit a scaling law and defend a flagship forecast with error bars
5–6	Phase 02	You can implement an MoE layer and debug a collapsing router
7–8	Phase 03	You can do the latency napkin and defend an architecture shape
9+	The four artifacts below	The portfolio

The parallel track: build the artifacts he asks for

While working the phases, produce the four things he explicitly says he screens on (see jd.md §8):

Handwritten How To Scale Your Model exercises, on video.
A transformer from scratch — with a FLOP counter and a memory profile, not just a forward pass.
A real, benchmarked contribution to vLLM / SGLang / TensorRT.
A citation-tree writeup for one topic, ending with the paper that contradicts the consensus.

Phases 00, 05, and 11 feed directly into #1 and #2. Phase 12 is #4 as a method.

Prerequisites

Required: Python (comfortable), and the ability to sit with algebra. That is genuinely it — every other concept is built from scratch in the WARMUPs.

Helpful but not assumed: having seen a transformer before; having trained any model; basic probability. If you have none of these, read TRANSCRIPT-DISSECTED §Part 0 first — it defines the nine words the rest depends on.

Explicitly not required: a GPU, a cloud account, a PhD, or a job at a lab. Feinberg himself did not finish the PhD, and his slide "Future Pretrain Research Ideas – Without Big Costs!" is a list of open problems you can work on with a laptop.

Senior AI Engineer — the from-scratch implementation track: tokenizer, transformer, autograd, LoRA, serving internals. Best companion; this track assumes the mechanism and focuses on the decision.
LLM Inference Engineer — deeper on serving stacks.
Model Accuracy & AI Performance — deeper on quantization tooling and compilers.
GPU Engineering — deeper on CUDA and the hardware itself.

Key takeaways for the whole track

You get one shot. Everything upstream of the flagship run exists to make one irreversible decision defensible.
A scaling law describes your recipe, not the universe. Change the recipe, refit the law.
Loss forecasting is recipe selection. Compare fitted curves at the target FLOP count.
Optimize lifetime cost, not training cost — while knowing the inference term is unforecastable.
Nothing is free; costs move. MoE trades compute for memory, communication, instability, and data hunger.
Prefill and decode are different machines. Compute-bound vs bandwidth-bound.
Power is the bill. Quantization is an energy lever before it is a memory lever.
Distillation is variance reduction, and its hard part is storage.
Goodput is a research deliverable. 78% vs 94% on the same hardware is days of a 40-day run.
The artifact is the signal. Build things other people can check.

Frontier Pre-Training Lead — Scaling Laws, MoE, Distillation & Inference Co-Design