Phase 03 — Roofline, MFU & Inference Co-Design

The phase where a product requirement becomes an architecture constraint. Feinberg's second research vertical is building "neural architectures that are efficient to run inference on" by choosing network topology, "shapes of the matrices" and "attention shapes, num heads" that "fully utilize the hardware." This phase gives you the arithmetic that makes those choices defensible instead of aesthetic — and reproduces the napkin math from his talk that turns "we want a real-time agent" into "the model must be smaller."

Why this phase exists

Two models with identical parameter counts can differ by 3× in serving speed, purely from shape choices made before training. And after training, the shape is frozen forever. So the decisions in this phase are among the few in the entire pipeline that are genuinely irreversible.

There is also a widespread confusion this phase kills. Feinberg spends part of the interview correcting people who see a "low" MFU number and conclude incompetence. To hit 100% you would need to be "doing a bunch of matmuls in a loop without reading any memory" — which is not a neural network, because real nets "have to apply activation functions or do attention or write intermediate outputs." 35% MFU is an accounting identity, not a grade. Once you can produce the breakdown, that number stops being a scoreboard and becomes an agenda.

The third thing this phase establishes is the split that governs all of serving:

PREFILL                              DECODE
processes the whole prompt at once   generates one token at a time
COMPUTE bound                        MEMORY-BANDWIDTH bound
high arithmetic intensity            1–50 FLOP/byte (far below any ridge)
fix it with pipelining, chunking     fix it with batching, GQA, quantization

They are two different machines. Applying one's optimization to the other is the most common serving mistake there is, and the reason prefill/decode disaggregation exists.

Concept map

   PRODUCT REQUIREMENT                    HARDWARE
   "respond in < 1 second"                peak FLOP/s, HBM bandwidth, HBM capacity
            │                                        │
            └──────────────┬─────────────────────────┘
                           ▼
                  ┌─────────────────┐
                  │    ROOFLINE     │   arithmetic intensity = FLOPs / bytes
                  │  ridge point =  │   below ridge -> reduce BYTES
                  │  peak / bandwidth│  above ridge -> reduce FLOPs
                  └────────┬────────┘
                           │
          ┌────────────────┴────────────────┐
          ▼                                 ▼
    ┌───────────┐                     ┌───────────┐
    │  PREFILL  │ compute-bound       │  DECODE   │ memory-bound
    │  2N/token │                     │ re-reads  │
    └─────┬─────┘                     │ ALL weights│
          │                           └─────┬─────┘
          └──────────────┬──────────────────┘
                         ▼
              ┌──────────────────────┐
              │   LATENCY NAPKIN     │  prefill + decode + scaffolding <= budget?
              └──────────┬───────────┘
                         │ if no...
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     more chips     smaller model   better shapes
     (expensive,    (Flash!)        (GQA, tiles,
      finite)                        depth/width)

What you will be able to do

  1. Compute arithmetic intensity and place any operation on a roofline; say whether to attack bytes or FLOPs.
  2. Compute the ridge point of any accelerator from its spec sheet, and explain why H100's (~296 FLOP/byte) is higher than A100's (~154) despite being the faster chip.
  3. Produce an MFU budget — decompose wall-clock into matmul / vector / memory / comms / optimizer — and read off the biggest lever.
  4. Distinguish MFU from HFU and explain why HFU is always higher (~33% for free).
  5. Reproduce Feinberg's napkin math: Llama3-70B on v5e, ~5.8 s of prefill on one chip, a 4×4 station to get under the 0.5 s API limit.
  6. Show that at batch 1, decode costs ~3.8× prefill — and explain why that changes the provisioning answer.
  7. Quantify three co-design levers: tile quantization, GQA group size, and depth-vs-width.
  8. Prove that decode stays memory-bound even at batch 1024, and explain what follows from that.

The lab

LabWhat you build
Lab 01 — Roofline, MFU Budget & the Latency NapkinThe full co-design toolkit: roofline analysis with ridge points and a "which lever" verdict, MFU/HFU accounting and budget decomposition, prefill/decode latency models, a chip-count solver for a latency budget, and the shape levers — tile efficiency, GQA saving, decode arithmetic intensity vs batch, depth-vs-width serial cost

Success criteria. LAB_MODULE=solution pytest test_lab.py -v → 50 passed; python solution.py reproduces the talk's ~5.8 s / 4×4 result and shows the chip count falling as the model shrinks.

Deliverables checklist

  • I can compute a ridge point and say what it means physically.
  • I can produce an MFU budget and name the biggest lever from it.
  • I always ask "MFU or HFU?" when shown a utilization number.
  • I have reproduced the ~5.8 s single-chip prefill figure.
  • I can explain why decode beats prefill at batch 1 and what that does to provisioning.
  • I can state the GQA saving as a ratio and in concurrent requests.
  • I can explain why decode never becomes compute-bound at realistic batch sizes.

Key takeaways

  1. Arithmetic intensity decides which resource you are fighting. It is a property of the algorithm, not the chip.
  2. Below the ridge, cut bytes; above it, cut FLOPs. Everything else is detail.
  3. MFU of 35% is an identity, not a grade. The breakdown is your agenda.
  4. HFU ≥ MFU, always. Activation checkpointing counted as useful work is ~33% for free.
  5. Prefill and decode are different machines. Never apply one's fix to the other.
  6. Decode is memory-bound at every realistic batch size — even 1024. That single fact explains batching, GQA, quantization and speculative decoding.
  7. n_kv_heads is the biggest single serving lever, it costs almost no quality, and it is frozen at pre-training time.
  8. Depth is serial and costs latency; width is parallel. As wide as quality allows, as shallow as quality tolerates.
  9. When the latency budget cannot be met, the answer is a smaller model, not more chips. That is why Flash and Flash-Lite exist, and why this is a pre-training concern.

References

  • Feinberg, Gemini Pretraining, Princeton, Apr 2025 — slides — the "Small Model Customers" and "Inference-optimized Scaling" sections, and the Llama3-70B/v5e napkin math
  • Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model, CACM 2009 — the original model
  • Pope et al., Efficiently Scaling Transformer Inference, 2022 — https://arxiv.org/abs/2211.05102 — the definitive treatment of prefill/decode arithmetic
  • Austin et al., How To Scale Your Model — https://jax-ml.github.io/scaling-book/ — do the exercises
  • Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (MQA), 2019 — https://arxiv.org/abs/1911.02150
  • Ainslie et al., GQA, 2023 — https://arxiv.org/abs/2305.13245
  • Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, 2022 — https://arxiv.org/abs/2205.05198 — where the MFU/HFU distinction is made precise
  • Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM), 2023 — https://arxiv.org/abs/2309.06180