Warmup Guide — Roofline, MFU & Inference Co-Design

How to read this. No prior knowledge of performance modelling is assumed. Arithmetic intensity, the roofline, MFU, memory-bound versus compute-bound, and the prefill/decode split are all built from nothing. If you already profile kernels, start at Chapter 4 and do not skip Chapter 6 or Chapter 8.

Table of Contents

• Chapter 1: The Question This Phase Answers
• Chapter 2: A Chip Is Several Machines
• Chapter 3: Arithmetic Intensity and the Roofline
• Chapter 4: MFU — an Accounting Identity, Not a Grade
• Chapter 5: Prefill and Decode Are Two Different Machines
• Chapter 6: The Latency Napkin
• Chapter 7: Shape Co-Design, Lever by Lever
• Chapter 8: The Decode Wall
• Chapter 9: What Co-Design Actually Looks Like as a Job
• Lab Walkthrough
• Success Criteria
• Interview Q&A
• Tips & Takeaways
• References

Chapter 1: The Question This Phase Answers

Feinberg's talk poses a product scenario and then does the arithmetic live. Paraphrasing his setup: a web-interaction agent with a 128k context but only 8k incremental per turn, 128 decode tokens to emit an action, no more than a second of latency between actions, and 250 ms of that already consumed by "scaffolding, load balancing, request validation, kv cache retrieval" — which he flags as an optimistic allowance.

Then: Llama3-70B on v5e chips. What happens?

His answer, in one line from the slide: "Uh oh… 5.7 seconds for 1 chip." And the conclusion: to hit a 0.5 s API limit on prefill alone you already need a 4×4 station of v5e.

That calculation is the entire phase. It is how a product requirement becomes an architecture constraint, and it is why the pre-training team cares about serving at all. The job title says "pre-training," but the objective function has a serving term in it, and this chapter is that term.

Why this belongs to pre-training and not to the serving team

Because the levers that matter most are frozen when training starts:

• n_kv_heads — decided before training, changes serving throughput by up to 64×
• depth vs width at fixed N — decided before training, changes decode latency directly
• matrix dimensions — decided before training, changes how well the hardware tiles
• total parameter count — decided before training, and it is the dominant term in everything

A serving engineer inherits all of these. Only the pre-training lead can choose them. That is what "inference co-design" means, and why it is one of Feinberg's three verticals.

Chapter 2: A Chip Is Several Machines

The mental model that fixes most confusion

People treat an accelerator as one number ("an H100 does a petaflop"). It is not. It is several units with wildly different throughputs, and your model must use all of them:

┌──────────────────────────────────────────────────────────┐
│  ACCELERATOR                                             │
│                                                          │
│  ┌────────────────┐   VERY fast. This is the number      │
│  │  MATMUL UNIT   │   on the spec sheet.                 │
│  │  ~990 TFLOP/s  │   (tensor cores / systolic array)    │
│  └────────────────┘                                      │
│                                                          │
│  ┌────────────────┐   50-100x SLOWER than matmul.        │
│  │  VECTOR UNIT   │   gelu, softmax, norms, residuals.   │
│  └────────────────┘   Counted in nobody's "peak".        │
│                                                          │
│  ┌────────────────┐   ~3.35 TB/s. Every activation       │
│  │  HBM           │   in and out crosses this.           │
│  └────────────────┘                                      │
│                                                          │
│  ┌────────────────┐   ~0.05-0.9 TB/s. Collectives.       │
│  │  INTERCONNECT  │   Slowest by far.                    │
│  └────────────────┘                                      │
└──────────────────────────────────────────────────────────┘

"Peak FLOP/s" describes the matmul unit alone. Everything else your model does is time during which that unit sits idle. Chapter 4 turns this observation into an accounting identity.

The number to internalize

Bandwidth is the scarce one, and it has been getting relatively scarcer for two decades. Compute has grown faster than memory bandwidth every generation — which is why more and more workloads are memory-bound over time, and why the roofline's ridge point keeps moving right.

for chip, (peak, hbm, bw, watts) in HARDWARE.items():
    print(f"{chip:10s} {peak/1e12:6.0f} TFLOP/s  {bw/1e12:5.2f} TB/s  "
          f"-> ridge {peak/bw:6.0f} FLOP/byte")

H100          990 TFLOP/s   3.35 TB/s  -> ridge    296 FLOP/byte
A100-80       312 TFLOP/s   2.03 TB/s  -> ridge    154 FLOP/byte
TPU v5e       197 TFLOP/s   0.82 TB/s  -> ridge    241 FLOP/byte
TPU v5p       459 TFLOP/s   2.77 TB/s  -> ridge    166 FLOP/byte

A counter-intuitive fact worth holding: the H100's ridge point (296) is higher than the A100's (154), even though the H100 is the better chip. Compute improved 3.2×; bandwidth only 1.7×. Newer hardware is harder to keep fed, so memory-bound workloads get relatively worse on it, not better. Anyone who says "just wait for better GPUs" has not looked at this column.

Chapter 3: Arithmetic Intensity and the Roofline

Arithmetic intensity

$$ I = \frac{\text{FLOPs performed}}{\text{bytes moved}} $$

"How much arithmetic do I get per byte I drag out of memory?" It is a property of the algorithm and its blocking, not of the chip.

def arithmetic_intensity(flops, bytes_moved):
    if bytes_moved <= 0:
        raise ValueError("bytes_moved must be positive")
    return flops / bytes_moved

The ridge point

$$ I_{\text{ridge}} = \frac{\text{peak FLOP/s}}{\text{HBM bandwidth}} $$

The intensity at which the two resources are exactly balanced.

• I < I_ridge → memory-bound. The matmul unit is starved. Adding FLOPs is free; removing bytes is what helps.
• I > I_ridge → compute-bound. Memory keeps up. Removing FLOPs is what helps.

The roofline itself

achievable
FLOP/s
   peak ┤              ┌──────────────────────────  compute-bound (flat roof)
        │             ╱
        │            ╱
        │           ╱   slope = HBM bandwidth
        │          ╱
        │         ╱      memory-bound (sloped roof)
        │        ╱
        └───────┴──────────────────────────────────► arithmetic intensity
                ▲
            ridge point (~296 FLOP/byte on H100)

def roofline_throughput(chip, intensity):
    peak, _hbm, bw, _w = HARDWARE[chip]
    return min(peak, bw * intensity)      # the whole model, in one line

That single min() is the entire roofline model. Its power is not the formula; it is that it tells you which lever to pull, which stops you optimizing the wrong thing for a week.

Three operations, placed

big matmul  (8192^3)   AI=  2730.7  compute   100.0% of peak  -> reduce FLOPs
decode, batch=1        AI=     1.0  memory      0.3% of peak  -> reduce bytes moved
layernorm (8k x 8k)    AI=     1.2  memory      0.4% of peak  -> reduce bytes moved

Read the middle row again. Batch-1 decode achieves 0.3% of an H100's peak — and that is not a bug, it is the arithmetic. You are reading 140 GB of weights to do 140 GFLOPs of work. No kernel engineer can fix that; only batching, quantization, or a smaller model can.

The third row is why kernel fusion exists: a layernorm does almost no arithmetic but has to read and write the whole activation tensor. Fusing it into the neighbouring matmul's epilogue removes the round-trip entirely.

The misconception

"Our MFU is low because our kernels are bad."

Maybe. But if your operation sits at intensity 1 on a chip whose ridge is 296, no kernel can be good. The roofline tells you the ceiling before you start optimizing, which is precisely its value: it distinguishes "we implemented this badly" from "this is what the algorithm costs."

Chapter 4: MFU — an Accounting Identity, Not a Grade

The definition

$$ \text{MFU} = \frac{\text{useful model FLOPs per second}}{\text{peak FLOP/s} \times \text{chips}} $$

Where "useful model FLOPs" is the 6ND arithmetic the model requires — not including recomputation.

Why it is never near 100%

Feinberg addresses this directly, because people on social media see a number like 35% and conclude someone is incompetent. His point: to reach 100% you would need to be "doing a bunch of matmuls in a loop without reading any memory," and that is not a neural network. Real nets "have to apply activation functions or do attention or write intermediate outputs."

So build the budget:

def mfu_budget(matmul_s, vector_s, memory_s, comms_s, optimizer_s):
    parts = {"matmul": matmul_s, "vector": vector_s, "memory": memory_s,
             "comms": comms_s, "optimizer": optimizer_s}
    total = sum(parts.values())
    out = {f"{k}_fraction": v / total for k, v in parts.items()}
    out["mfu"] = matmul_s / total
    losses = {k: v for k, v in parts.items() if k != "matmul"}
    out["biggest_lever"] = max(losses, key=losses.get)
    return out

MFU (matmul busy)      35.7%
lost to vector         16.1%
lost to memory         21.4%
lost to comms          17.9%
lost to optimizer       8.9%
-> biggest lever: memory. That is your next week's work.

35.7% is not a failure. It is a sum that adds to 100%. And notice what you now have that you did not have from the single number: an ordered work queue.

Feinberg's framing of the goal connects straight back to co-design: the aim is "choosing shapes for this neural net that fully saturate all of those hardware units" — not maximizing one number.

MFU vs HFU — the trap

HFU (Hardware FLOPs Utilization) counts recomputation from activation checkpointing as useful work. Full checkpointing takes hardware FLOPs per token from ~6N to ~8N:

def hfu(model_flops, seconds, chip, n_chips, recompute_factor=8/6):
    return mfu(model_flops * recompute_factor, seconds, chip, n_chips)

MFU (honest)                        11.8%
HFU (counts recompute as useful)    15.7%
Same run. Same hardware. ~33% apart, for free.

HFU is always ≥ MFU. When a blog post, a vendor benchmark, or a colleague quotes a utilization number, ask which one. A third of the difference between two teams' reported numbers is often just this definition.

Two more comparability traps. (1) MFU is only comparable within a fixed hardware/precision class — an FP8 number uses a different (larger) denominator than a bf16 one. (2) "Peak with sparsity" doubles the denominator again and assumes a 2:4 structured sparse model you almost certainly do not have.

Chapter 5: Prefill and Decode Are Two Different Machines

This is the most important structural fact in serving, and everything in Chapters 6–8 follows from it.

Prefill

You have the user's whole prompt. Process all T tokens at once. Every weight you read is used for T tokens, so arithmetic intensity is high.

def prefill_seconds(n_tokens, n_params, chip, n_chips, mfu_frac=1.0):
    """2N FLOPs per token, forward only. COMPUTE bound."""
    peak, _hbm, _bw, _w = HARDWARE[chip]
    return (2.0 * n_params * n_tokens) / (n_chips * peak * mfu_frac)

Decode

You produce one token at a time. To produce it you must read every weight in the model, and then do 2N FLOPs with them. Intensity ≈ 1.

def decode_seconds(n_tokens, n_params, chip, n_chips, bytes_per_param=2):
    """Every generated token re-reads EVERY weight. MEMORY-BANDWIDTH bound."""
    _peak, _hbm, bw, _w = HARDWARE[chip]
    return n_tokens * n_params * bytes_per_param / (n_chips * bw)

The table to memorize

Why this is not a technicality

Because the optimizations are opposite:

• Pipelining works beautifully for prefill (thousands of tokens to chunk, so transfers hide behind compute) and is actively harmful for decode — it adds S−1 serial hops to every single token, with no parallel work to overlap.
• Batching is the primary decode lever and does comparatively little for prefill, which is already compute-saturated.

That asymmetry is exactly why prefill/decode disaggregation exists — running the two phases on separately provisioned hardware — and it is the reason the Flash 2.0 fix was pipelined prefill specifically, not pipelined inference in general.

Chapter 6: The Latency Napkin

Now reproduce the calculation from the talk, end to end.

The setup

• 8k incremental prefill per turn
• 128 decode tokens (enough to emit one agent action)
• 1.0 s total budget
• 250 ms of it already spent on scaffolding
• Llama3-70B (140 GB in bf16) on TPU v5e (197 TFLOP/s, 16 GB, 819 GB/s)

The first thing to check: does it even fit?

def weights_fit_chips(n_params, chip, bytes_per_param=2):
    _peak, hbm, _bw, _w = HARDWARE[chip]
    return math.ceil(n_params * bytes_per_param / hbm)

weights_fit_chips(70e9, "TPU v5e") → 9 chips, just to hold the model. Before a single consideration of speed. Capacity and latency push in the same direction, which is a recurring theme.

The sweep

   1 chips: prefill   5.822s  decode  21.880s  total  27.952s  MISS
   4 chips: prefill   1.455s  decode   5.470s  total   7.176s  MISS
  16 chips: prefill   0.364s  decode   1.368s  total   1.981s  MISS
  64 chips: prefill   0.091s  decode   0.342s  total   0.683s  OK
 128 chips: prefill   0.045s  decode   0.171s  total   0.466s  OK

The prefill column reproduces the talk exactly. One chip gives 5.8 s, matching his "5.7 seconds for 1 chip"; a 4×4 = 16-chip station brings prefill to 0.36 s, under the 0.5 s API limit he quotes.

The part most summaries get wrong

Look at the decode column. At batch 1, generating 128 tokens costs ~3.8× more than prefilling 8192 of them. 8192 tokens of prefill is 1.1 PFLOPs of compute; 128 tokens of decode is 17.9 TB of memory traffic. The chip is fast at the former and slow at the latter.

So while the prefill station needs 16 chips, the full turn needs ~64. The talk's 4×4 figure is about prefill specifically — and the honest full-budget answer is larger.

The caveat that keeps this honest: batch-1 decode is the worst case. Real serving batches many concurrent requests, which amortizes the weight read across all of them — that is precisely why continuous batching exists. But an interactive agent that must respond within a second cannot always wait to fill a batch. The tension between latency (small batch) and throughput (large batch) is the central trade-off of serving, and this table is where you feel it.

The conclusion that pays for the whole phase

N= 70.0B ->    64 chips for a 1.0s turn   (9 just to hold weights)
N= 35.0B ->    32 chips for a 1.0s turn   (5 just to hold weights)
N= 17.0B ->    16 chips for a 1.0s turn   (3 just to hold weights)
N=  8.0B ->     8 chips for a 1.0s turn   (1 just to hold weights)

Halving N halves both columns. Prefill FLOPs are linear in N; decode bytes are linear in N. There is no shape trick that beats simply having a smaller model.

That is the economic case for Flash and Flash-Lite, stated as arithmetic — and it is why a pre-training lead owns it. The serving team cannot make the model smaller. Only the recipe can.

Chapter 7: Shape Co-Design, Lever by Lever

Three levers, all frozen at pre-training time.

Lever 1 — tile quantization

Matmul units work on fixed tiles (128×128 on a TPU systolic array; multiples of 8/16/64 for tensor cores). A dimension that is not a multiple gets padded, and you pay for the padding.

d_ff= 11000 ->  99.9% of the padded matmul is real work
d_ff= 11008 -> 100.0%
d_ff=  8192 -> 100.0%
d_ff=  4097 ->  97.0%
d_ff=  4096 -> 100.0%

Individually small. But it applies to d_model, d_ff, head dimension, expert count, and every shard boundary, and it compounds multiplicatively across dozens of layers. This is why production models use dimensions like 4096, 8192, 11008 rather than round decimal numbers.

The worst case is one element past a boundary: tile_efficiency(129, tile=128) = 50.4%. You pay for two tiles to use one and a bit.

Lever 2 — KV heads (the big one)

64 kv heads (group  1):   687.2 GB of KV cache     1.0x  (MHA)
 8 kv heads (group  8):    85.9 GB of KV cache     8.0x smaller
 1 kv heads (group 64):    10.7 GB of KV cache    64.0x smaller

Grouped-Query Attention shares one KV head across a group of query heads. Quality barely moves; the cache shrinks by exactly the group factor.

Translate that into what a serving team actually cares about — concurrent requests on 8×H100 serving a 70B model at 8k context: 22 with MHA, 182 with GQA-8. An 8× throughput difference from one integer in a config file, chosen months before the model exists.

This is the single clearest example of inference co-design there is. It is also irreversible: the KV projections have different shapes, so you cannot convert afterwards without retraining.

Lever 3 — depth vs width

At fixed N you can be deep and narrow or shallow and wide.

• Depth is serial. Layer k+1 cannot start until layer k finishes. Depth costs decode latency directly, and adds pipeline stages that must synchronize.
• Width is parallel and produces larger, more efficient matmuls.

Deeper models are often slightly better per parameter, so there is a real trade. The co-design heuristic: as wide as quality allows, as shallow as quality tolerates.

Chapter 8: The Decode Wall

The most consequential single fact in LLM serving.

def decode_batch_intensity(n_params, n_layers, n_kv_heads, d_head, seq_len, batch):
    """Weights are read ONCE and shared across the batch; KV is per request."""
    flops = 2.0 * n_params * batch
    kv = kv_cache_bytes(n_layers, n_kv_heads, d_head, seq_len, batch)
    return flops / (n_params * 2 + kv)

H100 ridge point: 296 FLOP/byte
batch=    1: AI=    1.0 FLOP/byte  MEMORY bound  ( 0.3% of the way to the ridge)
batch=    8: AI=    6.9 FLOP/byte  MEMORY bound  ( 2.3%)
batch=   64: AI=   28.7 FLOP/byte  MEMORY bound  ( 9.7%)
batch=  256: AI=   43.3 FLOP/byte  MEMORY bound  (14.7%)
batch= 1024: AI=   49.6 FLOP/byte  MEMORY bound  (16.8%)

Even at batch 1024, decode reaches only 17% of the way to the ridge. Decode does not become compute-bound at any batch size you would actually run.

Why batching stops helping

Look at the shape of that curve — 1 → 8 gives 6.9×, but 256 → 1024 gives only 1.15×. The reason is in the denominator:

bytes = (weights, FIXED)  +  (KV cache, GROWS WITH BATCH)

At small batch, the fixed weight term dominates and batching amortizes it beautifully. At large batch, the KV term dominates and grows with the batch, so the ratio saturates. The KV cache is what limits batching, which is why PagedAttention, prefix sharing, and GQA are all ultimately about the same thing.

Everything that follows from this one fact

Every one of those is an attack on bytes, not FLOPs. That is what "memory-bound" means in practice, and why Feinberg's third vertical (quantization) is an energy and bandwidth lever before it is a memory-capacity one.

Chapter 9: What Co-Design Actually Looks Like as a Job

Pulling it together into the workflow a pre-training lead runs.

1. Get the product's latency budget and expected traffic shape.
      "sub-second turns, 8k incremental context, 128-token actions"

2. Compute the napkin for candidate model sizes.
      -> a table of (N, chips needed, chips to hold weights)

3. Find the largest N that meets the budget at acceptable cost.
      -> this is now a CONSTRAINT on the pre-training recipe

4. Feed that back into the scaling law (Phase 01).
      "we can afford 17B active parameters; what is the best 17B we can train
       with our compute budget, and how much loss do we give up vs Chinchilla?"

5. Choose shapes inside that budget.
      n_kv_heads (biggest lever), tile-aligned dimensions, depth/width,
      dense vs MoE (Phase 02: active params drive FLOPs, total drives memory)

6. Verify with a roofline + MFU budget on the real hardware, then iterate.

Step 4 is the seam where this phase meets Phase 01, and it is the actual job. The scaling law says "for this compute budget, N = 92B minimizes loss." The napkin says "92B needs 3 chips just to hold and blows the latency budget." So you deliberately undershoot Chinchilla, train a smaller model on more tokens, and accept slightly worse loss for dramatically cheaper serving.

That deliberate deviation is inference-aware scaling, and it is why Feinberg's slide notes that Chinchilla-style scaling "ignores inference cost."

Lab Walkthrough

Lab 01 — Roofline, MFU Budget & the Latency Napkin

Implement in this order:

1. arithmetic_intensity, ridge_point, roofline_throughput, roofline_report. The roofline is one min(); the value is the lever verdict it returns.
2. mfu, hfu, mfu_budget. test_hfu_is_always_at_least_mfu is the one to internalize.
3. prefill_seconds, decode_seconds, interactive_latency, chips_for_latency_budget, weights_fit_chips.
4. tile_efficiency, kv_cache_bytes, gqa_saving, decode_batch_intensity, depth_vs_width.

The money tests:

• test_prefill_reproduces_the_talks_number — ~5.8 s on one v5e chip.
• test_a_4x4_station_brings_prefill_under_the_half_second_limit.
• test_a_smaller_model_needs_fewer_chips — the whole economic argument, in one assertion.
• test_decode_stays_memory_bound_even_at_huge_batch.

The traps:

• Prefill is 2N per token (forward only), not 6N. That is training.
• Decode time is driven by bytes, not FLOPs. If your decode_seconds has peak in it, it is wrong.
• chips_for_latency_budget must return None rather than loop forever — "no amount of hardware fixes this" is a real and important answer.
• A budget below the scaffolding overhead is impossible; raise rather than return something.

Success Criteria

• LAB_MODULE=solution pytest test_lab.py -v → 50 passed.
• Your lab.py reaches 50 passed.
• python solution.py runs and you can explain all nine sections.
• You can compute a ridge point and explain why H100's is higher than A100's.
• You can produce an MFU budget and name the biggest lever from it.
• You reflexively ask "MFU or HFU?" when shown a utilization figure.
• You can reproduce the ~5.8 s single-chip prefill number and the 4×4 conclusion.
• You can explain why decode beats prefill at batch 1, and the batching caveat.
• You can state the GQA saving as a ratio and as concurrent requests.

Interview Q&A

Q: What is the roofline model and what is it for? Plot achievable throughput against arithmetic intensity — FLOPs per byte moved. Below the ridge point (peak FLOP/s ÷ bandwidth) you are memory-bound and throughput is bandwidth × intensity; above it you are compute-bound and capped at peak. Its value is that it tells you which lever to pull before you spend a week optimizing: below the ridge, reduce bytes; above it, reduce FLOPs. It also gives you a ceiling, which distinguishes "we implemented this badly" from "this is what the algorithm costs."

Q: Our training run is at 38% MFU. Is that bad? No — that is the normal band. MFU is the fraction of peak matmul throughput achieved, and a transformer is not pure matmul: it runs vector ops (norms, activations, softmax), moves activations to and from HBM, runs collectives, and executes the optimizer step. Each of those is time the matmul unit is idle. The useful move is to decompose it — matmul / vector / memory / comms / optimizer — because the breakdown is an optimization agenda. And I would check whether that 38% is MFU or HFU: HFU counts activation recomputation as useful work and is always higher, typically by about a third.

Q: Why is decoding so much slower than prefill per token? Different bottlenecks. Prefill processes the whole prompt at once, so each weight read is amortized across thousands of tokens — arithmetic intensity is in the hundreds and it is compute-bound. Decode produces one token at a time, so it reads every weight in the model to do 2N FLOPs — intensity around 1, memory-bandwidth-bound. On a 70B model that is 140 GB of traffic per token. It is not a kernel problem; it is arithmetic.

Q: How would you make decode faster? Attack bytes, not FLOPs. Batching, to amortize the weight read across concurrent requests — the single biggest lever. Quantization, to halve or quarter the dominant term. GQA or MQA, to shrink the KV cache, which is what limits how large a batch you can fit. Speculative decoding, which verifies several tokens per weight-read pass. And ultimately a smaller model. Notably, batching saturates: past a few hundred, the KV term grows with the batch and the intensity curve flattens.

Q: You need sub-second agent turns from a 70B model. Walk me through it. Napkin first. 8k incremental prefill at 2N per token is 1.1 PFLOPs; on a v5e at 197 TFLOP/s that is ~5.8 s on one chip, so 16 chips gets prefill under half a second. But 128 decode tokens at batch 1 is 17.9 TB of memory traffic — about 3.8× the prefill cost — so the full turn needs around 64 chips. Also, 140 GB of weights on 16 GB chips means 9 chips just to hold the model. Then the real conclusion: 64 chips per concurrent conversation is not a viable product, so the answer is not more hardware, it is a smaller model. Halving N halves both prefill and decode. That is the economic argument for a Flash-class model, and it is a pre-training decision, not a serving one.

Q: What is inference co-design and why does the pre-training team own it? Choosing architecture shapes with the serving target in mind — matrix dimensions that tile cleanly onto the hardware, KV-head sharing, depth versus width, dense versus MoE. The pre-training team owns it because these are all frozen the moment training starts. A serving engineer inherits n_kv_heads; they cannot change it without retraining. And it is a huge lever: 64 KV heads versus 8 is an 8× difference in concurrent requests at essentially no quality cost.

Q: Why is the H100's ridge point higher than the A100's? Because compute scaled faster than bandwidth: 3.2× versus 1.7× between those generations. The ridge is peak ÷ bandwidth, so it moved right. The practical consequence is that memory-bound workloads — which is all of decode — get relatively worse on newer hardware, not better. This is a long-running trend and it is why bandwidth-saving techniques keep gaining importance.

Tips & Takeaways

Tips

• Compute arithmetic intensity before optimizing anything. It tells you the ceiling and the lever in one number.
• Always ask "MFU or HFU?" and "at what precision?" Both change the denominator.
• Never quote a utilization number without the breakdown. The breakdown is the useful part.
• Do the napkin before choosing a model, not after. It takes two minutes and routinely changes the answer.
• Check "does it even fit?" first. Capacity often binds before latency does.
• Treat prefill and decode as separate systems with separate budgets and separate hardware provisioning.
• Write down n_kv_heads in the design doc with its serving justification. It is the highest-leverage irreversible number in the config.
• Prefer tile-aligned dimensions. Free performance; costs nothing but attention.

Takeaways

1. A chip is several machines; "peak" describes only the matmul unit.
2. Arithmetic intensity decides which resource you are fighting. Below the ridge, cut bytes.
3. Newer hardware has a higher ridge point — memory-bound work gets relatively worse.
4. MFU of 35% is an accounting identity. The decomposition is your work queue.
5. HFU ≥ MFU always. Ask which.
6. Prefill is compute-bound; decode is memory-bound. Two machines, opposite optimizations.
7. Pipelining helps prefill and hurts decode. This is why disaggregation exists.
8. Decode never reaches the ridge at any realistic batch size — even 1024.
9. The KV cache is what makes batching saturate, which is why GQA matters so much.
10. When latency cannot be met, the answer is a smaller model. That is a pre-training decision.

References

• Feinberg, Gemini Pretraining, Princeton, Apr 2025 — slides — "Small Model Customers", "Why Do Real-time Use Cases Imply Smaller Models", and the v5e napkin math
• Developing Dev interview · video — the MFU discussion and the inference co-design vertical
• Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 2009
• Pope et al., Efficiently Scaling Transformer Inference, 2022 — https://arxiv.org/abs/2211.05102
• Austin et al., How To Scale Your Model — https://jax-ml.github.io/scaling-book/
• Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, 2022 — https://arxiv.org/abs/2205.05198 (MFU vs HFU)
• Shazeer, Fast Transformer Decoding (MQA), 2019 — https://arxiv.org/abs/1911.02150
• Ainslie et al., GQA, 2023 — https://arxiv.org/abs/2305.13245
• Dao et al., FlashAttention, 2022 — https://arxiv.org/abs/2205.14135 (the canonical memory-traffic win)
• Kwon et al., PagedAttention / vLLM, 2023 — https://arxiv.org/abs/2309.06180
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022 — https://arxiv.org/abs/2211.17192
• Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit, 2017 — https://arxiv.org/abs/1704.04760