Prefill vs Decode

Phase 2 · Document 07 · Transformer Foundations Prev: 06 — KV Cache · Next: 08 — MoE and Dense Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Every LLM request has two phases with opposite performance profiles, and almost every latency/cost decision comes down to knowing which one you're fighting. Prefill processes the whole prompt in parallel and sets TTFT; decode generates tokens one at a time and sets TPOT. They are bottlenecked by different hardware resources (compute vs memory bandwidth), so they're optimized by different techniques. Confuse them and you'll "optimize" the wrong thing — buying compute when you're bandwidth-bound, or shrinking prompts when the problem is output length. This is the unifying lens for Phase 7 serving.

2. Core Concept

The two phases

Prefill — the model ingests the entire prompt in one parallel forward pass, computing and caching K/V for all prompt tokens (06). Because all tokens are processed at once, the GPU's matrix units are saturated: prefill is compute-bound (FLOPs). It scales with prompt length and largely determines TTFT (time to first token).

Decode — the model then generates output one token at a time (05). Each step is a single-token forward pass that must read the entire model weights and the whole KV cache from memory to produce one token. There's little arithmetic per step but lots of data movement: decode is memory-bandwidth-bound. It determines TPOT (time per output token).

total_latency ≈ TTFT(prefill) + TPOT(decode) × output_tokens

Why the bottleneck differs (the key insight)

• Prefill does ~prompt_len tokens' worth of matmuls in parallel → high arithmetic intensity → limited by GPU FLOPs.
• Decode does 1 token of arithmetic but must stream all weights + KV from VRAM each step → low arithmetic intensity → limited by memory bandwidth.

This is why a GPU can prefill thousands of tokens quickly but generates only tens-to-hundreds of tokens/sec per request: decode underuses the compute units while saturating memory bandwidth.

Consequences and optimizations

• Batching helps decode most: since each decode step reads the full weights anyway, serving many requests in the same step amortizes that read across them → big throughput win (continuous batching, Phase 1.05).
• Prefill can starve decode: a giant prompt's prefill can stall other requests' decoding; chunked prefill interleaves them for fairness (06).
• Disaggregated serving (advanced): some systems run prefill and decode on separate hardware pools because their resource profiles differ so much.

3. Mental Model

PREFILL  =  read & understand the whole question at once   → COMPUTE-bound  → sets TTFT
DECODE   =  write the answer one word at a time            → BANDWIDTH-bound → sets TPOT

   prompt ──[prefill: 1 big parallel pass, fills KV]──► first token  (TTFT)
                                  │
                                  ▼
            ──[decode: 1 token/step, re-reads weights+KV each step]──► … (TPOT × N)

   total ≈ TTFT + TPOT × output_tokens

   Long PROMPT → prefill/TTFT problem  → cache the prefix, shorten input.
   Long OUTPUT → decode/TPOT problem   → fewer tokens, speculative decoding, smaller model.
   Many USERS  → batch the decode      → amortize weight reads (throughput).

4. Hitchhiker's Guide

What to understand first: which phase your latency problem is in. Long prompt + slow start → prefill/TTFT. Long answer + slow stream → decode/TPOT.

What to ignore at first: disaggregated prefill/decode serving and arithmetic-intensity math — know the direction (compute vs bandwidth) and move on.

What misleads beginners:

• Treating "tokens/sec" as one number — prefill throughput and decode throughput are wildly different.
• Trying to fix slow streaming by shortening the prompt (that's prefill, not decode).
• Buying more compute when decode is bandwidth-bound (a faster-FLOPs GPU with the same bandwidth barely helps decode).

How experts reason: they decompose latency into TTFT + TPOT×N, attribute each to prefill/decode, and apply the matching lever (cache the prefix vs speculative-decode the output). They know batching mainly helps decode.

What matters in production: separate TTFT and TPOT SLOs; prefix/prompt caching for prefill-heavy (long, repeated prompts) workloads; speculative decoding + batching for decode-heavy (long-output, high-concurrency) ones.

How to verify: stream a request and measure TTFT vs inter-token latency; vary prompt length (moves TTFT) vs max_tokens (moves total via TPOT×N) and watch which metric responds.

Questions to ask: What are your p95 TTFT and TPOT? Is prompt/prefix caching available (prefill)? Speculative decoding (decode)? How does batching affect per-request TPOT under load?

What silently gets expensive: long shared prompts without prefix caching (repeated prefill); long outputs/reasoning tokens (decode); big prompts stalling co-located requests without chunked prefill.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Every request is prefill then decode; total ≈ TTFT + TPOT × output_tokens.
• Prefill is compute-bound (FLOPs) and scales with prompt length → drives TTFT.
• Decode is memory-bandwidth-bound (reads all weights + KV per token) → drives TPOT.
• Batching helps decode most by amortizing the per-step weight read across requests.
• Quantization speeds decode chiefly by reducing bytes streamed per step (and saves memory).
• Speculative decoding/MTP attack decode by emitting >1 token per model step.
• Prefix/prompt caching attacks prefill by skipping recomputation of repeated prefixes.
• A faster-FLOPs GPU with the same bandwidth barely improves decode — match the resource to the phase.

9. Observations from Real Systems

• vLLM/TGI/SGLang schedule prefill and decode together with continuous batching and offer chunked prefill to stop big prompts from stalling decodes.
• Provider latency pages report TTFT and inter-token latency separately — the prefill/decode split, surfaced.
• Prompt caching (OpenAI/Anthropic) is a prefill optimization (and a cost discount); speculative decoding (vLLM) is a decode optimization.
• Splitwise / disaggregated serving runs prefill and decode on different GPU pools precisely because their resource profiles differ.
• Reasoning models are decode-heavy (long hidden token streams) → TPOT dominates their latency (09).

10. Common Misconceptions

11. Engineering Decision Framework

Attribute the latency problem:
  Slow to START (high TTFT)?         → PREFILL (compute, prompt length).
     → prompt/prefix caching, shorter prompt, FlashAttention, chunked prefill.
  Slow to STREAM / long total (high TPOT×N)? → DECODE (bandwidth, output length).
     → fewer output tokens, speculative decoding/MTP, smaller model, quantization.
  Low throughput under many users?   → batch the DECODE (continuous batching).

Set SLOs separately for TTFT and TPOT; optimize the binding one.

Match hardware to phase:
  prefill-heavy → compute (FLOPs).   decode-heavy → memory bandwidth.

12. Hands-On Lab

Goal

Measure TTFT and TPOT separately, then prove that prompt length moves TTFT while output length moves total time via TPOT×N.

Prerequisites

• Python 3.10+, an OpenAI-compatible streaming endpoint (hosted or local vLLM/llama.cpp).

Setup

pip install openai
export OPENAI_BASE_URL=... OPENAI_API_KEY=...

Steps

import time
from openai import OpenAI
client = OpenAI()

def measure(prompt, max_tokens):
    t0 = time.perf_counter(); first=None; n=0
    for ch in client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role":"user","content":prompt}],
        max_tokens=max_tokens, stream=True):
        if ch.choices[0].delta.content:
            if first is None: first = time.perf_counter()
            n += 1
    end = time.perf_counter()
    return (first-t0)*1000, (end-first)/max(n-1,1)*1000, (end-t0)*1000  # TTFT, TPOT(ms), total

short = "Summarize the benefits of unit tests."
long  = short + " " + ("Context: " + "lorem ipsum "*800)  # ~big prompt

print("vary PROMPT length (fixed output):")
for tag, p in [("short", short), ("long", long)]:
    ttft, tpot, tot = measure(p, 64)
    print(f"  {tag:5} TTFT={ttft:.0f}ms TPOT={tpot:.1f}ms total={tot:.0f}ms")

print("vary OUTPUT length (fixed short prompt):")
for mt in (32, 128, 512):
    ttft, tpot, tot = measure(short, mt)
    print(f"  max={mt:>3} TTFT={ttft:.0f}ms TPOT={tpot:.1f}ms total={tot:.0f}ms")

Expected output

• Longer prompt → higher TTFT, ~unchanged TPOT.
• Larger max_tokens → ~unchanged TTFT, larger total (≈ TTFT + TPOT×N).

Debugging tips

• TTFT ≈ 0 / weird TPOT? Confirm you're actually streaming (stream=True).
• Prompt-caching providers may flatten TTFT for the repeated long prompt — that itself is the prefill-caching lesson.

Extension task

Re-run the long prompt twice in a row on a prompt-caching provider; show TTFT drops on the cached second call (prefill optimization in action).

Production extension

Aggregate p50/p95 TTFT and TPOT over a small load test at concurrency 1/4/16 and note where each SLO breaks (ties into Phase 1.05/Phase 7).

What to measure

TTFT vs prompt length; total vs output length; TPOT stability; (extension) prefix-cache TTFT drop.

Deliverables

• A table separating TTFT and TPOT effects of prompt length vs output length.
• A note assigning each observed cost to prefill or decode and naming the right lever.

13. Verification Questions

Basic

1. What does prefill do vs decode, and which metric does each set?
2. Why is decode memory-bandwidth-bound while prefill is compute-bound?
3. Write the total-latency formula.

Applied 4. A user complains the answer "takes forever to start." Which phase, and two fixes? 5. A user complains "it types slowly once it starts." Which phase, and two fixes?

Debugging 6. You upgraded to a higher-FLOPs GPU with the same memory bandwidth and decode barely improved. Why? 7. Big prompts intermittently stall other users' streaming. Cause and fix?

System design 8. Design serving for a long-shared-prompt, short-answer product vs a short-prompt, long-report product. Different levers for each.

Startup / product 9. Your support bot reuses a 3,000-token system prompt on every call and answers briefly. Which phase dominates cost, and what's the single biggest lever?

14. Takeaways

1. Every request is prefill (compute, →TTFT) then decode (bandwidth, →TPOT).
2. total ≈ TTFT + TPOT × output_tokens — attribute latency to a phase, then fix that phase.
3. Prompt length → prefill/TTFT; output length → decode/TPOT.
4. Prefix/prompt caching optimizes prefill; speculative decoding/MTP, quantization, batching optimize decode.
5. Batching helps decode most (amortizes per-step weight reads).
6. Match hardware to phase — compute for prefill, bandwidth for decode.

15. Artifact Checklist

• Code: TTFT/TPOT measurement harness.
• Table: prompt-length effect (TTFT) vs output-length effect (total).
• Notes: the compute-vs-bandwidth distinction and matching levers.
• (Extension) Prefix-cache TTFT-drop demonstration.
• Decision record: levers for one prefill-heavy and one decode-heavy workload.
• Diagram: the two-phase timeline with TTFT/TPOT labeled.

Next: 08 — MoE and Dense Models