Diagrams 01 — Inference Internals

Diagrams 1–6: how a prompt becomes tokens and tokens become output, and the systems tricks that make it fast. Pairs with Phase 2, Phase 6, and What Happens When You Prompt an LLM Agent.

1. Transformer inference flow

 prompt text
    │
    ▼
[ Tokenizer ]  text → token IDs
    │
    ▼
[ Embedding ]  IDs → vectors (+ positional info)
    │
    ▼
┌─────────────── N × Transformer blocks ───────────────┐
│  [ Self-Attention ] each token attends to all others  │
│         │   (uses/writes KV-cache, diagram 4)         │
│         ▼                                             │
│  [ Feed-Forward (MLP / MoE experts) ]                 │
│         │  (+ residual + layer norm around each)      │
└──────────────────────┬────────────────────────────────┘
                       ▼
[ Final LayerNorm ] → [ LM Head ] → logits (score per vocab token)
                       │
                       ▼
[ Sampling ] (temperature/top-p/top-k) → next token ID
                       │
                       ▼
            detokenize → append → repeat (autoregressive)

flowchart TD
  P[Prompt text] --> T[Tokenizer: text → IDs]
  T --> E[Embedding + positional]
  E --> B[N × Transformer blocks]
  subgraph Block
    A[Self-Attention\nuses KV-cache] --> F[Feed-Forward MLP/MoE]
  end
  B --> LN[Final LayerNorm]
  LN --> H[LM Head → logits]
  H --> S[Sampling temp/top-p/top-k]
  S --> O[Next token ID]
  O -->|append, autoregressive| E

2. Tokenization to generation

"What is RAG?"
   │  tokenizer (BPE)
   ▼
[ "What", " is", " R", "AG", "?" ]  →  IDs [ 3923, 374, 432, 1929, 30 ]
   │  model forward (prefill all prompt tokens)
   ▼
logits → sample → " R"           ← first output token
   │  append, forward 1 token (decode)
   ▼
" RAG" → " is" → " retrieval" → ... → [EOS]
   │  detokenize
   ▼
"RAG is retrieval-augmented generation..."

flowchart LR
  A["What is RAG?"] --> B[BPE tokens]
  B --> C[Token IDs]
  C --> D[Prefill: forward all prompt tokens]
  D --> E[Logits → sample first token]
  E --> F[Decode: 1 token at a time]
  F -->|until EOS| F
  F --> G[Detokenize → output text]

3. Prefill vs decode

        PREFILL (once, the prompt)              DECODE (per output token)
   ┌──────────────────────────────┐      ┌──────────────────────────────┐
   │ process ALL prompt tokens in │      │ process ONE token at a time   │
   │ parallel                     │      │ (sequential)                  │
   │ COMPUTE-bound (matmul heavy) │      │ MEMORY-BANDWIDTH-bound        │
   │ builds the KV-cache          │      │ reads weights+KV each step    │
   │ → determines TTFT            │      │ → determines TPOT / tok-per-s │
   └──────────────────────────────┘      └──────────────────────────────┘
   tok/s(decode) ≈ memory_bandwidth ÷ model_bytes     (the bandwidth law)

flowchart LR
  subgraph Prefill [Prefill — once]
    P1[All prompt tokens in parallel] --> P2[Compute-bound] --> P3[Build KV-cache] --> P4[→ TTFT]
  end
  subgraph Decode [Decode — per token]
    D1[One token at a time] --> D2[Memory-bandwidth-bound] --> D3[Reuse KV-cache] --> D4[→ TPOT]
  end
  Prefill --> Decode

4. KV cache

Without KV-cache: every new token re-computes attention over ALL previous tokens → O(n²) waste.
With KV-cache: store each token's Key & Value once; new token attends to cached K/V → O(n) per step.

 step t:   new token's Query  ──▶  attend over  [K0 K1 ... K(t-1)]  (cached)
                                                 [V0 V1 ... V(t-1)]  (cached)
           then append (K_t, V_t) to the cache.

 KV-cache memory ≈ 2 × layers × heads × head_dim × seq_len × batch × bytes
   → grows with context length × batch  → the real VRAM cost of long context

flowchart TD
  Q[New token Query_t] --> ATT[Attention]
  KC[(KV-cache: K0..K t-1, V0..V t-1)] --> ATT
  ATT --> OUT[Token output]
  Q --> APP[Append K_t, V_t to cache]
  APP --> KC

5. PagedAttention

Problem: KV-cache per request is variable-length → contiguous allocation fragments memory,
         wasting VRAM and limiting how many requests fit (low throughput).

PagedAttention (vLLM): split KV-cache into fixed-size BLOCKS (like OS memory pages),
         mapped by a per-sequence block table. No fragmentation; share blocks across requests.

  Sequence A blocks:  [#3][#7][#1]            Physical KV blocks (pool)
  Sequence B blocks:  [#2][#5]            →   [#0][#1][#2][#3][#4][#5][#6][#7]...
  (block table maps logical → physical)       (allocated on demand, freed on finish)

  Result: high GPU memory utilization → many concurrent sequences → CONTINUOUS BATCHING

flowchart LR
  subgraph SeqA[Sequence A block table]
    A1[blk 3] --> A2[blk 7] --> A3[blk 1]
  end
  subgraph SeqB[Sequence B block table]
    B1[blk 2] --> B2[blk 5]
  end
  SeqA --> POOL[(Physical KV block pool)]
  SeqB --> POOL
  POOL --> R[High utilization → continuous batching]

6. Speculative decoding / MTP

Plain decode: big model generates 1 token per forward pass (slow, sequential).

Speculative decoding:
  1) small DRAFT model proposes k tokens quickly:   t1 t2 t3 t4
  2) big TARGET model verifies all k in ONE parallel pass
  3) accept the longest correct prefix; reject the rest; continue
     → fewer big-model steps for the same output (same distribution) → lower latency

   draft:  [t1][t2][t3][t4]        (cheap, fast)
   verify:  ✓   ✓   ✓   ✗          (one big-model pass)
   keep:   [t1][t2][t3] + resample from target at the mismatch

flowchart LR
  D[Draft model: propose k tokens] --> V[Target model: verify k in parallel]
  V --> A{Accept prefix?}
  A -->|accepted t1..t3| K[Keep tokens]
  A -->|reject t4| RS[Resample from target]
  K --> D
  RS --> D

Next: Diagrams 02 — Serving & Gateways · Concepts: Phase 2, Phase 6.07, Phase 7.04