MTP and Speculative Decoding

Phase 6 · Document 07 · Local Inference Prev: 06 — Quantization Guide · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A claim like "Gemma MTP — run Gemma GGUFs locally at ~2× faster speed" is exactly the kind of marketing you must be able to decode and verify rather than take on faith. The "2×" comes from speculative decoding / Multi-Token Prediction (MTP) — clever techniques that produce multiple tokens per expensive step to beat the fundamental decode bottleneck (01, Phase 2.07). They're real and powerful, but the speedup is conditional — it depends on the prompt, the acceptance rate, batch size, and memory. This doc explains the mechanism from zero, why it works, the exact terms (draft/verifier/acceptance rate, MTP heads), and when the "2×" does and doesn't hold — turning a screenshot into an engineering judgment.

2. Core Concept

Plain-English primer: why decode is slow, and the trick to beat it

Generating text is autoregressive: the model produces one token, appends it, and runs the entire network again for the next (Phase 2.05). Each step must read (almost) all the weights from memory, so decode is memory-bandwidth-bound — and crucially, a forward pass that produces one token costs almost the same as one that checks several tokens, because the bottleneck is reading the weights, not the math (what-happens §1.D).

That asymmetry is the whole opportunity: if we could cheaply guess the next few tokens, we could verify all the guesses in a single forward pass and accept the ones the model agrees with — getting multiple tokens for the price of one step.

Speculative decoding (draft + verify)

The classic form uses two models:

• a small, fast draft model that cheaply proposes the next K tokens (e.g., K=4), and
• the real, large target/verifier model that, in one forward pass, scores all K proposed tokens at once.

The verifier accepts the longest prefix of the draft that matches what it would have produced (via a probabilistic acceptance test that guarantees the output distribution is identical to the target model's — speculative decoding is lossless, not an approximation). On a mismatch it rejects the rest and emits its own correct token, then drafting resumes.

draft (cheap) proposes:   the  cat  sat  on
verifier (1 pass) checks: the  cat  sat  XX   → accept "the cat sat", reject "on",
                                                emit verifier's token → 3–4 tokens, ~1 big step

Acceptance rate = the fraction of drafted tokens the verifier accepts. It's the master dial:

speedup ≈ (accepted tokens per verify step)  ÷  (1 + draft overhead)

High acceptance (draft agrees with target) → near-K× fewer big steps. Low acceptance (draft is bad / text is unpredictable) → you waste draft work and gain little. The draft must be fast and well-aligned with the target (often a smaller model from the same family).

MTP (Multi-Token Prediction) — self-speculation

A draft model is extra memory and another thing to align. MTP removes the separate draft by giving the target model itself extra lightweight prediction heads that propose the next few tokens in one shot — then the model verifies them like above. It's self-drafting: no second model, the "draft" comes from the model's own extra heads (this is the lineage of Medusa and EAGLE, and what "Gemma MTP" refers to). Some models are even trained with an MTP objective (e.g., DeepSeek-V3), which both improves training signal and provides ready-made speculation heads.

Trade-off vs draft-model speculation: MTP needs no separate model to host/align (simpler, less memory for a draft), but the extra heads add some memory and a bit of compute, and acceptance still governs the win.

Why "2× faster" is conditional

The headline speedup holds under the benchmark's conditions and erodes under others:

• Acceptance rate varies by content — predictable/boilerplate text (code, structured output) drafts well (high speedup); novel/creative text drafts poorly (low speedup).
• Batch size: speculation shines at low batch / single-stream (where you're bandwidth-bound and the GPU is underused). At high batch, the GPU is already saturated doing useful work for many users, so speculation's spare-capacity trick yields less or no gain — sometimes a regression (Phase 7.03).
• Memory: the draft model or MTP heads + their KV add memory (02) — on a tight box that's a real cost.
• Verification cost: checking K tokens isn't exactly free; very large K with low acceptance loses.
• Per-prompt variance: "2×" is an average — a given prompt/device/batch may see 1.2× or 3×.

So: lossless quality, real average speedup at low batch, but verify on your workload.

3. Mental Model

   NORMAL DECODE:  [big step]→t1  [big step]→t2  [big step]→t3   (1 token / big step)

   SPECULATIVE:    draft proposes t1..t4 (cheap)
                   ONE big step VERIFIES all 4  → accept t1 t2 t3, reject t4
                   ⇒ ~3 tokens / big step      (because verifying many ≈ cost of one)

   DRAFT-MODEL spec:  small model = the drafter   (extra model, must align)
   MTP / self-spec:   the model's OWN extra heads = the drafter  (no 2nd model) ← "Gemma MTP"

   speedup ≈ accepted_tokens_per_step      master dial = ACCEPTANCE RATE
   shines at LOW batch (GPU underused); fades at HIGH batch (GPU already busy)
   LOSSLESS: output distribution == target model's

Mnemonic: guess cheap, verify in one pass, keep what the big model agrees with. Win = acceptance rate; best at low batch; quality unchanged.

4. Hitchhiker's Guide

What to look for first: is the speedup from a draft model or MTP/self-speculation, and what acceptance rate does the workload get? Those determine the real win.

What to ignore at first: the specific algorithm family names (Medusa/EAGLE/Lookahead) — they're variations on draft-and-verify. Focus on acceptance × batch.

What misleads beginners:

• "2× always." It's conditional on content, batch, and acceptance — measure (Phase 4.03).
• "It changes the output / lowers quality." No — proper speculative decoding is distribution-identical to the target.
• "It's free." Draft model / MTP heads cost memory and some compute (02).
• "It helps my busy multi-user server." Often the opposite — gains concentrate at low batch.

How experts reason: they enable speculation for latency-sensitive, low-concurrency paths (interactive single-user, agent steps, code completion) where the GPU is underused; pick a well-aligned, much smaller draft (or use built-in MTP); tune K to the observed acceptance; and disable/measure it under high batch where it may not pay.

What matters in production: measure end-to-end tok/s and TTFT with speculation on vs off at your real batch sizes and content; watch the extra memory; ensure the draft stays aligned after model updates.

How to debug/verify: engines report acceptance rate — low acceptance = bad draft/alignment or unpredictable content. Compare tok/s on vs off; if off is faster, you're batch-saturated.

Questions to ask vendors: Is the speedup draft-based or MTP? Measured at what batch size and on what content? What acceptance rate? What extra memory? Does it hold at my concurrency?

What silently gets expensive/unreliable: speculation enabled on a high-throughput server (wasted compute), an unaligned draft after a model bump (acceptance collapses), and memory pressure from draft/heads on a tight box.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Decode is bandwidth-bound, so verifying many tokens costs ≈ one step — the basis of all speculation (Phase 2.07).
• Speculative decoding is lossless — the accept/reject test makes the output distribution identical to the target model's.
• Acceptance rate is the master dial: speedup ≈ accepted tokens per verify step.
• MTP = self-speculation via the model's own extra heads — no separate draft model (Gemma MTP, Medusa, EAGLE; DeepSeek-V3 trains an MTP objective).
• Draft model needs to be fast and aligned (usually a small same-family model); misalignment crushes acceptance.
• Gains concentrate at low batch / single-stream; at high batch the GPU is already busy and speculation can stop helping (Phase 7.03).
• It costs memory (draft model or MTP heads + KV) (02).
• "2×" is workload-specific — predictable text (code/structured) drafts well; creative text doesn't. Measure.

9. Observations from Real Systems

• Unsloth "Gemma MTP" GGUFs package MTP for llama.cpp to advertise ~2× local decode — verify the acceptance/batch conditions (Phase 3.05).
• vLLM supports both draft-model and self-speculative (num_speculative_tokens, draft configs) and documents the high-batch caveat explicitly (Phase 7).
• DeepSeek-V3 ships an MTP training objective and inference heads — a frontier example of built-in speculation.
• Code/autocomplete tools benefit most: predictable, high-acceptance text on latency-critical single-stream paths (Phase 11).
• Provider TPS claims often quietly assume speculation at low batch — a serving-fidelity/benchmark-reading nuance.

10. Common Misconceptions

11. Engineering Decision Framework

SHOULD I ENABLE SPECULATION / MTP?
 1. Is the path LATENCY-sensitive and LOW-concurrency (interactive, agent step, autocomplete)?
       NO (high-throughput, many users) → likely skip; measure, often no gain. [Phase 7.03]
       YES ↓
 2. Memory headroom for a draft model OR MTP heads + their KV?            [02]
       NO → skip or use lighter MTP.
       YES ↓
 3. Choose source:
       model ships MTP / self-spec heads → use them (simplest).
       else → pick a small, SAME-FAMILY draft model (alignment ⇒ acceptance).
 4. Tune K (num_speculative_tokens) to observed ACCEPTANCE rate.
 5. MEASURE tok/s + TTFT on vs off, on YOUR content + batch. Keep only if it wins.

12. Hands-On Lab

Goal

Measure the real speedup of MTP/speculative decoding on your hardware and show it depends on content and batch size — verifying or debunking a "2×" claim.

Prerequisites

• An engine that supports speculation: llama.cpp (draft model via -md/--model-draft, or an MTP GGUF) or vLLM (--speculative-config / num_speculative_tokens). A target model + a compatible small draft (or an MTP build).

Setup

# llama.cpp with a draft model (example)
llama-server -m target-7b-Q4_K_M.gguf -md draft-1b-Q4_K_M.gguf --draft-max 4 \
  -ngl 99 -c 8192 --port 8080
# or an MTP GGUF per the model's instructions (e.g., Unsloth Gemma MTP)

Steps

1. Baseline: generate with speculation off; record tok/s and TTFT on two prompt types: (a) predictable (write boilerplate code / fill a JSON schema), (b) creative (a novel short story).
2. Speculation on: repeat both prompts; record tok/s, TTFT, and the acceptance rate the engine reports.
3. Compute speedup per content type: tok/s_on ÷ tok/s_off. Expect higher for predictable text.
4. Batch sweep: if on vLLM, run 1, 4, 16 concurrent requests with speculation on vs off; show the gain shrinks as batch grows.
5. Memory: record the extra memory from the draft/MTP heads (02).

Expected output

A table: (content × batch) → tok/s on/off, speedup, acceptance, extra memory — demonstrating high gain at low batch on predictable text and diminishing gain at high batch / creative text.

Debugging tips

• Speedup < 1 (slower!) → acceptance too low (misaligned/too-large draft, or unpredictable content) or batch already saturated; lower K or disable.
• No acceptance reported → speculation isn't actually engaged; check flags/build.

Extension task

Vary K (draft length) and plot speedup vs acceptance — find the K that maximizes net tokens/step.

Production extension

Enable speculation only on the low-latency single-user route in a gateway and A/B it against the default path on live latency (Phase 8).

What to measure

tok/s on/off, TTFT, acceptance rate, speedup by content and batch, extra memory.

Deliverables

• A speedup report by content type and batch size.
• The acceptance rate observed and the chosen K.
• A recommendation: which routes should use speculation, and the memory cost.

13. Verification Questions

Basic

1. Why does verifying several tokens cost about the same as generating one?
2. What is the acceptance rate and why is it the master dial?
3. How does MTP differ from draft-model speculation?

Applied 4. Predict whether speculation helps more for code generation or poetry, and why. 5. Why might a "2× faster" claim vanish on your 32-user production server?

Debugging 6. Speculation makes generation slower. Two causes and fixes. 7. Acceptance rate is 20%. What does that imply and what would you change?

System design 8. Design which request routes in a multi-model service should use speculation, given mixed latency/throughput needs.

Startup / product 9. How would you present a "2× faster" feature honestly to customers, given its conditionality?

14. Takeaways

1. Decode is bandwidth-bound, so guess-then-verify yields multiple tokens per big step.
2. Speculative decoding is lossless; the acceptance rate sets the speedup.
3. MTP = self-speculation (extra heads, no second model) — what "Gemma MTP" means.
4. Gains concentrate at low batch + predictable text; high batch can negate them.
5. It costs memory and is workload-specific — always measure the "2×" on your content/batch.

15. Artifact Checklist

• A speedup-by-content table (predictable vs creative).
• A batch sweep showing the gain shrink at high concurrency.
• The observed acceptance rate and chosen K.
• The extra memory from draft/MTP heads.
• A route recommendation for where to enable speculation.

Up: Phase 6 Index · Next: 08 — Local Model Debugging