Lab 06 — Speculative Decoding & Disaggregation Simulator

Goal: turn the two most hyped serving techniques of the moment — speculative decoding (K01 §9) and prefill/decode disaggregation (the DistServe/Mooncake/Dynamo architecture, covered in the frontier Knowledge 09 module) — into numbers you can defend in front of a customer, using a deterministic analytic simulator. No GPU, no network, no dependencies: pure-stdlib Python, closed-form math, bit-for-bit reproducible on any laptop. Like Lab 05, this is pure analysis — and it is exactly the kind of back-of-envelope modeling a principal engineer does before anyone spends a week on a proof-of-concept.

Knowledge prereq: K01 — LLM Inference Internals (especially §7 batching math and §9 speculative decoding), K00 — Hardware Foundations (roofline, memory- vs compute-bound), K08 — Capacity Sizing (SLOs, goodput).

Stack: spec_decode_sim.py (stdlib only) + test_sim.py (pytest).

Table of Contents

• 1. Why this lab exists
• 2. Theory A: speculative decoding, derived
  • 2.1 The acceptance model and E[n]
  • 2.2 The cycle-time model
  • 2.3 Why the speedup collapses at high utilization
  • 2.4 The presets: small draft, MTP, Medusa, EAGLE-3
• 3. Theory B: colocated vs disaggregated serving, derived
  • 3.1 The two architectures
  • 3.2 M/M/c queues and Erlang-C, from zero
  • 3.3 The colocated model
  • 3.4 The disaggregated model and the KV-transfer arithmetic
  • 3.5 Goodput under an SLO pair
• 4. How to run
• 5. Reading the tables
• 6. Guided experiments
• 7. Model vs reality: what this simulator ignores
• 8. Success criteria
• 9. Interview Q&A
• 10. References

1. Why this lab exists

K01 §9 tells you that speculative decoding gives "2–3× on decode" and that it "helps less at high batch". The frontier material (EAGLE-3, DeepSeek's MTP, DistServe/Mooncake-style disaggregation) adds more claims. In a customer meeting, buzzwords die fast. The questions you will actually get are:

• "The vendor says speculative decoding gives 3×. We run at batch 64. Will we see 3×?" (Almost certainly not — and you must show why with arithmetic, not vibes.)
• "Should we split prefill and decode onto separate pools?" (It depends on the prompt/output mix, the SLO pair, the interconnect, and the fleet size — four numbers this lab teaches you to combine.)

Both questions have closed-form answers under honest, clearly-stated assumptions. This lab builds those closed forms, sweeps them, and makes you find the crossovers yourself. The deliverable mindset is the same as Lab 05's: a number you can defend, with its assumptions attached.

2. Theory A: speculative decoding, derived

Recap of the mechanism (K01 §9): a cheap draft proposes \(k\) tokens sequentially; the expensive target model verifies all of them in one parallel forward pass and keeps the longest accepted prefix, plus one token of its own (the corrected token at the first rejection, or the \(k{+}1\)-th token if everything was accepted). The rejection-sampling acceptance rule of Leviathan et al. and Chen et al. guarantees the output distribution is exactly the target model's — speculation changes latency, never quality.

2.1 The acceptance model and E[n]

Assume each drafted token is accepted independently with probability \(\alpha\) (the i.i.d. approximation — optimistic, see §7). Let \(A \in {0,\dots,k}\) be the length of the accepted prefix and \(n = A + 1\) the tokens emitted per cycle (the \(+1\) is the bonus token the verification pass always yields).

For a non-negative integer random variable, \(\mathbb{E}[A] = \sum_{i \ge 1} P(A \ge i)\). The prefix reaches length \(i\) only if the first \(i\) drafts are all accepted, so \(P(A \ge i) = \alpha^i\). Therefore

\[ \mathbb{E}[n] ;=; 1 + \sum_{i=1}^{k} \alpha^i ;=; \sum_{i=0}^{k} \alpha^i ;=; \frac{1 - \alpha^{k+1}}{1 - \alpha} \qquad (\alpha < 1), \]

by the finite geometric series \(\sum_{i=0}^{k} x^i = (1-x^{k+1})/(1-x)\). Sanity limits (all asserted in test_sim.py):

• \(\alpha = 0\): \(\mathbb{E}[n] = 1\) — every draft rejected, but the verify pass still emits one true target token. Spec decode never emits less than plain decoding per target pass; it can only cost more time.
• \(\alpha = 1\): \(\mathbb{E}[n] = k + 1\) — the theoretical ceiling.
• \(\mathbb{E}[n]\) is monotone increasing in both \(\alpha\) and \(k\), but saturates in \(k\): the marginal value of the \(k\)-th draft token is \(\alpha^k\), which shrinks geometrically. That is why an optimal, finite \(k^*\) exists.

2.2 The cycle-time model

One cycle = \(k\) sequential draft steps + one verification pass:

\[ T_{\text{cycle}} = k \cdot t_{\text{draft}} + t_{\text{verify}}(k), \qquad t_{\text{verify}}(k) = t_{\text{target}},(1 + c,k). \]

Why is \(t_{\text{verify}}\) close to a single target step even though it processes \(k{+}1\) tokens? Because a decode step is memory-bound (K01 §4, K00 roofline): its time is dominated by streaming the weights through HBM once. Verifying \(k{+}1\) tokens re-uses that same weight traffic (it is a micro-batch of width \(k{+}1\)), so the marginal cost is only the extra compute — captured by the small coefficient \(c\) (default 0.05). Define the draft-cost ratio \(d = t_{\text{draft}} / t_{\text{target}}\). The speedup vs plain autoregressive decoding (rate \(1/t_{\text{target}}\)) is

\[ S(k, \alpha, d, c) ;=; \frac{\mathbb{E}[n] / T_{\text{cycle}}}{1 / t_{\text{target}}} ;=; \frac{\sum_{i=0}^{k} \alpha^i}{,k,d + 1 + c,k,}. \]

Read the trade directly off the formula: the numerator saturates in \(k\); the denominator grows linearly in \(k\). Spec decode helps iff \(\sum_{i=0}^{k}\alpha^i > 1 + k(d + c)\) — and hurts otherwise (small \(\alpha\), expensive draft: test_spec_decode_can_hurt shows \(\alpha{=}0.3, d{=}0.5, k{=}4 \Rightarrow S = 0.45\), i.e. spec decode halves your throughput).

2.3 Why the speedup collapses at high utilization

The "verification is nearly free" argument holds only while the GPU has spare compute — i.e., while batched decode sits below the compute roof (K01 §7). Let \(u \in [0,1)\) be the fraction of the compute roof already consumed by the serving batch. The simulator gates the width term by the spare fraction:

\[ t_{\text{verify}}(k, u) = t_{\text{target}} \Big( 1 + \frac{c,k}{1 - u + \varepsilon} \Big), \qquad \varepsilon = 10^{-3}. \]

At \(u = 0\) this reduces to §2.2. As \(u \to 1\), the marginal compute of the \(k\) extra verified tokens can no longer hide under the memory-bound roof — it queues behind the batch's own compute, and the effective width cost inflates without bound. This is the quantitative version of K01's warning: spec decode spends spare compute to buy latency; at high batch there is no spare compute. With the default MTP-like preset the simulator shows the optimal \(k^*\) shrinking \(6 \to 4 \to 2 \to 1\) as \(u\) rises, and speedup crossing below 1.0 around \(u \approx 0.93\) — the point where a production scheduler should disable speculation. The gating function is a modeling choice (documented, monotone, saturating) — not microarchitecture; see §7.

2.4 The presets: small draft, MTP, Medusa, EAGLE-3

The presets are just parameter bundles \((\alpha, d, c)\) — read their docstrings in SPECDEC_PRESETS:

These reproduce the published pecking order (EAGLE-3 > MTP > separate draft > Medusa on speedup) from first principles, not by curve-fitting: acceptance dominates once the draft is cheap.

3. Theory B: colocated vs disaggregated serving, derived

3.1 The two architectures

Prefill and decode are different regimes: prefill is compute-bound and bursty; decode is memory-bound and steady (K01 §4).

• Colocated (vLLM default + chunked prefill, Sarathi-Serve): every node does both. Chunked prefill interleaves prompt chunks between decode steps so a long prompt doesn't stall everyone — but interleaving still steals decode step time, inflating TPOT.
• Disaggregated (DistServe, Mooncake, NVIDIA Dynamo): dedicated prefill pool and decode pool. Decode never sees prefill interference — but the prompt's KV cache must be shipped from a prefill node to a decode node, adding transfer time to TTFT, and the fixed fleet is split into two smaller pools (worse queueing statistics, §3.2).

3.2 M/M/c queues and Erlang-C, from zero

To model TTFT we need queueing theory's most useful workhorse. An M/M/c queue is defined by three assumptions: arrivals form a Poisson process with rate \(\lambda\) (M = memoryless interarrival times); each job's service time is exponentially distributed with rate \(\mu\) (mean service time \(S = 1/\mu\)); there are \(c\) identical servers pulling from one shared queue. Define the offered load \(a = \lambda/\mu\) ("Erlangs" — the average number of busy servers) and utilization \(\rho = a/c\). Stability requires \(\rho < 1\).

The Erlang-C formula gives the probability that an arriving job finds all \(c\) servers busy and must wait:

\[ E_C(c, a) ;=; \frac{\dfrac{a^c}{c!}}{;(1-\rho)\displaystyle\sum_{n=0}^{c-1} \frac{a^n}{n!} ;+; \frac{a^c}{c!};}. \]

From it, two facts we use directly (both standard results):

\[ \mathbb{E}[W_q] = \frac{E_C(c,a)}{c\mu - \lambda}, \qquad P(W_q > t) = E_C(c,a), e^{-(c\mu-\lambda)t}. \]

The code computes \(E_C\) via the numerically stable Erlang-B recursion \(B_0 = 1,; B_n = a B_{n-1}/(n + a B_{n-1})\), then \(E_C = B_c / (1 - \rho(1 - B_c))\) — no factorials overflow. test_mm2_hand_computed_value pins it to a by-hand M/M/2 case (\(a=1.5 \Rightarrow E_C = 9/14\), \(\mathbb{E}[W_q] = 9/7\) s).

One structural insight matters here: pooling wins. One shared \(c\)-server queue beats \(c\) separate single-server queues at equal total capacity (statistical multiplexing — an idle server can absorb anyone's burst). Disaggregation un-pools your fleet, and at small \(N\) that loss is real; the simulator makes it visible.

3.3 The colocated model

All \(N\) nodes serve both phases. With prompt length \(P\) and per-node prefill throughput \(R_p\) tok/s:

• Prefill service time \(S = P / R_p\); TTFT queueing is M/M/\(N\) with \(\mu = 1/S\): \(\text{TTFT} = W_q + S\).
• Prefill utilization \(\rho_p = \lambda S / N\); decode utilization \(\rho_d = \lambda O / (N R_d)\); stability needs both \(< 1\).
• Interference: chunked-prefill work stretches decode steps. The documented simple model:

\[ \text{TPOT}{\text{coloc}} = \text{TPOT}{\text{base}} \cdot (1 + \beta,\rho_p), \qquad \beta = 0.5 \text{ by default.} \]

More prefill load per node → more chunks interleaved between decode steps → proportionally inflated TPOT. \(\beta\) bundles chunk size and scheduler policy into one honest knob you can sweep (--beta).

3.4 The disaggregated model and the KV-transfer arithmetic

\(N_p\) prefill nodes + \(N_d\) decode nodes (\(N_p + N_d = N\); the simulator scans every split and keeps the goodput-best one). Prefill queueing is M/M/\(N_p\); decode runs interference-free at \(\text{TPOT}_{\text{base}}\). The new TTFT term is the KV shipment. From K01 §3:

\[ \text{KV bytes}(P) = 2 \cdot L \cdot H_{kv} \cdot d_{head} \cdot \text{bytes} \cdot P, \qquad t_{\text{xfer}} = \frac{\text{KV bytes}(P)}{BW_{\text{link}}}. \]

For Llama-70B GQA-8 (L=80, \(H_{kv}\)=8, \(d_{head}\)=128, FP16): \(2\cdot80\cdot8\cdot128\cdot2 = 327{,}680\) B/token = 320 KiB/token → 0.33 GB per 1k prompt tokens. Shipping that:

Same formula, three orders of magnitude — this table is why "just disaggregate" is not free advice. Over NVLink the transfer is noise; over commodity TCP it can eat the whole TTFT budget. (test_llama70b_1k_tokens_hand_calc pins these numbers.)

\[ \text{TTFT}{\text{disagg}} = W_q^{(N_p)} + S + t{\text{xfer}}. \]

3.5 Goodput under an SLO pair

Following K08 and Lab 03: raw throughput is meaningless; what sells is goodput — requests/s that meet both SLOs. Using the M/M/c waiting-time distribution from §3.2 (TTFT is \(W_q\) plus deterministic terms; TPOT is deterministic in this model):

\[ \text{goodput} = \lambda \cdot P(\text{TTFT} \le X) \cdot \mathbb{1}[\text{TPOT} \le Y], \qquad P(\text{TTFT} \le X) = 1 - E_C, e^{-(c\mu-\lambda)(X - S - t_{\text{xfer}})}. \]

This is where the two architectures' fates diverge: colocation risks the TPOT indicator (interference), disaggregation risks the TTFT probability (transfer + un-pooled queueing). The crossover is a fight between those two failure modes.

4. How to run

cd "AI Specialist/jd2-inference-optimization-track/labs/lab-06-spec-decode-and-disagg-sim"

# Part 1 — speculative decoding economics
python spec_decode_sim.py specdec                      # grid + presets + utilization sweep
python spec_decode_sim.py specdec --preset mtp         # utilization sweep for one preset
python spec_decode_sim.py specdec --util 0.7           # re-run the grid at high batch load

# Part 2 — colocated vs disaggregated
python spec_decode_sim.py disagg --preset chat         # also: rag | agent
python spec_decode_sim.py disagg --preset rag --link tcp
python spec_decode_sim.py disagg --preset chat --nodes 2 --link tcp

# Both parts with defaults
python spec_decode_sim.py all

# The proofs
python -m pytest test_sim.py -q                        # 22 deterministic tests, <0.1 s

No requirements.txt — stdlib only; pytest is the only tool you need beyond Python ≥ 3.10.

5. Reading the tables

specdec [1] Speedup grid — rows are draft length \(k\), columns acceptance \(\alpha\); * marks the optimal \(k\) per column. Watch two things: values rise then fall down each column (saturating numerator vs linear denominator, §2.2), and the * marches downward as \(\alpha\) grows — optimal draft length grows with acceptance.

specdec [2] Presets — best-\(k\) speedup per preset at \(u=0\) (idle-ish GPU) and \(u=0.7\) (busy fleet). The ranking is stable; the magnitude compresses with load.

specdec [3] Utilization sweep — the "when to turn it off" table: optimal \(k\) shrinks with \(u\) and the verdict flips to HURTS near saturation.

disagg table — one row per prompt/output mix (the preset's own mix plus prefill-heavy/balanced/decode-heavy at the same total tokens). For each: colocated mean TTFT (ms), TPOT (ms), goodput (req/s meeting both SLOs) side-by-side with the best disagg split (2p/6d = 2 prefill + 6 decode nodes). The [CROSSOVER] section scans link bandwidth and prints the smallest link at which disagg strictly beats colocated — the "disagg wins when…" sentence you'd put in a memo.

6. Guided experiments

1. Find where spec decode starts hurting under load. Run specdec --util 0.7 and compare with --util 0.0. At fixed \(k=4\) (defaults \(d=0.1, c=0.05\)), the denominator at \(u=0.7\) is \(1 + 0.266k \approx 2.06\), so speedup needs \(\sum_{0}^{4}\alpha^i > 2.06\) → spec decode at k=4 hurts below \(\alpha \approx 0.53\). Verify against the grid.
2. Find the utilization kill-switch. specdec --preset mtp: the verdict flips to HURTS between \(u = 0.90\) and \(u = 0.95\) even at \(\alpha = 0.85\). Expected conclusion: a latency feature, not a throughput feature — production schedulers should gate it on batch occupancy.
3. Rank the draft families. Compare presets at \(u=0\): eagle3 ≈ 3.1×, mtp ≈ 2.4×, small-draft ≈ 1.8×, medusa ≈ 1.6×. Now explain why medusa loses to small-draft despite a 2.7× cheaper draft (hint: \(\alpha\) is in the exponent, \(d\) is only linear).
4. Find the bandwidth where disagg stops winning for RAG. disagg --preset rag → crossover reports disagg wins when link ≥ 2.5 GB/s (805 ms KV transfer for the 6k prompt still fits the 2 s TTFT budget). Re-run with --link tcp and see the disagg column's TTFT blow up. Then reason: what changes if the TPOT SLO loosens from 28 ms to 32 ms? (Colocated interference — TPOT 29.8 ms — becomes SLO-legal, and colocated wins back the balanced mixes.)
5. Show pooling loss at tiny scale. disagg --preset chat --nodes 2 --link tcp: disagg's decode pool of one node can't even absorb \(\lambda O\) (unstable → goodput 0) while colocated pools both nodes and cruises. Expected conclusion: disaggregation is a scale play — below a handful of nodes the un-pooling tax dominates.

7. Model vs reality: what this simulator ignores

State these limits unprompted when you present numbers — that's what makes them defensible:

• i.i.d. acceptance is optimistic. Real acceptance is bursty and position-dependent (easy syntax runs accept 8-straight; a hard token rejects everything after it). Real \(\mathbb{E}[n]\) is lower than the geometric formula at equal average \(\alpha\). Tree/multi-branch drafting (Medusa, EAGLE-2/3 dynamic trees) also isn't modeled — a single linear draft chain is.
• The utilization gate is a cartoon. \(1/(1-u)\) is a documented, monotone stand-in for a scheduler + roofline interaction that in reality involves kernel occupancy, wave quantization, and batch composition. The direction is right; the knee location must be measured (Lab 03/04).
• No scheduling effects. Continuous batching, preemption, prefix-cache hits, draft-model memory pressure (a separate draft steals KV/weight room — K01 §3), and PP bubbles (K05) are all absent.
• M/M/c is an approximation. Real arrivals are burstier than Poisson (peak factors — K08 §4), service times are deterministic-ish per prompt length rather than exponential, and TTFT tails come from the mix of prompt lengths, which we collapse to one \(P\).
• Disagg's hidden machinery is free here. KV-transfer scheduling/overlap (layer-by-layer streaming can hide most of \(t_{\text{xfer}}\)), transfer-engine CPU cost, KV-aware routing, and decode-side admission control all cost engineering and sometimes latency; Mooncake and Dynamo exist precisely because this layer is hard.
• TPOT is deterministic in-model. Real TPOT has a distribution; a p99 TPOT SLO would interact with the interference term differently than our mean-based indicator.

The honest claim this lab supports: directionally correct crossovers and defensible first-order magnitudes — the input to a proof-of-concept plan, not a substitute for Lab 03-style measurement.

8. Success criteria

You are done when you can, without notes:

• Derive \(\mathbb{E}[n] = (1-\alpha^{k+1})/(1-\alpha)\) from \(P(A \ge i) = \alpha^i\) on a whiteboard, and state both limits (\(\alpha=0 \Rightarrow 1\), \(\alpha=1 \Rightarrow k{+}1\)).
• Explain in one breath why verification of \(k{+}1\) tokens costs ≈ one target step at low batch (memory-bound step; width rides along under the weight-streaming roof) and why that argument dies at high batch.
• Compute in your head KV-transfer time for any (model, prompt, link): bytes/token × \(P\) ÷ bandwidth — e.g. 320 KiB/token × 1k ÷ 400 GB/s ≈ 0.8 ms.
• State what Erlang-C takes and gives (\(c, \lambda, \mu\) → probability of queueing → mean/tail wait), and why pooling beats splitting at small \(N\).
• Name three things the simulator ignores and the direction each bias points.

Résumé bullet: "Built a deterministic analytic simulator for speculative-decoding economics and prefill/decode disaggregation; quantified the batch-utilization point where speculation hurts and the link-bandwidth/SLO crossover where disaggregation wins, with a 22-test verified closed-form model."

9. Interview Q&A

Q1: When would you turn speculative decoding OFF in production? A: Spec decode converts spare compute into lower latency; the decision variable is whether spare compute exists and whether acceptance covers the draft tax. Concretely: (1) High batch occupancy — once batched decode approaches the compute roof, the k-token verification width stops being free; in my cycle model speedup falls below 1.0 around 90–95% compute utilization even at α = 0.85, so a throughput-oriented fleet at high load should disable it — and modern schedulers gate speculation dynamically on batch size for exactly this reason. (2) Low measured acceptance — on domains far from the draft's training distribution (creative text, low-resource languages, some code), α drops and \(\sum α^i\) can't cover \(1 + k(d+c)\); at α ≈ 0.3 with a mid-cost draft it's a ~2× slowdown. So instrument acceptance-rate telemetry per workload and treat it as an SLO input, not a constant. (3) Memory pressure — a separate draft model steals KV headroom, which is often the binding constraint; MTP/EAGLE-style trunk-sharing heads sidestep most of that cost.

Q2: Why does prefill/decode disaggregation win at scale, and what's the tax? A: The win is interference isolation and independent scaling: prefill is compute-bound and bursty, decode is memory-bound and steady; colocating them means long prompts inflate everyone's TPOT (even with chunked prefill — Sarathi reduces, doesn't eliminate, the interference), so one SLO hostage-takes the other. Splitting pools lets you hit a tight TPOT SLO on decode nodes regardless of prompt bursts, and size each pool to its own utilization target. The tax is threefold: (1) KV transfer — bytes/token × prompt length ÷ link bandwidth, added to TTFT; 0.33 GB per 1k tokens for a 70B GQA model, negligible over NVLink-class fabric (~1 ms) but a quarter-second per 1k tokens over 10G TCP; (2) un-pooling — two small M/M/c pools have worse queueing than one big one, so at small node counts colocated wins on pure statistics; (3) machinery — KV-aware routing, transfer scheduling, two autoscaling loops (this is what DistServe/Mooncake/Dynamo actually engineer). Rule of thumb from the model: disagg wins with tight TPOT SLOs, prefill-heavy or mixed traffic, fast interconnect, and enough nodes that each pool is still well-pooled; colocated wins small fleets, slow links, or loose TPOT SLOs.

Q3: A vendor promised a customer "3× from speculative decoding". Sanity-check it. A: I'd decompose the claim into the four numbers it silently assumes: acceptance α, draft-cost ratio d, verification overhead c, and operating utilization u. Speedup = \(\sum_{i=0}^{k}\alpha^i / (1 + k(d + c/(1-u)))\). For 3× you need \(\mathbb{E}[n] \gtrsim 3\)–5 at practical k, which requires α ≥ ~0.85–0.9 — EAGLE-3-class acceptance, typically measured on chat/code benchmarks aligned with the draft's training data. Then the two killers: (1) whose domain — acceptance on the customer's actual traffic can be 10–20 points lower, and α enters exponentially; (2) whose operating point — 3× headlines are single-request or small-batch; at the customer's batch occupancy the spare-compute term shrinks the win (my model: 3.1× at idle → ~2× at 70% utilization → <1× above ~95%). My response as the engineer in the room: accept the mechanism, re-price the claim with the customer's α measured on a traffic sample and their target batch size, and put the answer in a table next to the do-nothing baseline before anyone commits.

10. References

• Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding: arXiv:2211.17192
• Chen et al. — Accelerating Large Language Model Decoding with Speculative Sampling: arXiv:2302.01318
• Cai et al. — Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads: arXiv:2401.10774
• Li et al. — EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty: arXiv:2401.15077
• Li et al. — EAGLE-3: Scaling up Inference Acceleration of LLMs via Training-Time Test: arXiv:2503.01840
• DeepSeek-AI — DeepSeek-V3 Technical Report (Multi-Token Prediction): arXiv:2412.19437
• Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve: arXiv:2403.02310
• Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: arXiv:2401.09670
• Qin et al. — Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving: arXiv:2407.00079
• Erlang-C / M/M/c: any standard queueing text, e.g. Kleinrock, Queueing Systems, Vol. 1 (1975), ch. 3; or Harchol-Balter, Performance Modeling and Design of Computer Systems (2013), ch. 14.