Knowledge 09 — Frontier Inference: MoE, Disaggregation, and Reasoning Models (2025–2026)

Goal: modules 00–08 taught you to size and serve a dense model, with prefill and decode colocated on the same GPUs, for chat-shaped requests (1k in / 500 out). Between 2024 and 2026, production broke all three assumptions. This module extends — never repeats — the earlier math to the frontier: sparse MoE models, disaggregated and KV-centric serving, sub-8-bit numerics, grammar-constrained outputs, and reasoning workloads. Every section states which earlier formula it modifies and how. As of mid-2026, this is the shape of large-scale inference.

Table of Contents

• 1. The three assumption-breaks (why this module exists)
• 2. MoE inference from first principles
• 3. Prefill/decode disaggregation in production
• 4. KV-cache tiering, offload, and prefix-cache economics
• 5. The speculative-decoding frontier: EAGLE-3 and MTP
• 6. The low-precision frontier: FP4 and rotations
• 7. Constrained/structured decoding as a runtime feature
• 8. Serving reasoning models (the workload break)
• 9. The 2026 stack cheat-map
• 10. References

1. The three assumption-breaks (why this module exists)

Everything you sized in Knowledge 08 rested on three quiet assumptions:

1. The model is dense — every parameter participates in every token, so "weight bytes read per decode step" is a constant and the batching math of Knowledge 01 §7 amortizes it linearly in batch size. Broken by MoE: frontier models (DeepSeek-V3, Mixtral-class, most 2025+ flagships) activate a small fraction of their parameters per token. Memory holds everything; compute touches a sliver; the amortization curve bends (§2).
2. Prefill and decode share the GPU — one engine, one pool, chunked prefill to referee the interference. Broken by disaggregation: at scale, prefill and decode now run on separate pools connected by a KV-transfer fabric, each pool sized and scheduled to its own bottleneck (§3) — with the KV cache itself promoted to a managed, tiered, routable storage layer (§4).
3. Requests are chat-shaped — modest, predictable output lengths, so capacity math is prefill-and-concurrency-driven. Broken by reasoning models: o1/R1-class models buy quality with thousands of decode-side "thinking" tokens, making output length the dominant, high-variance cost term and invalidating TTFT as the UX metric (§8).

The remaining sections (§5 spec decode, §6 numerics, §7 constrained decoding) are the 2025–2026 frontier of threads you already hold from Knowledge 01 §9 and Knowledge 02.

Principal instinct: when a 2026 customer conversation starts, your first three qualification questions are now: dense or MoE? colocated or disaggregated (and do you have the scale to care)? chat or reasoning-shaped outputs? Those three bits select which sizing math applies before you touch a single knob.

2. MoE inference from first principles

What a sparse MoE layer is

A Mixture-of-Experts layer replaces the transformer's dense MLP (Knowledge 01 §1) with \(E\) parallel expert MLPs plus a tiny router (a linear layer + softmax over experts). Per token:

1. The router scores all \(E\) experts from the token's hidden vector.
2. The token is dispatched to the top-\(k\) experts (typically 1–8 of 8–256); only those \(k\) run.
3. Their outputs are summed, weighted by the router's gate values. Attention layers stay dense and shared.

        hidden vector x
             │
         [router]── scores over E experts, keep top-k
        ┌────┼────────────┐
        ▼    ▼            ▼            (E experts exist; only k compute)
   [expert 7][expert 42][shared expert]
        └────┴─────┬──────┘
              Σ gate-weighted → output

Some designs add a shared expert every token always visits (DeepSeek-V2/V3), so common knowledge isn't duplicated across routed experts.

Active vs total parameters — the number pair that defines every MoE:

\[ P_{\text{active}} = P_{\text{always-on}} + P_{\text{experts}} \cdot \frac{k}{E}, \qquad P_{\text{total}} = P_{\text{always-on}} + P_{\text{experts}} \]

DeepSeek-V3: 671B total, 37B active (256 routed experts per MoE layer, \(k=8\), + 1 shared expert). Per token it computes like a ~37B model but knows like a 671B model.

Why MoE won the frontier

Dense scaling couples capacity to per-token FLOPs: a 700B dense model costs ~\(2 \cdot 700\text{B}\) FLOPs per token, forever (Knowledge 00 §4). MoE decouples them: capacity grows with \(E\), per-token cost with \(k\). Training compute per token also follows active params, so labs get frontier quality at mid-size cost. That economics is why essentially every 2025+ frontier model is sparse.

What MoE does to YOUR sizing math (the key trap)

Take the Knowledge 01 §3/§7 and Knowledge 08 §2 machinery and change exactly two things:

• Memory follows TOTAL params. All \(E\) experts must be resident — any token might route anywhere. DeepSeek-V3 at FP8 is 671 GB of weights: a multi-GPU, often multi-node capacity problem for a model that computes like 37B. The weights-vs-KV balance of the memory budget shifts back toward weights.
• Per-token compute and bandwidth follow ACTIVE params — but only at batch 1. Here is the trap. Dense decode reads all weights once per step regardless of batch \(B\), so bytes/token = weights ÷ B — clean amortization. MoE at batch \(B\) must read every expert that any token in the batch selected. With uniform routing, the expected number of distinct hot experts per layer is \[ E_{\text{hot}}(B) = E\left[1 - \left(1 - \tfrac{k}{E}\right)^{B}\right]. \] For \(E=256, k=8\): batch 1 → 8 experts; batch 8 → ~57; batch 32 → ~163 (64% of all experts); batch 64 → ~222 (87%); batch 256 → effectively all. Batch amplifies expert coverage: weight reads grow with \(B\) instead of staying flat, so amortization doesn't kick in until \(B\) is large enough that coverage saturates — only then does MoE decode start amortizing like dense (over total params).

Numbers, so it hurts. Hypothetical frontier MoE, FP8: 400B total = 28B always-on (attention, shared expert, embeddings) + 372B routed experts, \(E=256, k=8\) → ~40B active. Bytes read per decode step ≈ \(28 + 372 \cdot E_{\text{hot}}(B)/256\) GB:

At batch 1 the MoE is cheaper per token than the dense 70B (active < dense). At batch 32–64 — exactly where Knowledge 01 §7 said serving gets economical — the MoE reads 4–5× more bytes per token, because coverage grew faster than the batch. The fair comparison is MoE vs the much larger dense model of equal quality, where MoE wins — but if you size an MoE assuming dense-style amortization at moderate batch, you will miss throughput by ~4×. This is the single most common MoE capacity-planning error.

Customer translation: "Your MoE has 400B parameters but only 40B work per token — that makes it cheap per token at low load and memory-hungry always. At the batch sizes that make serving economical, it streams most of the 400B every step. We size memory on 400, bandwidth on the coverage curve, and never on 40."

Expert parallelism (EP) — the fourth parallelism, now load-bearing

Knowledge 05 §5 introduced EP; here is what it costs and why you do it anyway.

The mechanism: experts are sharded across devices (device 0 holds experts 0–31, device 1 holds 32–63, …). Each MoE layer then needs two all-to-all collectives per forward step:

dispatch:  every device sends each of its tokens' hidden vectors
           to the k devices owning that token's chosen experts
compute:   each device runs its resident experts on the tokens it received
combine:   results return to each token's home device, gate-weighted sum

All-to-all is the worst collective for a fabric — every device talks to every device, with data-dependent, imbalanced volumes (Knowledge 05 §7). Traffic per token per MoE layer ≈ \(k \cdot d_{\text{model}} \cdot \text{bytes}\) each way. DeepSeek-V3 (\(d=7168\), \(k=8\), 58 MoE layers, FP8 dispatch / BF16 combine): roughly ~10 MB of all-to-all per generated token. At 10k tok/s per serving group that's ~100 GB/s of sustained all-to-all — trivial on NVLink, feasible on a 400 Gb/s RDMA rail, impossible over PCIe/TCP. For MoE, the interconnect, not HBM, is frequently the bottleneck; this is why DeepSeek wrote custom dispatch kernels (DeepEP-style) and why MoE nodes are specced fabric-first.

Why EP at all? Because it converts the coverage trap into a win: with EP=32, each device holds ~1/32 of the experts and reads only its resident hot experts per step — the expert weight reads are spread across 32 HBM systems, so aggregate bandwidth scales with the EP degree. Wide-EP deployments (DeepSeek's own serving; SGLang/vLLM wide-EP modes) run EP degrees of 32–320 across a rack or pod — and run different EP degrees for the prefill and decode pools (prefill groups smaller-EP/compute-dense, decode groups wide-EP/bandwidth-spread), which foreshadows §3.

Load balancing — the honest paragraph

Routers develop favorites. A hot expert receives a disproportionate share of tokens; its device becomes the straggler that the all-to-all — and therefore the whole step — waits on. The toolbox:

• Training time — auxiliary balance losses (Switch/GShard lineage): penalize imbalance during training. Works, but demonstrably taxes model quality — the loss fights the router's learned preferences.
• Training time — aux-loss-free balancing (DeepSeek-V3): a per-expert bias term added to routing scores only for top-k selection (not for the gate weight), nudged up/down each step to equalize load without distorting the learned mixture. This is why it matters: balance without the quality tax.
• Inference time — hot-expert replication: place copies of hot experts on multiple devices, route to the least-loaded replica. Costs memory; rebalance placement periodically from routing telemetry.
• Inference time — capacity factors and token dropping: each expert accepts at most \(C \cdot Bk/E\) tokens per step; overflow tokens skip the expert (residual path only). Cheap and bounded — but a silent quality tax you must measure, with exactly the honesty discipline of Knowledge 02 §11.

There is no free lunch: balance, memory, and quality form a real triangle. Ask an MoE-serving vendor what their token-drop rate is under load; the pause tells you a lot.

Worked mini-sizing: 70B dense vs 40B-active/400B-total MoE, same nodes

Node: 8×H100-80GB (640 GB HBM, ~26.8 TB/s aggregate bandwidth, NVLink).

1. Dense 70B FP8: 70 GB weights → fits one GPU with a little KV; comfortable at TP=2. Per node: four TP=2 replicas, ~570 GB left for KV → hundreds-to-thousands of concurrent 2k-context sequences (Knowledge 01 §3). Decode reads 70 GB/step/replica → batch 64 ≈ 1.1 GB/token → node aggregate in the ~24k tok/s class.
2. MoE 400B FP8: 400 GB weights → ≥6 of the 8 GPUs just for weights; realistic layout EP=8 (+ TP for attention) across the whole node, ~240 GB left for KV. It is one model instance where the dense filled the node with four.
3. MoE throughput at batch 64: ~351 GB/step ÷ 26.8 TB/s ≈ 13 ms/step → ~4,900 tok/s for the node — versus ~24,000 tok/s from the four dense replicas. And each step now carries 2 all-to-alls × ~58 MoE layers of scheduling and fabric traffic on top.
4. The honest conclusion you give a customer: this MoE is worth serving only because its quality matches a ≫70B dense model. You justify the ~4–5× per-node throughput haircut with quality-per-dollar, and you claw throughput back with wide-EP across nodes (spread the expert reads), FP4 experts (§6), very large batches (coverage saturation), and disaggregation (§3). If the customer's quality floor is met by the dense 70B, the dense model wins on every serving metric — model-rightsizing (Knowledge 08 §5) still rules.

3. Prefill/decode disaggregation in production

The DistServe insight, in two sentences (background: Knowledge 01 §4, Knowledge 04 §5): prefill is compute-bound and decode is memory-bound, so colocating them makes each other's tails worse — a long prefill stalls everyone's TPOT, and resident decodes steal compute from TTFT. DistServe (2024) showed that splitting them onto separate pools, each batched and scaled to its own bottleneck, can multiply goodput per GPU at tight SLOs.

            ┌────────────────────┐   KV cache    ┌────────────────────┐
requests →  │  PREFILL pool       │ ───────────▶ │  DECODE pool        │ → tokens
            │  compute-dense      │  (per-layer,  │  bandwidth/KV-rich  │
            │  big batches of     │   streamed)   │  huge continuous    │
            │  prompt tokens      │               │  batches            │
            └────────────────────┘               └────────────────────┘
              sized to TTFT SLO                     sized to TPOT SLO

The KV-transfer problem (do the arithmetic before you believe the architecture)

Disaggregation's price: the prefill node produces the prompt's KV cache, and the decode node needs it. Size it with the Knowledge 01 §3 formula. A Llama-70B-class GQA model runs ~320 KB of KV per token → a 1k-token prompt ≈ 0.32 GB that must move per request:

For scale: the 1k-token prefill itself takes ~45 ms on 4×H100 at FP8 (\(2 \cdot 70\text{B} \cdot 1024 \approx 143\) TFLOP at realistic MFU). Over TCP the transfer costs ~6× the prefill it's offloading — disaggregation over a slow fabric is strictly worse than doing nothing. Over RDMA with per-layer streaming (ship layer 1's KV while layer 2 prefills), the transfer hides almost entirely. The fabric is the feasibility condition — the same lesson as Knowledge 05 §8, one level up.

What shipped, 2024–2026

• Mooncake (Moonshot AI's platform for Kimi, 2024) — the KVCache-centric architecture: a pooled, tiered KV store spanning the whole cluster's DRAM and SSD (not just GPU HBM); prefill and decode pools scale independently; a cache-aware scheduler places requests where their prefix KV already lives and applies early admission control when an SLO can't be met. Its reported wins came as much from the cluster-wide cache as from the disaggregation itself — which is why §4 exists.
• NVIDIA Dynamo (2025) — the datacenter-scale orchestrator above the engines (works with vLLM, SGLang, TRT-LLM): disaggregated serving, KV-aware routing (§4), and a planner that dynamically reassigns GPUs between prefill and decode pools as the input/output-length mix drifts — because a pool split sized for 4k-in/500-out is wrong an hour later when traffic shifts to 500-in/4k-out (reasoning, §8).
• llm-d and Kubernetes-native patterns — the same disagg + KV-aware-routing pattern packaged as k8s inference infrastructure (vLLM-based); one line because the concepts, not the YAML, are what transfer.

The decision rule (what you tell a customer)

Disaggregate when ALL THREE hold:
  1. long prompts            (KV transfer amortizes; prefill interference severe)
  2. strict TPOT SLO         (the interference actually violates something)
  3. real scale + fast fabric (dedicated pools stay busy; RDMA-class links)
Otherwise → colocate with chunked prefill (Knowledge 01 §9 / 04 §5). Done.

The honest small-scale answer is: "you don't need disaggregation; you need max_num_batched_tokens tuned." Chunked prefill captures most of the interference relief with zero fabric cost and zero orchestration complexity. Recommending the boring thing at small scale is a trust-builder, not a cop-out.

4. KV-cache tiering, offload, and prefix-cache economics

Prefix caching reused KV blocks for shared prefixes within one server (Knowledge 04 §5). The 2025–2026 shift: agents and RAG made re-prefill the dominant cost. An agent loop re-sends a growing conversation + tool schemas + system prompt on every turn; in multi-turn traffic most "prefill" work is recomputing KV you already computed minutes ago. So the KV cache graduated from per-request scratch memory to a managed storage tier:

HBM (hot, serving)  →  host DRAM (warm, ~10–50 GB/s over PCIe)  →  NVMe (cold, ~5–14 GB/s)

Restore vs recompute — the arithmetic that governs the tier

Restoring \(N\) cached tokens moves \(N \cdot b\) bytes at tier bandwidth \(W\); recomputing them re-prefills at \(R\) tok/s. Restore wins when \(Nb/W < N/R\), i.e. when

\[ W > b \cdot R \]

— tier bandwidth must exceed per-token KV bytes × prefill speed, independent of \(N\). Worked both ways:

• Llama-70B GQA (\(b\) = 320 KB/token FP16), prefill ~10k tok/s on its TP group: \(bR = 3.2\) GB/s → NVMe (≥5 GB/s) already beats recompute; DRAM crushes it.
• Small 8B model (\(b\) ≈ 128 KB/token), prefill ~50k tok/s: \(bR = 6.4\) GB/s → NVMe is marginal, DRAM still wins.

Rule of thumb: the bigger the model — slow prefill and fat KV per token, both pushing the same way — the deeper you can profitably tier. FP8/FP4 KV (§6) halves/quarters \(b\), making restore win even more often and doubling every tier's capacity.

Prefix-cache-aware routing (the new load-balancing objective)

Once replicas hold big prefix caches, which replica serves a request changes its cost by ~10× (hit = skip most of prefill; miss = pay it all). So:

• The router should send a request to the replica already holding its prefix — cache hit rate becomes a first-class routing objective.
• But this conflicts with load-spreading: perfect affinity piles one tenant's traffic onto one replica (hotspot); perfect spreading scatters prefixes so every replica is cold.
• Production routers (Dynamo's KV-aware router, SGLang cache-aware LB, llm-d) score both — expected-hit-length minus a queue-depth penalty — and take the argmax.

When a customer reports "we added replicas and TTFT got worse," stale prefix affinity after the scale-out now belongs on your suspect list.

RadixAttention (SGLang) — automatic sharing done right

vLLM-style prefix caching matches block hashes of exact prefixes. RadixAttention generalizes it: all cached sequences live in one radix tree — a trie whose edges are token substrings, not single tokens, with nodes as reference-counted KV block lists (PagedAttention underneath) and LRU eviction at the leaves.

root ── [system prompt] ── [few-shot block] ─┬─ [user A history] ── [turn 3]
                                             ├─ [user B history] ── [turn 7]
                                             └─ [user A history'] ─ [retry branch]

A new request walks from the root, matching the longest shared prefix path — automatic sharing across any nesting, not just prefixes an operator predicted. For agentic tree-shaped workloads (many branches from one context: self-consistency, tool retries), hit rates go from "sometimes" to "structural."

Provider prompt caching = the same idea, productized

Anthropic/OpenAI/Google "prompt caching" discounts (up to ~90% off cached input tokens) are exactly this machinery exposed as a price sheet. Two consequences for customer conversations: (1) prompt structure is now a cost lever — stable prefix first, volatile content last, or the cache never hits; (2) in build-vs-API TCO, compare against the customer's cached effective $/token, not list price.

5. The speculative-decoding frontier: EAGLE-3 and MTP

Recap in one paragraph (full treatment: Knowledge 01 §9): a cheap draft proposes \(k\) tokens; the target verifies them in one parallel pass; accepted tokens are free decode steps. The lineage: separate draft model → Medusa (extra heads) → EAGLE/EAGLE-2 (feature-level drafting with a one-layer head reading the target's hidden state; dynamic draft trees). What's new since:

EAGLE-3 (2025) — two mechanism-level changes:

• Multi-layer feature fusion: the draft head reads low-, mid-, and high-level hidden states from the trunk (not just the top layer), because next-token evidence lives at several depths.
• Training-time test: EAGLE-1/2 trained the head to regress the target's features, which capped how much training data could help. EAGLE-3 drops feature regression and instead simulates multi-step drafting during training — the head learns on its own imperfect rollouts, exactly what it will see at inference. Result: acceptance keeps improving as draft training data scales ("training-time scaling" for drafts), with reported ~4–6× decode speedups at low batch.

MTP heads trained INTO the base model (DeepSeek-V3 style) — the structural endpoint of this evolution. The model is pretrained with a multi-token-prediction module: a small extra transformer block that, during training, predicts token \(t!+!2\) from the trunk's representation (sequentially, preserving the causal chain). At inference, the MTP head IS the draft:

• No separate draft model to build, deploy, or version — speculation becomes a config flag.
• It shares the trunk's full representation, so agreement with the main head is structurally high — DeepSeek reports ~85–90% acceptance for the extra token, ~1.8× decode TPS.
• Trade-off vs EAGLE-3: MTP must be baked in at pretraining (you can't add it to a checkpoint you don't own) and drafts fewer tokens; EAGLE-3 bolts onto any open model but needs its own training run.

The batch-size interaction, restated quantitatively. From Knowledge 01 §7: per-step time ≈ \(\max(\text{weight-read time},\ B \cdot t_{\text{compute}})\). Speculation converts the gap between those two terms — spare compute — into extra verified tokens:

• Small \(B\): gap is huge (decode is deep in memory-bound territory) → 2–6× wins.
• Large \(B\): gap → 0 at the compute roof; verification FLOPs (×(1 + draft length) tokens per sequence per step) now compete with other requests' real work — throughput can go down.
• Policy: enable per-workload. Latency-critical low-concurrency lanes: yes. Saturated throughput lanes: no.

Practical acceptance ranges to plan with: well-matched draft model ~60–80% per token; EAGLE-2/3 mean accepted length ~3–6; MTP ~0.85–0.9 for its extra token. Everything degrades with high temperature and creative/unpredictable text; code and greedy decoding are the friendly end. Lab 06 simulates exactly this speedup-vs-batch trade-off — run it and the crossover stops being folklore.

6. The low-precision frontier: FP4 and rotations

Fundamentals — scales, granularity, outliers, weight-vs-activation — are Knowledge 02. This section is what moved 2024→2026.

FP4 with block scales (MXFP4 / NVFP4)

FP4 (E2M1) represents only eight magnitudes (0, 0.5, 1, 1.5, 2, 3, 4, 6 — ± sign). One scale per tensor or channel (Knowledge 02 §3) cannot stretch eight values across a realistic dynamic range — so FP4 only works with per-block scaling: tiny groups of elements share a locally-fitted scale stored alongside.

The block is the outlier blast radius: one outlier now coarsens 16–32 neighbors instead of a whole channel — Knowledge 02 §3's "put the scale boundary where the outliers are," taken to its limit. NVFP4's finer blocks and real-valued scales measurably beat MXFP4 at the same ~4 bits. The reason this is a serving topic: Blackwell-class tensor cores execute FP4 natively at ~2× the FP8 rate, so W4 stops being a "dequantize-then-FP16-matmul" memory trick (Knowledge 02 §9) and becomes real compute throughput — for weights today, activations increasingly.

Rotation-based quantization (QuaRot / SpinQuant)

Knowledge 02 §8 established that activation outlier channels are THE problem. Rotations dissolve it. Insert an orthogonal matrix \(R\) (\(RR^{\top} = I\)) around a matmul:

\[ y = xW = (xR)(R^{\top}W) \]

• Why it's free mathematically: the output is exactly identical; \(R^{\top}W\) is folded into the weights offline, and \(xR\) is folded into the previous layer or applied as a fast \(O(d\log d)\) Hadamard transform online.
• Why it fixes outliers statistically: a Hadamard-type rotation rewrites each channel of \(xR\) as a ±\(1/\sqrt{d}\)-weighted sum of all input channels — a single 100× outlier channel is smeared into a small bump spread across thousands of channels. Kurtosis collapses, per-channel distributions become near-Gaussian, and plain uniform INT4/FP4 quantizers stop clipping.
• QuaRot uses randomized Hadamards; SpinQuant learns the rotations (optimization on the orthogonal manifold), buying a further accuracy step.

This is why 2026 W4A4-ish pipelines exist at all — and it composes with block scales: rotations tame the distribution, blocks catch what's left.

Status board (mid-2026), stated honestly

• W4 weights (FP4 / INT4-group): mainstream for serving, with rotation or AWQ/GPTQ-class treatment; near-lossless on chat-class tasks, still to be proven per task on reasoning/code — the Knowledge 02 §11 discipline applies unchanged.
• FP8 KV cache: routine (halves the KV formula's bytes). FP4 KV: usable with per-head/block scales; watch long-context degradation — attention re-reads KV thousands of times, so KV error compounds in a way weight error doesn't.
• A4 (4-bit activations): still frontier — works on some models with rotations + careful calibration; not a default.
• The recommendation you sign your name to: evaluate on the customer's task, report deltas with conditions, and treat vendor "lossless FP4" claims exactly the way Knowledge 06 §10 taught you to treat vendor benchmarks.

7. Constrained/structured decoding as a runtime feature

Why this became a serving-layer concern. Tool calling and JSON APIs turned "the model outputs valid JSON matching this schema" from a prompt-engineering hope into a contract. Retrying malformed outputs at $/token is a tax; guaranteeing validity inside the decode loop is nearly free. Every major engine (vLLM, SGLang, TRT-LLM) now ships grammar-constrained decoding as a first-class request parameter (response_format, guided_json).

How it works internally:

1. Compile the JSON Schema / regex / CFG into an automaton — an FSM for regular constraints, a pushdown automaton (stack-augmented) for recursive grammars like JSON's nested objects.
2. At each decode step, given the current automaton state, build a mask over the vocabulary: every token that cannot begin any continuation of a valid string gets its logit set to \(-\infty\) before sampling.
3. Advance the automaton with the sampled token; repeat. The model literally cannot emit an invalid token.

The token-level subtlety (the part that separates people who've read the code from people who've read the blog post): the automaton is defined over characters/terminals, but the model emits tokenizer tokens whose boundaries don't align with grammar terminals — one token "},\n closes a string, closes an object, adds a comma and a newline; its legality depends on the full stack state. So the mask must be computed per automaton state over the entire ~128k-token vocab, stepping each candidate token's characters through the automaton — naively, that costs more than the forward pass. The implementations earn their keep here:

• XGrammar: precomputes vocab masks for all context-independent automaton states offline (the vast majority), handles the context-dependent remainder with an efficient runtime stack, and overlaps mask computation with the GPU forward pass → near-zero added latency.
• llguidance (token-prefix automata) and Outlines (FSM-based; the approach that started the wave) occupy the same design space.

Two warnings you owe every customer:

1. Constrained ≠ correct. The grammar guarantees syntax, not truth — a schema-valid hallucinated "account_id": "12345" sails through. You still need task-level evals; and over-tight constraints can hurt quality by forcing the model off its preferred phrasing mid-generation.
2. Interaction with speculative decoding (§5): draft tokens must be masked by the same automaton state machine, or drafts wander into invalid strings and acceptance collapses. Engines that integrate the two (SGLang, vLLM V1) thread the grammar state through the draft loop.

8. Serving reasoning models (the workload break)

What test-time compute is. o1/R1-class models are trained (RL on verifiable rewards — R1's recipe) to emit a long chain of thought — hundreds to tens of thousands of "thinking" tokens — before the visible answer. Quality now scales with inference tokens spent, not just parameters: the s1/test-time-scaling line showed accuracy climbing as the model is allowed to think longer. Commercially: you buy quality per request, at decode prices, on demand — a knob that did not exist in the Knowledge 08 worldview.

Every serving consequence derives from math you already own:

• (a) Capacity becomes output-token-dominated. Knowledge 08 §2 assumed out ≈ 500 with modest variance. Reasoning outputs run 2k–50k with huge, workload-dependent variance — and the KV cache now grows during generation, not mostly during prefill: a 20k-thought request ends holding 20k × per-token KV bytes that no prefix cache can ever share (thoughts are unique per request).
• (b) TTFT stops being the UX metric. The first thinking token arrives fast and means nothing. The SLOs that matter become time-to-first-answer-token (≈ TTFT + thinking-tokens × TPOT — 2,000 thoughts at 40 ms = 80 s before the user sees a word, unless you stream summarized thinking) and total completion time. Rewrite the Knowledge 01 §8 SLO conversation accordingly.
• (c) Thinking budgets are the new cost dial. Production APIs expose effort/budget knobs (max reasoning tokens; low/medium/high effort). Cap the budget and you cap the tail; adaptive budgets (easy query → low effort, hard → high) are the 2026 version of the small-model/big-model router from Knowledge 08 §11.
• (d) Reasoning + agents multiply turns → the same context re-enters prefill over and over → §4 prefix-cache economics stop being an optimization and become the P&L. A reasoning agent without KV reuse can spend more on re-prefill than on thinking.
• (e) Batch composition variance: one 20k-thought request occupies a slot for minutes, skewing continuous-batching occupancy and KV pressure — which is why length-prediction-aware scheduling (predict output length; admit/place accordingly) went from papers to production features in one year.

Worked example: the support bot upgrades to a reasoning model

Baseline (Knowledge 08 §11-style): 70B-class, prompts ~1k, answers ~500; a TP=4 instance sustains ~10k tok/s decode goodput at p95 TPOT 40 ms → 20 req/s per instance (10k ÷ 500); mean live context ~1.25k tokens → ~0.4 GB KV/seq (320 KB/token), so hundreds of concurrent sequences fit.

Upgrade: reasoning model, same size, out ≈ 3,000 (500 answer + ~2,500 thinking), p95 out ≈ 2.5× the mean ≈ 7,500:

1. Throughput: the same 10k tok/s now yields 3.3 req/s per instance — a 6× instance-count increase for the same request rate at unchanged $/token. The bill scales with tokens, and tokens just went 6×.
2. KV: mean live context 1k + 1.5k = 2.5k → ~0.8 GB/seq; a p95 request peaks near 8.5k tokens → ~2.7 GB/seq. Concurrency per instance drops 2–3× on memory while each request holds its slot 6× longer (120 s vs 20 s at 40 ms TPOT) — occupancy, not arrival rate, now sets the queue.
3. UX: time-to-first-answer-token ≈ 2,500 × 40 ms ≈ 100 s at p50 unless thinking is streamed or summarized. The old "p95 TTFT < 1 s" SLO is testing the wrong thing entirely.
4. The levers, in order of cost: thinking-budget cap (3,000 → ~1,200 for easy tickets) → difficulty router (most tickets don't need reasoning; send them to the old model) → prefix caching across agent turns (§4) → FP8 KV (halves item 2) → length-aware scheduling (tames p99) → then hardware.

Presenting that ordered lever list — cheapest first — instead of "you need 6× the GPUs" is the Knowledge 08 trusted-advisor move performed on 2026 material. The customer hears the same physics either way; only one version sounds like advice.

9. The 2026 stack cheat-map

Where the pieces sit, mid-2026 — the layer picture from the track README §3 with the new occupants:

• vLLM V1 — the 2025 re-architecture of the default engine: isolated per-step scheduler loop (multiprocess; CPU scheduling overlapped with GPU compute), chunked prefill on by default (no prefill/decode phase distinction in the scheduler — one token budget), zero-overhead prefix caching (on by default; hashing made negligible), tight spec-decode and structured-output integration. The "just use vLLM" answer got stronger.
• SGLang — RadixAttention (§4) for structural prefix sharing, fastest-in-class constrained decoding (§7), strong wide-EP MoE and disaggregation support; it's what several frontier labs' own serving most resembles. Pick for agent-heavy, cache-heavy, MoE-heavy work.
• TensorRT-LLM — maximum single-node NVIDIA performance: AOT-compiled engines, FP8/FP4 kernels first, deepest Blackwell exploitation. Pays back its rigidity (engine rebuilds) in latency-critical, NVIDIA-committed deployments — the Knowledge 04 §7 logic unchanged.
• Dynamo / Mooncake / llm-d — the orchestration layer above engines (§3–4): disaggregated pools, KV-aware routing, planners. You now design two layers: engine config per node, and cache/pool topology across nodes.

Principal instinct: frameworks now churn quarterly; the concepts — coverage-vs-batch (§2), transfer-vs-fabric (§3), restore-vs-recompute (§4), spare-FLOPs-vs-batch (§5), scale-granularity-vs-outliers (§6), mask-vs-vocab (§7), tokens-are-the-bill (§8) — are the stable layer. Interview and advise from the concepts; look up the flags.

10. References

• DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) — MLA + DeepSeekMoE. arXiv:2405.04434
• DeepSeek-AI, DeepSeek-V3 Technical Report (2024) — 671B/37B MoE, aux-loss-free balancing, MTP, FP8 training, cross-node EP serving. arXiv:2412.19437
• Fedus, Zoph, Shazeer, Switch Transformers (2021) — top-1 routing, capacity factor, aux loss. arXiv:2101.03961 · Lepikhin et al., GShard (2020) — the original sharded-MoE + all-to-all. arXiv:2006.16668
• Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (2024). arXiv:2401.09670
• Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (2024). arXiv:2407.00079
• Agrawal et al., Sarathi-Serve: Taming Throughput-Latency Tradeoff via Chunked Prefill (2024). arXiv:2403.02310
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (2023) — RadixAttention. arXiv:2312.07104
• Li et al., EAGLE-3: Scaling up Inference Acceleration of LLMs via Training-Time Test (2025). arXiv:2503.01840 · Gloeckle et al., Better & Faster LLMs via Multi-token Prediction (2024). arXiv:2404.19737
• Ashkboos et al., QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024). arXiv:2404.00456 · Liu et al., SpinQuant: LLM Quantization with Learned Rotations (2024). arXiv:2405.16406
• Rouhani et al., Microscaling Data Formats for Deep Learning (2023) + the OCP MX specification; NVIDIA NVFP4 / Blackwell documentation. arXiv:2310.10537
• Dong et al., XGrammar: Flexible and Efficient Structured Generation for LLMs (2024). arXiv:2411.15100 · Willard & Louf, Efficient Guided Generation for LLMs (Outlines) (2023). arXiv:2307.09702
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025). arXiv:2501.12948 · Muennighoff et al., s1: Simple Test-Time Scaling (2025). arXiv:2501.19393 · OpenAI o1 system card.
• NVIDIA Dynamo docs (disaggregated serving, KV-aware routing, planner); vLLM V1 architecture blog; llm-d project docs; Mooncake open-source repo.

➡ Next: Labs — turn all nine modules into measured, defensible results (Lab 06 covers the §5 speculative-decoding trade-off). Back to: Track README.