MoE and Dense Models

Phase 2 · Document 08 · Transformer Foundations Prev: 07 — Prefill vs Decode · Next: 09 — Reasoning Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

"Dense vs MoE" is the architectural choice behind the most confusing numbers on a model page — 8x7B, 26B-A4B, 671B-A37B. Misreading them leads to the most common hardware mistake in LLM engineering: assuming an MoE's active size is what you need to fit, then OOM-ing because you must hold all experts in memory. Getting this right lets you correctly predict memory (tracks total params) and speed (tracks active params), decide when an MoE's "big-model quality at small-model speed" is worth its memory and routing complexity, and read frontier model releases — most of which are now MoE.

2. Core Concept

Dense models

In a dense model, every parameter participates in every token's forward pass. A 70B dense model does 70B-worth of compute per token. Simple, predictable latency, no routing — but compute scales directly with size, so making it "smarter" by adding parameters makes every token proportionally more expensive.

Mixture-of-Experts (MoE)

MoE replaces the single FFN in each layer (03) with many expert FFNs plus a small router. For each token, the router scores the experts and activates only the top-k (often 1–2 of dozens). Attention layers are usually still dense and shared.

dense layer:   token → ONE FFN
MoE layer:     token → router → pick top-k of N experts → only those FFNs run

The crucial split:

• Total parameters = all experts (must be resident in memory).
• Active parameters = the few experts that run per token (drive compute/speed).

This is exactly what the model-page shorthand encodes:

• 8x7B (Mixtral): 8 experts of ~7B each → ~47B total, ~13B active (2 experts/token).
• 26B-A4B: 26B total, ~4B active.
• 671B-A37B (DeepSeek-style): 671B total, ~37B active.

Why MoE exists

It decouples capacity from per-token compute. You can grow total parameters (and thus knowledge/quality) without growing the FLOPs spent per token — "big-model quality at small-model speed." The price is memory (all experts resident) and complexity (routing, load-balancing across experts, and lumpy expert-parallel serving).

The costs and gotchas

• Memory: you pay for total parameters. A 26B-A4B needs ~26B-worth of VRAM/RAM, not 4B. This catches almost everyone.
• Routing imbalance: if the router sends too many tokens to a few experts, utilization drops; training uses load-balancing losses, and serving uses expert parallelism to spread experts across devices.
• Latency variance: which experts fire varies per token/batch, so MoE serving can be lumpier than dense.
• Quantization still helps — and matters more — because the FFN/expert matrices are most of the (large) total memory (Phase 1.06).

3. Mental Model

DENSE  = one chef cooks every dish fully     → compute ∝ size, predictable, simple
MoE    = a kitchen of specialist chefs;      → router sends each dish to a few specialists
         a maître d' (router) picks top-k

  "AxB" / "T-Ak" notation:
     TOTAL params  = all experts  → must FIT IN MEMORY   (the surprise)
     ACTIVE params = top-k experts → drive COMPUTE/SPEED

  26B-A4B:  hold 26B in memory, compute like ~4B per token.

  MoE buys: big-model quality at small-model SPEED.
  MoE costs: big-model MEMORY + routing/serving complexity + latency variance.

4. Hitchhiker's Guide

What to read first on a model page: is it dense or MoE, and if MoE, total vs active parameters. Memory planning uses total; latency intuition uses active.

What to ignore at first: router algorithm details (top-k gating, load-balancing losses) — know that routing exists and can imbalance.

What misleads beginners:

• Reading 26B-A4B as "a 4B model" — you must fit 26B in memory.
• Expecting dense-like steady latency from MoE — it varies with routing.
• Assuming MoE always wins — for tight memory, a smaller dense model may be the only thing that fits.

How experts reason: memory budget from total params; speed expectation from active params. They choose MoE when they need higher quality at controlled per-token compute and can afford total-param memory; otherwise a dense model.

What matters in production: can you fit total params (+ KV cache) on your hardware? Does your serving stack handle MoE (expert parallelism)? Is latency variance acceptable for your SLOs?

How to verify: in config.json, read num_local_experts/num_experts and num_experts_per_tok; compute total vs active params; confirm your engine (vLLM/llama.cpp) supports the MoE architecture.

Questions to ask: Dense or MoE? Total and active params? How many experts, how many active? Does the provider/engine support it, and what's the latency variance?

What silently gets expensive/unreliable: budgeting for "active" size and OOM-ing on total; MoE latency spikes under skewed routing; serving MoE on a stack that doesn't support expert parallelism.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Dense: every parameter runs per token. Total = active.
• MoE: a router activates top-k experts per token. Total ≠ active.
• Memory tracks TOTAL parameters; speed tracks ACTIVE parameters.
• 8x7B ≈ 47B total / ~13B active; 26B-A4B = 26B total / 4B active.
• You must hold all experts in memory even though few run per token — the classic MoE surprise.
• MoE gives big-model quality at small-model per-token compute, at the cost of memory + routing complexity.
• Routing imbalance hurts utilization; serving uses expert parallelism and training uses load-balancing losses.
• Quantization matters more for MoE because total (mostly expert/FFN) memory is large.

9. Observations from Real Systems

• Mixtral (8x7B), Qwen-MoE, DeepSeek-V2/V3, and many recent frontier models are MoE — the architecture now dominates the high end.
• models.dev / HF cards report total and (often) active params; the "AxB"/"T-Ak" labels are the Section 2 split in shorthand.
• vLLM/SGLang support MoE with expert parallelism; not every engine/version does — a real compatibility check.
• llama.cpp serves MoE GGUFs but must hold all experts in memory — users routinely underestimate the footprint.
• Quantized MoE releases exist precisely because total memory is large; 4-bit makes big MoEs runnable locally.

10. Common Misconceptions

11. Engineering Decision Framework

Read the label:
  "AxB"  → A experts of ~B each → total ≈ A×B (minus sharing), active ≈ k×B.
  "T-Ak" → T total, A active.

Dense vs MoE:
  Need higher quality, latency-sensitive, CAN afford total-param memory & MoE serving?
     → MoE (quality at controlled per-token compute).
  Memory-constrained, want predictable latency, simple serving?
     → smaller DENSE model.

Can I run this MoE?
  fits = total_params × bytes(quant) + KV cache + overhead ≤ memory   (Phase 6)
  AND my engine supports the MoE architecture / expert parallelism.
  If not → quantize, smaller (dense or MoE) model, or more/bigger GPUs.

12. Hands-On Lab

Goal

Compute total vs active parameters for real MoE configs, predict memory and relative compute, and contrast with a dense model.

Prerequisites

• Python 3.10+, transformers (config read only; no weights download needed).

Setup

pip install transformers

Steps

from transformers import AutoConfig

def moe_report(name):
    c = AutoConfig.from_pretrained(name)
    experts = getattr(c, "num_local_experts", getattr(c, "num_experts", None))
    active  = getattr(c, "num_experts_per_tok", None)
    print(f"{name}: experts={experts} active_per_tok={active} "
          f"layers={c.num_hidden_layers} hidden={c.hidden_size} d_ff={getattr(c,'intermediate_size',None)}")
    if experts and active:
        print(f"  → memory tracks ALL {experts} experts; compute tracks {active} per token "
              f"(ratio {experts/active:.0f}x more capacity than active compute)")

# Try an MoE and a dense model (use ones you can access):
for m in ["mistralai/Mixtral-8x7B-v0.1", "Qwen/Qwen2.5-1.5B"]:
    try: moe_report(m)
    except Exception as e: print(m, "skip:", e)

# Memory estimate helper (total params × bytes/param)
def mem_gb(total_params_b, bytes_per): return total_params_b * 1e9 * bytes_per / 1e9
for label, total in [("Mixtral~47B", 47), ("26B-A4B", 26), ("dense 13B", 13)]:
    print(f"{label}: FP16≈{mem_gb(total,2):.0f}GB  4-bit≈{mem_gb(total,0.5):.0f}GB (weights only)")

Expected output

• MoE configs show num_local_experts and num_experts_per_tok; the capacity-vs-active ratio prints.
• Memory estimates use total params — making clear a 26B-A4B needs ~26B-worth of memory.

Debugging tips

• Field missing? MoE configs vary (num_local_experts vs num_experts); inspect the config object.
• Gated model? Use a public MoE config or hardcode the published numbers.

Extension task

For one MoE, compute the FFN/expert share of total params and estimate how much 4-bit quantization saves — then state whether it fits a 24 GB GPU.

Production extension

Add a fits(model, gpu_gb, quant, ctx, concurrency) check combining total-param weight memory + KV cache (06) — extending the Phase 6 calculator to MoE.

What to measure

Experts and active-per-token; capacity/active ratio; total-param memory at FP16/4-bit; fit on a target GPU.

Deliverables

• A total-vs-active table for ≥1 MoE and ≥1 dense model.
• Memory estimates (using total params) at two precisions.
• A fit verdict for one MoE on a specific GPU.

13. Verification Questions

Basic

1. In a dense model, how many parameters run per token? In MoE?
2. What do total and active parameters each determine?
3. Decode 8x7B and 26B-A4B into total/active.

Applied 4. Why does a 26B-A4B model need ~26B-worth of memory, not 4B? 5. When would you pick a smaller dense model over an MoE?

Debugging 6. Your MoE OOMs on hardware sized for its "active" parameters. Explain and fix. 7. MoE latency is variable under load. What architectural reason explains it?

System design 8. Choose dense vs MoE for a latency-sensitive assistant with generous memory but strict p95; justify.

Startup / product 9. Pitch why an MoE could improve your product's quality-per-dollar — and the one infrastructure caveat to budget for.

14. Takeaways

1. Dense: every parameter runs per token (total = active). MoE: a router activates top-k experts.
2. Memory tracks TOTAL params; speed tracks ACTIVE params — the core distinction.
3. Labels: 8x7B ≈ 47B/13B; 26B-A4B = 26B/4B; 671B-A37B = 671B/37B.
4. MoE = big-model quality at small-model per-token compute, paid for in memory + complexity.
5. You must hold all experts in memory and use a stack that supports MoE.
6. Choose dense for tight memory/predictable latency; MoE for quality with ample memory.

15. Artifact Checklist

• Code: total-vs-active param + memory calculator for MoE/dense.
• Table: total/active and FP16/4-bit memory for ≥2 models.
• Fit verdict: one MoE on a specific GPU (weights + KV).
• Notes: the memory-tracks-total / speed-tracks-active rule.
• Decision record: dense vs MoE for one workload + constraints.
• Diagram: router + experts and the total/active split.

Next: 09 — Reasoning Models