Knowledge 05 — Distributed Inference

The JD bullet: "Optimize workloads for LLM and GenAI models across both multi-SoC and multi-card architectures" and "distributed inference methods … large-scale model deployments." When a model or its KV cache doesn't fit on one accelerator — or when one chip can't hit the throughput target — you split the work across chips. This module covers the parallelism strategies, the collective communications that bind them, and the interconnect physics that decides which strategy wins. It builds directly on the interconnect section of Knowledge 00.

Table of Contents

• 1. Why distribute at all: the two reasons
• 2. The four parallelism dimensions
• 3. Tensor parallelism (TP) in detail
• 4. Pipeline parallelism (PP) in detail
• 5. Expert parallelism (EP) for MoE
• 6. Sequence / context parallelism for long context
• 7. Collective communications (the NCCL primitives)
• 8. The interconnect decides everything
• 9. Choosing a strategy: the decision tree
• 10. Multi-SoC and non-NVIDIA scaling
• 11. Worked example: place a 405B model
• 12. References

1. Why distribute at all: the two reasons

Distribution is never free (it adds communication and complexity), so you only do it for one of two reasons:

1. Capacity — it doesn't fit. Weights + KV cache + activations exceed one chip's HBM. Llama-3-70B in FP16 is 140 GB > 80 GB H100 → must span ≥2 chips. This is a hard constraint.
2. Throughput/latency — it's too slow. It fits, but one chip can't serve the QPS or hit the TPOT target. You add chips to add aggregate bandwidth/compute.

These two reasons map to different strategies. "Doesn't fit" pushes you toward tensor/pipeline parallelism (split the model). "Too slow but fits" pushes you toward data parallelism / replication (more copies). Misdiagnosing which problem you have is the classic sizing error.

First sizing question: "Does it fit on one chip, and is the binding limit weights or KV cache?" The answer selects the whole strategy. (Memory budget math: Knowledge 01 §11.)

2. The four parallelism dimensions

You can split a transformer along four orthogonal axes. Real deployments combine them ("3D/4D parallelism").

The mental model: TP splits inside each layer (fine-grained, chatty, wants NVLink); PP splits between layers (coarse, less chatty, tolerates network); DP replicates the whole thing (no model-internal comms). You pick a combination so that the chatty parallelism (TP) stays on the fast link (NVLink, intra-node) and the less chatty (PP/DP) spans the slow link (InfiniBand, inter-node).

3. Tensor parallelism (TP) in detail

TP (Megatron-LM, Shoeybi et al. 2019) splits the weight matrices of each layer across GPUs so each holds a slice and computes part of every layer.

How an MLP splits: for Y = GeLU(X·A)·B:

• Split A column-wise across GPUs → each computes GeLU(X·A_i) independently (no communication — the GeLU is elementwise).
• Split B row-wise → each computes a partial Y_i; an all-reduce sums the partials into the full Y.
• So each MLP costs one all-reduce. Attention splits analogously (heads across GPUs) with one all-reduce per attention block.

So TP=N adds ~2 all-reduces per transformer layer (one for attention, one for MLP). For an 80-layer model that's 160 all-reduces per forward pass — per token in decode. Each all-reduce moves ~2·batch·hidden_size bytes and is bounded by interconnect bandwidth.

The consequence: TP's overhead is communication-bound and grows with how slow your link is. On NVLink (900 GB/s) it's cheap → TP=8 within a node is standard. Over PCIe (64 GB/s) the all-reduces dominate → TP becomes a slowdown. This is the single most important practical fact about TP, and the one customers get wrong when they buy PCIe-only boxes expecting TP performance.

Whiteboard: "Tensor parallelism makes every layer a team effort — after each layer the GPUs must all-reduce to combine their slices. That's fine over NVLink, fatal over PCIe. So TP lives inside a node; you don't tensor-parallel across the network."

4. Pipeline parallelism (PP) in detail

PP splits the model by layers into sequential stages: GPU 0 holds layers 1–20, GPU 1 holds 21–40, etc. A request flows through stages like an assembly line. Communication is only the activations passed between adjacent stages — small and infrequent compared to TP's per-layer all-reduces, so PP tolerates slower inter-node links.

The catch: the pipeline bubble. With one request in flight, GPU 1 sits idle while GPU 0 works on stage 1, etc. — only one stage is busy at a time. You hide this by keeping many micro-batches/requests in flight so every stage always has work (like continuous batching across stages). Bubble fraction ≈ (stages − 1) / (stages − 1 + microbatches) → more in-flight requests → smaller bubble.

PP is how you span nodes (where TP can't go because of the network). Typical large deployment: TP within each node (over NVLink), PP across nodes (over InfiniBand).

Trade-off: PP adds latency (a request traverses all stages sequentially) and needs high concurrency to fill the pipe. It's a throughput-at-scale tool, not a single-request-latency tool.

5. Expert parallelism (EP) for MoE

Mixture-of-Experts models (Mixtral, DeepSeek-V3, Grok, GPT-MoE-style) replace the dense MLP with many experts, of which a router activates only a few per token (e.g., 2 of 8/256). This gives huge parameter count (capacity) at low per-token FLOPs (only active experts compute).

Expert parallelism places different experts on different GPUs. Each token is routed to its chosen experts, which requires an all-to-all communication: tokens scatter to wherever their experts live, compute, then gather back. All-to-all is communication-heavy and load can be imbalanced (a "hot" expert gets too many tokens → stragglers). Serving MoE well is its own specialty (expert placement, capacity factors, token dropping, all-to-all overlap).

Why it's on the rise: MoE is the leading way to scale parameters without scaling inference FLOPs — DeepSeek-V3 (671B total, ~37B active) is the poster child. A JD2 specialist increasingly must serve MoE, where memory holds all experts but only a fraction compute per token — a different sizing equation (capacity-heavy, FLOP-light) than dense models. (The full MoE serving treatment — batch-vs-expert-coverage math, all-to-all traffic sizing, wide-EP deployments, and load balancing — is Knowledge 09 §2.)

6. Sequence / context parallelism for long context

For very long contexts (128k–1M tokens), the activations and attention for the sequence itself become the bottleneck, independent of model size. Sequence parallelism splits the sequence dimension across GPUs; ring attention computes attention by passing K/V blocks around a ring of GPUs so each computes its share without materializing the full sequence on one chip. This is the frontier for long-context and is combined with the other parallelisms.

7. Collective communications (the NCCL primitives)

All distributed inference is built on collective operations — coordinated multi-GPU communication patterns. On NVIDIA these are NCCL (NVIDIA Collective Communications Library); equivalents exist everywhere (RCCL on AMD, oneCCL, Gloo, MPI, vendor libs). Know the primitives:

Cost model (worth knowing roughly): a ring all-reduce of N bytes across P GPUs moves about 2N·(P−1)/P bytes per GPU → time ≈ 2N/β (bandwidth-bound) plus a latency term ~(P−1)·α. So all-reduce time is bandwidth-bound for big tensors, latency-bound for small ones and many GPUs. This is why decode (tiny per-step tensors, frequent all-reduces) is so sensitive to interconnect latency, and why high TP degree over a slow link is death.

The number to internalize: TP all-reduce cost ∝ hidden_size · batch / interconnect_bandwidth, per layer, per token in decode. Plug in NVLink vs PCIe and you can predict whether a TP config will fly or flop before you ever run it.

8. The interconnect decides everything

Restating Knowledge 00 §9 because it's the crux of distributed inference:

Fastest → slowest (and that ordering dictates which parallelism goes where):
  NVLink/NVSwitch (~900 GB/s, intra-node)  → put TENSOR parallelism here
  PCIe Gen5      (~64 GB/s)                → avoid TP here; OK for PP/DP/weights
  InfiniBand/RoCE(~50 GB/s/port, inter-node)→ put PIPELINE/DATA parallelism here

The canonical large-model topology falls right out of this:

• TP = (#GPUs per NVLink domain) — e.g., TP=8 within a DGX node.
• PP across nodes over InfiniBand.
• DP to replicate the whole TP×PP unit for more throughput.

A principal sizes parallelism to the physical interconnect, never abstractly. The most common expensive mistake is configuring high TP across a slow link.

Customer catch (pure JD2 value): a customer bought 8 GPUs in PCIe-only servers (2/box, no NVLink) and can't understand why TP=8 is slower than TP=2. You explain the all-reduce-over-PCIe physics, recommend TP=2 within each box + DP/PP across boxes, or an NVLink-equipped box if they need a single big-model instance. That diagnosis saves them a re-architecture — exactly the "trusted technical advisor" deliverable.

9. Choosing a strategy: the decision tree

Does the model (weights + KV at target batch/context) fit on ONE chip?
├─ YES → use DATA PARALLEL / replication to scale throughput. Simplest. Done.
└─ NO  → must split the model:
     Is there a fast intra-node link (NVLink) and enough GPUs in one node?
     ├─ YES → TENSOR PARALLEL up to the NVLink domain size (e.g., TP=8).
     │        Still doesn't fit / need more scale?
     │        └─ add PIPELINE PARALLEL across nodes (over InfiniBand).
     └─ NO (only PCIe / cross-node) → avoid high TP; prefer PIPELINE PARALLEL
              (light comms tolerate slow links) + DP.
     Is it a Mixture-of-Experts model?  → add EXPERT PARALLEL (all-to-all).
     Is context extremely long (≥128k)? → add SEQUENCE/CONTEXT PARALLEL.
Then: pick the smallest configuration that meets the SLO (every extra chip is $ and comms overhead).

The objective is always the smallest, cheapest topology that meets the SLO and accuracy floor — over-sharding wastes money and adds latency. Combine with quantization first: quantizing 70B to INT8 (70 GB) may let it fit with TP=2 instead of TP=4, halving the chip count. Quantize before you shard.

10. Multi-SoC and non-NVIDIA scaling

JD2 says "multi-SoC and multi-card." The principles transfer; the names change:

• Qualcomm Cloud AI: scale across multiple AI100 cards in a server and across SoCs; the qaic stack handles multi-card model splitting; the interconnect is PCIe-class, so pipeline-style splitting and replication fit the physics better than chatty TP. (Know that the interconnect tier shapes the strategy, just like NVIDIA.)
• AWS: Inferentia/Trainium nodes connected by NeuronLink intra-node + EFA inter-node; the Neuron SDK exposes TP/PP.
• Google TPU: ICI (Inter-Chip Interconnect) wires TPUs into 2D/3D torus pods; XLA does the sharding (GSPMD). TPU pods are the original "everything is one giant parallel computer" design.
• Groq/Cerebras: scale by streaming weights across many chips (their whole architecture is distributed by default).

The transferable skill: identify the interconnect tiers of the target system, map chatty parallelism to fast links and coarse parallelism to slow links, and size to the SLO. That reasoning is vendor-independent — which is exactly why JD2 wants someone who understands principles, not one SDK's flags.

11. Worked example: place a 405B model

Customer wants Llama-3.1-405B, 8k context, on a cluster of 8×H100-80GB DGX nodes (NVLink within node, InfiniBand between).

1. Memory: 405B FP16 = 810 GB ≫ 80 GB. Even one node (8×80=640 GB) doesn't hold FP16 weights + KV. → quantize first: FP8 → 405 GB; INT4 → ~200 GB.
2. At FP8 (405 GB weights): fits in one node (640 GB) with room for KV. → TP=8 within the node over NVLink. One node serves one model instance.
3. Scale throughput: replicate the node (data parallel across nodes over InfiniBand) — N nodes = N× throughput, no cross-node model comms.
4. If you must stay FP16 (810 GB): one node can't hold it → TP=8 within each node + PP=2 across two nodes (layers split: node A layers 1–63, node B 64–126; activations cross InfiniBand — light enough for PP).
5. KV cache at 8k context (405B GQA): size it (Knowledge 01 §3); if it caps batch below the throughput target, add INT8 KV or more replicas.
6. The recommendation: "FP8 + TP=8 per node + DP across nodes" — minimal sharding, all chatty comms on NVLink, scale-out on the network. Defend it with the memory math and the SLO.

This is the system-design interview for this role. Be able to do it on a whiteboard with the numbers.

12. References

• Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (2019) — tensor parallelism.
• Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM" (SC 2021) — 3D parallelism (TP+PP+DP).
• Pope et al., "Efficiently Scaling Transformer Inference" (2022) — partitioning math for inference specifically.
• Rajbhandari et al., ZeRO (SC 2020) and Huang et al., GPipe (2019) — memory sharding and pipeline parallelism.
• Liu et al., "Ring Attention" (2023) — sequence/context parallelism.
• NCCL documentation and the ring/tree all-reduce cost analyses; NVIDIA NVLink/NVSwitch and InfiniBand docs.
• DeepSeek-V3 technical report (MoE + expert parallelism at scale); Mixtral report.

➡ Next: Knowledge 06 — Performance Profiling & Benchmarking — proving where the time actually goes.