Interview Bank 02 — Serving, Scheduling & Distributed (Phases 06–08)

Full answers in the phases' WARMUP Interview Q&A; this is the index + drills.

GPU scheduling & sharing (Phase 06)

Q: Compare MPS, MIG, time-slicing. MPS = co-resident processes, weak isolation, trusted small jobs. MIG = hardware partition, strong isolation, multi-tenant/SLO. Time-slicing = whole-GPU rotation, no isolation, bursty/dev. Pick on isolation-vs-utilization. [P06 WARMUP Q1]

Q: 8-GPU job won't schedule though 12 are free — why? Cluster fragmentation: 12 scattered, the job needs 8 co-located (gang + topology) — stranded GPUs. Fix: best-fit/consolidation, reservation, topology-aware admission. [P06 WARMUP Q2]

Q: Customer wants "0.5 GPU" tiers — how, honestly? "Isolated half (MIG 3g.40gb) or cheaper shared half (time-slice, slow when neighbors busy)?" Allocation ≠ isolation. [P06 WARMUP Q1/Q4]

Drill: what is gang scheduling and what deadlock does it prevent? (All-or- nothing multi-GPU placement; prevents two big jobs each grabbing partial sets and mutually blocking. [P06 WARMUP Q3])

Drill: how does k8s know a node has GPUs? (Device plugin: register → ListAndWatch → Allocate injects nodes+env; allocation not isolation. [P06 WARMUP Q4])

LLM serving & KV-cache (Phase 07)

Q: Derive KV cache for Llama-3-8B at 8K context; implication? 2·32·8·128·8192·2 ≈ 1.07 GB/seq → ~20 concurrent on a 40 GB GPU → the cache, not compute, caps concurrency. [P07 WARMUP Q1]

Q: Explain PagedAttention and why it's an allocator. Fixed KV blocks + per-seq block table = OS virtual memory for the cache; kills external fragmentation, 2–4× more sequences. The innovation is allocation, not attention. [P07 WARMUP Q2]

Q: Static vs continuous batching; what is goodput? Static = head-of-line blocking + formation delay; continuous = iteration-level admit/retire, 3–10×. Goodput = throughput meeting SLO; beats raw throughput (gameable by oversized batches). [P07 WARMUP Q3]

Drill: chat with a 2K-token system prompt on every request — what do you do? (Prefix caching with COW; >90% prefill cut. [P07 WARMUP Q4])

Drill: memory pressure mid-generation — options? (Admission control; preempt with recompute vs swap. [P07 WARMUP Q5])

Drill: choose among vLLM/TensorRT-LLM/SGLang for a multi-model platform. (Don't standardize; gateway + route per model; goodput-at-SLO evidence. [P07 WARMUP Q6])

Distributed & collectives (Phase 08)

Q: Derive ring all-reduce's cost; why bandwidth-optimal? reduce-scatter + all-gather = 2(N-1)/N·V ≈ 2V/rank, N-independent; 2V is the information-theoretic minimum per rank's links. [P08 WARMUP Q1]

Q: DP vs TP vs PP — which is NVLink-bound? TP (per-layer all-reduce on the critical path); DP all-reduces once/step (overlappable, IB ok); PP passes activations (cheap). [P08 WARMUP Q2]

Q: Why does TP across nodes destroy throughput, with numbers? Per-layer all-reduce ×20 slower on IB vs NVLink × dozens of layers, unhideable → 2–5× slowdown. [P08 WARMUP Q5]

Drill: 512-GPU run at 60% MFU — where do you look? (Exposed comm, topology misconfig, pipeline bubbles, stragglers — find what's not overlapped. [P08 WARMUP Q3])

Drill: one GPU fails in a 1024-GPU job — what happens, how survivable? (Silent hang; timeouts+watchdog, checkpointing, elastic, blast-radius placement. [P08 WARMUP Q4])

The cross-cutting synthesis question

Q: Design the GPU allocation for serving a 70B model to multi-tenant customers with SLOs. 70B doesn't fit one GPU → tensor-parallel across an NVLink island (P08 — TP must stay on NVLink), so gang-schedule the TP group on one node (P06). KV cache is TP-sharded (P07/P08 Ch. 9). Multi-tenant isolation → MIG or dedicated, never time-slice paid tenants together (P06). Serving = continuous batching + paged KV + prefix caching, tuned for goodput-at-SLO (P07). The answer composes P06+P07+P08 — the role's distributed-serving core.