🛸 Hitchhiker's Guide — Phase 06: GPU Scheduling & Virtualization
Read this if: you can run a CUDA job but "how do two tenants share an H100" and "why won't my 8-GPU job schedule" are mysteries. Field notes; derivations in the WARMUP.
0. The 30-second mental model
No GPU cgroup controller ⇒ the OS can't enforce GPU fractions ⇒ sharing needs GPU-side mechanisms (MPS/MIG/time-slicing) and cluster-side scheduling (bin-packing + gang + topology). Every choice is a point on isolation ⇄ utilization. Kubernetes learns about GPUs only through a device plugin, which does allocation, not isolation.
1. Node-level sharing card
| Mechanism | Isolation | Utilization | Use when |
|---|---|---|---|
| MPS | weak (shared context/fault) | high for small coop jobs | trusted microservices, MPI ranks |
| MIG | strong (HW partition) | medium (fixed profiles, quantization waste) | multi-tenant, SLO, regulated |
| Time-slicing | none (shared mem+faults) | high (overcommit) | dev/notebooks, bursty, cheap |
| Dedicated | total | low | one big job / strict latency |
"Half a GPU?" → "isolated half (MIG 3g.40gb) or cheap shared half (time-slice that's slow when neighbors are busy)?" Knowing to ask that is the whole skill.
2. Cluster-level card
- Bin-packing: first-fit (fast, strands), best-fit (tight, less stranding), spread (failure/thermal). Stranded GPUs = free GPUs no pending job can use.
- Gang scheduling: all-or-nothing for multi-GPU jobs → prevents the partial-allocation deadlock (two 8-GPU jobs each grabbing 6 on 16 GPUs).
- Topology: 8 GPUs on one NVLink island ≫ 8 spread over IB for TP jobs (Phase 08). Topology-aware placement decides collective bandwidth.
- Backfill + reservation: small jobs fill gaps while a big gang assembles — utilization without starving the big job.
3. The k8s GPU path
device-plugin (DaemonSet) --register--> kubelet --advertises--> nvidia.com/gpu: 8
--ListAndWatch (healthy IDs)-->
scheduler places pod requesting nvidia.com/gpu: 1
kubelet --Allocate(ids)--> plugin returns device nodes + mounts + NVIDIA_VISIBLE_DEVICES
(= Phase 04 injection)
Fractional GPU = plugin advertises one physical device as N units (time-slice,
no isolation) OR exposes MIG instances as nvidia.com/mig-1g.10gb (real
isolation). Allocation ≠ isolation — tattoo it.
4. Numbers & facts
| Thing | Value |
|---|---|
| MIG max instances (A100/H100) | 7 |
| MIG smallest profile | 1g.10gb (1/7 compute, 10 GB) |
| NVLink vs IB (placement stakes) | ~900 vs ~50 GB/s |
| GPU preemption cost | checkpoint (GBs) or kill — ≫ CPU preempt |
| Idle GPU in a typical single-model serving | 70%+ (why sharing exists) |
5. War stories
- "We bought 0.5-GPU tiers and customers complained of jitter" — sold time-slicing as if it were isolated. Free tenants interfered. Fix: MIG for paid isolation tiers; never co-place free + paid on one physical GPU.
- "Cluster at 60% but jobs queueing" — stranded GPUs from first-fit + big-gang jobs. Switched to best-fit + reserved whole-node pool for ≥4-GPU jobs. Queue drained.
- "Training job hangs at start, no error" — partial gang allocation; 6 of 8 ranks up, NCCL init waiting forever. Gang scheduling fixed it.
- "TP throughput half of expected" — scheduler spread 8 GPUs across 2 nodes; per-layer all-reduce hit IB not NVLink. Topology constraint added.
- "MPS client crash took down the node's other models" — shared context, shared fault domain. Moved tenants to MIG.
6. Exit bar
You can pick a sharing mechanism from requirements, explain stranded-GPU fragmentation and gang/topology scheduling, draw the k8s device-plugin path, and state allocation≠isolation with conviction. You've built a cluster scheduler sim and a device plugin. Next: Phase 07 — the most commercially valuable scheduling of all, request scheduling inside an LLM server (KV cache, continuous batching), the JD's named skill.