Phase 06 — GPU Scheduling, Sharing & Virtualization
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Head of Engineering [GPU], Platform/Infra Engineer, Cluster Scheduler Engineer Hardware needed: none — both labs are simulators/models (Python + Go)
Why This Phase Exists
The JD lists "GPU scheduling" as a core skill and the whole role is about building an "orchestration platform." Phase 04 ended on the load-bearing fact: there is no GPU cgroup controller — the kernel cannot enforce a fraction of a GPU. That single gap creates an entire industry: MPS, MIG, time-slicing, fractional-GPU vendors, Kubernetes device plugins, and cluster schedulers that bin-pack the most expensive compute in the datacenter.
This phase is where you learn to allocate GPUs at two altitudes: within a node (how do two tenants share one H100 safely and fairly?) and across a cluster (which of 1,000 GPUs gets this job, and how do you avoid stranding capacity?). You'll build a cluster scheduler simulator that exposes bin-packing, gang scheduling, fragmentation, and fairness, and a Kubernetes device-plugin model that shows exactly how a GPU becomes an allocatable resource — including a fractional-GPU scheme.
Concepts
- The sharing problem restated: no GPU cgroup → sharing needs GPU-side mechanisms
- MPS (Multi-Process Service): one context, multiple processes co-resident; cooperative, weak isolation, no hard memory limits (until recent partitioning)
- MIG (Multi-Instance GPU): hardware partitioning into isolated instances (separate SMs, L2 slices, memory); strong isolation, fixed profiles (1g.10gb …), the only real "fractional GPU"
- Time-slicing: the scheduler context-switches whole-GPU among clients; no isolation, oversubscription convenience
- Choosing among them: latency-sensitive vs batch, trusted vs multi-tenant, the isolation/utilization tradeoff
- Cluster scheduling: bin-packing (first-fit/best-fit/dominant-resource), gang scheduling (all-or-nothing for multi-GPU jobs — the deadlock it prevents), topology-aware placement (NVLink islands, Phase 08)
- Fragmentation at cluster scale: stranded GPUs (a node with 1 free GPU can't take an 8-GPU job); bin-packing policy as the lever
- Fairness: FIFO, priority, preemption, fair-share/DRF; the latency-vs-utilization-vs-fairness trilemma
- Kubernetes integration: the device plugin API, extended resources (
nvidia.com/gpu),nvidia-device-plugin, MIG/time-slicing exposure, the scheduler's view - Inference-specific scheduling: how serving (Phase 07) admission control differs from training job scheduling (Phase 08)
Labs
Lab 01 — GPU Cluster Scheduler Simulator (Python)
| Field | Value |
|---|---|
| Goal | Simulate a GPU cluster and implement/compare placement policies; measure utilization, fragmentation (stranded GPUs), queue wait, and gang-scheduling deadlock avoidance. |
| Concepts | Bin-packing (first/best-fit), gang scheduling, topology-aware placement, fragmentation metrics, fairness. |
| Steps | 1) python solution.py — runs a workload trace through 4 policies, prints a comparison table. 2) Read the scheduler core against WARMUP Ch. 5–7. 3) Reproduce the stranded-GPU result; see gang scheduling prevent the partial-allocation deadlock. 4) Extensions: preemption, DRF fairness, topology-aware NVLink placement. |
| Stack | Pure Python (stdlib) |
| Output | A simulator + a policy comparison table (utilization, p50/p99 wait, stranded-GPU count, deadlocks). |
| How to Test | Built-in asserts: best-fit strands fewer GPUs than first-fit on the adversarial trace; gang scheduling yields zero partial-allocation deadlocks; utilization within expected bands. |
| Talking Points | Why an 8-GPU job starves on a fragmented cluster; gang scheduling's all-or-nothing and the deadlock it prevents; topology-aware placement and the NVLink-island argument (Phase 08). |
| Resume Bullet | "Built a GPU cluster scheduler simulator comparing first-fit/best-fit/gang/topology-aware policies; quantified stranded-GPU fragmentation and demonstrated deadlock-free gang scheduling on multi-GPU job traces." |
| Extensions | Priority preemption with checkpoint cost; Dominant Resource Fairness; a topology model with NVLink islands + an inter-island penalty. |
Lab 02 — Kubernetes Device Plugin + Fractional GPU (Go)
| Field | Value |
|---|---|
| Goal | Implement the Kubernetes device-plugin gRPC interface (the real API) for a fake GPU, then extend it to advertise fractional GPUs (time-sliced replicas) — the actual mechanism behind GPU sharing in k8s. |
| Concepts | Device plugin API (ListAndWatch, Allocate), extended resources, how the kubelet/scheduler see GPUs, time-slicing as replica advertisement, MIG profiles as distinct resources. |
| Steps | 1) go test ./... — the plugin and a mock kubelet exercise the full registration→list→allocate flow. 2) Read plugin.go against WARMUP Ch. 8. 3) Flip replicas: 4 and watch one physical GPU advertise 4 allocatable units (time-slicing); observe the isolation caveat. 4) Extensions: MIG-profile resources, health reporting, the allocate-time env injection (NVIDIA_VISIBLE_DEVICES). |
| Stack | Go (stdlib + a vendored device-plugin proto subset) |
| Output | A working device plugin + tests proving registration, listing, allocation, and fractional advertisement. |
| How to Test | go test: the mock kubelet registers the plugin, receives the device list (including fractional replicas), and a pod "allocation" returns the right device IDs + injected env. |
| Talking Points | Why time-slicing "fractional GPUs" share memory with no isolation (and when that's acceptable); how MIG profiles appear as distinct schedulable resources; what the device plugin does and doesn't enforce (allocation ≠ isolation — Phase 04). |
| Resume Bullet | "Implemented the Kubernetes device-plugin gRPC interface for GPU advertisement including fractional (time-sliced) and MIG-profile resources; documented the allocation-vs-isolation boundary for multi-tenant clusters." |
| Extensions | MIG geometry as multiple resource names; device health/unhealthy transitions; a gpu-feature-discovery-style node labeler. |
Deliverables Checklist
- Scheduler sim: policy comparison table reproduced; stranded-GPU and gang results explained
- You can state when to choose MPS vs MIG vs time-slicing from requirements
- Device plugin: full registration→allocate flow passing; fractional advertisement working
- One page: "our platform's GPU-sharing strategy" — which mechanism for which tenant tier (feeds Phase 12)
- You can draw the k8s GPU allocation path (plugin → kubelet → scheduler → pod) from memory
Interview Relevance
- "Compare MPS, MIG, and time-slicing. When each?"
- "An 8-GPU training job won't schedule though 12 GPUs are free. Why, and fix it."
- "What is gang scheduling and what failure does it prevent?"
- "How does Kubernetes know a node has GPUs? Walk me through the device plugin."
- "Your customer wants '0.5 GPU' tiers. How do you actually deliver that, and what do you tell them about isolation?"