Warmup Guide — GPU Scheduling, Sharing & Virtualization

Zero-to-expert primer for Phase 06. Builds on Phase 04 (no GPU cgroup controller) and Phase 01 (SMs, MIG-able hardware). By the end you can choose sharing mechanisms from requirements and design a cluster scheduler.

Table of Contents

Chapter 1: The Sharing Problem, Restated

Phase 04 Ch. 8 established the fact that organizes this entire phase: the Linux kernel has no GPU resource controller. You can cap a process's CPU and memory with cgroups; you cannot cap its share of a GPU's SMs or HBM through any kernel mechanism. The GPU is opaque to the OS scheduler — the driver (and the GPU's own front-end) owns GPU time, and they don't expose a fractional knob to the kernel.

So why share at all? Economics. An H100 costs more than the engineer using it, and a single inference model or notebook frequently uses 10–30% of one. Leaving 70% idle across a fleet is the largest controllable line item in a GPU cloud's P&L. Every mechanism in this phase exists to reclaim that idle capacity without letting tenants corrupt or starve each other.

The three answers, previewed: MPS (let processes co-reside in one context), MIG (cut the silicon into isolated pieces), time-slicing (rotate the whole GPU among clients). They sit at different points on the isolation-vs-utilization curve, and choosing among them per workload is a core platform-leadership skill (Ch. 5).

Chapter 2: MPS — Cooperative Co-Residency

The mechanism: normally, processes sharing a GPU time-slice at context granularity (the GPU switches between their contexts, serializing). The Multi-Process Service runs a daemon that funnels multiple processes' kernels into one shared context, so their kernels execute concurrently on the SMs — filling the gaps a single under-utilizing process leaves.

  • Win: higher utilization for many small co-operating jobs (classic case: MPI ranks, or several small inference models). No context-switch overhead between them.
  • Isolation: weak. Originally no memory limit (one process could OOM the others' allocations) and no real fault isolation — a fault in one client can poison the shared context (Phase 02 Ch. 4), taking down co-tenants. Volta+ MPS added per-client memory limits and some execution-resource provisioning (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE), narrowing but not closing the gap.
  • Verdict: great for trusted, cooperating workloads (your own microservices); not a multi-tenant security boundary.

Chapter 3: MIG — Hardware Partitioning

The mechanism (Ampere+, A100/H100): physically partition one GPU into up to 7 instances, each with dedicated SMs, dedicated L2 cache slices, dedicated memory controllers and HBM, and a separate fault domain. A MIG instance presents to software as its own little GPU.

  • Profiles: fixed geometries named <compute>g.<memory>gb — e.g., 1g.10gb (1/7 compute, 10 GB), 2g.20gb, 3g.40gb, 7g.80gb (whole). Memory and compute are partitioned together; you choose from the menu, you don't get arbitrary fractions.
  • Isolation: strong — hardware-enforced. A tenant in 1g.10gb cannot touch another instance's memory, cannot steal its SMs, and a fault is contained. This is the only "fractional GPU" with real multi-tenant isolation.
  • Cost: rigidity (fixed profiles, reconfiguration usually needs the GPU idle/reset), and quantization waste (a job needing "1.5 instances" rounds up).
  • Verdict: the default for multi-tenant fractional serving where isolation is a contractual requirement (sovereign/regulated customers — this JD's nice-to-haves).

Chapter 4: Time-Slicing — Oversubscription Without Isolation

The mechanism: the simplest scheme — the GPU scheduler context-switches the whole GPU among clients in turn (the default behavior when multiple processes use one GPU without MPS/MIG; in k8s, exposed by advertising N "replicas" of one physical GPU).

  • Win: trivial oversubscription — pack 4 bursty notebooks onto 1 GPU; when one is idle, others get full GPU during their slice. Zero hardware requirement.
  • Isolation: none — clients share all memory (sum must fit!), no performance isolation (a heavy client starves others during its slices), full fault sharing.
  • Verdict: dev/test, bursty interactive workloads, trusted oversubscription where occasional interference is acceptable; never for latency-SLO or multi-tenant security.

Chapter 5: Choosing a Sharing Mechanism

The decision table to internalize (and the heart of the "0.5 GPU tier" interview question):

NeedMechanism
Trusted small jobs, max utilization, no SLOMPS
Multi-tenant, isolation is contractual, predictable QoSMIG
Bursty dev/notebooks, oversubscription, cost over isolationTime-slicing
One big job using the whole GPUNone (dedicated)
Latency-critical serving + isolationMIG (or dedicated)

The underlying axis is isolation vs utilization: MIG buys isolation by giving up the flexibility to overcommit; time-slicing buys utilization by giving up isolation; MPS sits between. A platform usually offers tiers mapping to these — and the honest answer to a customer asking for "half a GPU" is "which do you need: a guaranteed isolated half (MIG 3g.40gb), or a cheaper shared half that may be slow when neighbors are busy (time-slicing)?" Saying that sentence is what separates a platform leader from a salesperson.

Chapter 6: Cluster Scheduling — Bin-Packing and Fragmentation

Zoom out from one node to thousands. A scheduler maps jobs (each wanting k GPUs, maybe with constraints) onto nodes (each with some free GPUs). This is bin-packing, NP-hard, solved with heuristics:

  • First-fit: place a job on the first node that fits. Fast, simple, packs loosely → strands capacity.
  • Best-fit: place on the node that leaves the least leftover. Packs tighter, reduces fragmentation, slightly more expensive.
  • Worst-fit / spread: place on the emptiest node. Good for thermal/failure spread and leaving room for big jobs, worse for packing.
  • Dominant Resource Fairness (DRF) when jobs want heterogeneous resources (GPUs + CPU + RAM): fair-share along each job's dominant resource.

Cluster fragmentation — the stranded-GPU problem (Lab 01 reproduces it): free GPUs scattered one-per-node can't satisfy an 8-GPU gang job, even though total free ≥ 8. This is Phase 05's external fragmentation at datacenter scale — and the fix is the same family: placement policy (best-fit consolidates; defragmentation by migration where checkpointable), and shape-aware admission (reserve whole nodes for big jobs). The metric to watch: stranded GPUs = free GPUs that no pending job can use given placement constraints.

Chapter 7: Gang Scheduling and Topology Awareness

Gang scheduling: a multi-GPU job needs all its GPUs simultaneously (a distributed training step can't start with 6 of 8 ranks — the all-reduce would hang, Phase 08). So the scheduler must allocate all-or-nothing: either place the whole gang or none of it. Why it matters: without it, two 8-GPU jobs on a 16-GPU cluster can each grab 4–6 GPUs, neither can proceed, neither will release — deadlock. Gang scheduling (and its cousin, reservation) prevents the partial-allocation deadlock. Lab 01 demonstrates both the deadlock and its prevention.

Topology awareness: not all GPU slots are equal. 8 GPUs on one node behind NVLink (900 GB/s, Phase 01 Ch. 9) crush 8 GPUs spread across nodes on InfiniBand (~50 GB/s) for a tensor-parallel job. A topology-aware scheduler prefers NVLink islands for tightly-coupled jobs and is willing to wait or consolidate to get them. This is where Phase 06 (scheduling) and Phase 08 (collectives) meet: the scheduler's placement decides the collective's bandwidth, which decides the job's throughput. Lab 01's extension models NVLink islands with an inter-island penalty.

Chapter 8: Kubernetes Device Plugins — How a GPU Becomes Schedulable

Kubernetes knows nothing about GPUs natively. The device plugin API (a gRPC contract) teaches it. The flow (Lab 02 implements it end to end):

  1. The plugin (e.g., nvidia-device-plugin, a DaemonSet pod) registers with the kubelet over a Unix socket, claiming a resource name: nvidia.com/gpu.
  2. ListAndWatch: the plugin streams the list of healthy device IDs to the kubelet; the node now advertises nvidia.com/gpu: 8 as an extended resource.
  3. The scheduler treats nvidia.com/gpu like CPU/memory for placement (pods request it in resources.limits).
  4. When a pod lands, the kubelet calls the plugin's Allocate with the chosen device IDs; the plugin returns the device nodes, mounts, and environment to inject (NVIDIA_VISIBLE_DEVICES=...) — which is exactly the container-injection mechanism from Phase 04 Ch. 9.

The crucial boundary (say it in interviews): the device plugin does allocation, not isolation. It advertises and assigns; the actual isolation is whatever the underlying mechanism provides (whole device, MIG instance, or — for time-slicing — nothing, you just advertised replicas: 4 of one physical GPU). Fractional GPUs in k8s are this: the plugin lists one physical GPU as N allocatable units. MIG appears as distinct resource names (nvidia.com/mig-1g.10gb). Lab 02 builds the plugin, then the fractional advertisement, so the allocation/isolation line is concrete, not slogan.

Chapter 9: Fairness, Priority, and Preemption

The trilemma every scheduler navigates: latency vs utilization vs fairness — you can optimize any two.

  • FIFO: simple, starves big/low-priority jobs behind a long queue.
  • Priority + preemption: high-priority jobs evict lower ones — but preempting a GPU job means checkpointing (expensive) or killing (wasteful); the cost of preemption is much higher than for CPU jobs, so policies lean toward priority admission over mid-flight eviction.
  • Fair-share / DRF: each tenant gets a fair slice over time; prevents one team monopolizing the cluster.
  • Backfill: let small jobs run in the gaps while a big gang job waits for its reservation — utilization without starving the big job.

For inference serving specifically (Phase 07), "scheduling" shifts meaning: it's admission control and continuous batching at millisecond granularity on an already-placed model, not job placement at minute granularity. A platform leader keeps these two scheduling worlds — training-job placement (this phase) and request admission (Phase 07) — distinct in design and in conversation.

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02 (cluster view, then the node-level k8s mechanism).

Lab 01 (cluster sim, Python):

  1. python solution.py; read the policy comparison table, then the scheduler core against Ch. 6–7.
  2. Find the stranded-GPU result in first-fit vs best-fit — explain it with the Ch. 6 fragmentation story.
  3. Watch the gang test: with naive allocation two big jobs deadlock; with gang scheduling neither does. Connect to Ch. 7.
  4. Extensions: preemption (with checkpoint cost), DRF, topology islands.

Lab 02 (device plugin, Go):

  1. go test ./... — the mock kubelet drives registration→list→allocate.
  2. Read plugin.go against Ch. 8; trace one pod's allocation end to end.
  3. Set replicas: 4; re-run; one physical GPU now advertises 4 units. Read the isolation caveat in the test output — that's the allocation≠isolation lesson.
  4. Extensions: MIG profiles as resource names, health transitions.

Success Criteria

  • You can explain why no GPU cgroup controller forces GPU-side sharing (Ch. 1)
  • You can contrast MPS/MIG/time-slicing on isolation, utilization, and use case, and pick from requirements (Ch. 2–5)
  • You can explain cluster fragmentation/stranded GPUs and how best-fit helps (Ch. 6)
  • You can define gang scheduling and the deadlock it prevents, and the NVLink-island argument (Ch. 7)
  • You can draw the k8s device-plugin allocation path and state allocation≠isolation (Ch. 8)
  • Both labs pass; you can map sharing mechanisms to platform tiers

Interview Q&A

Q1: Compare MPS, MIG, and time-slicing. A: All three share one physical GPU because the kernel can't. MPS merges multiple processes into one CUDA context so their kernels run concurrently — best utilization for trusted cooperating jobs, but weak isolation (shared fault domain; memory limits only added in Volta+). MIG physically partitions the GPU into instances with dedicated SMs, L2, and HBM and separate fault domains — the only option with hardware-enforced multi-tenant isolation, at the cost of fixed profiles and reconfiguration rigidity. Time-slicing rotates the whole GPU among clients — trivial oversubscription, zero isolation (shared memory, no QoS, shared faults). Decision: trusted+utilization → MPS; multi-tenant+isolation → MIG; bursty dev/cost → time-slicing; latency-SLO → MIG or dedicated.

Q2: An 8-GPU job won't schedule though 12 GPUs are free. Why? A: Cluster fragmentation — the 12 free GPUs are scattered (e.g., 1–2 per node) and the job needs 8 co-located (gang + likely topology constraint for the collective). No single node/island offers 8, so it can't place despite sufficient total capacity — the stranded-GPU problem, external fragmentation at cluster scale. Fixes: best-fit/consolidating placement to keep whole nodes free, reserving capacity for large jobs, draining/migrating checkpointable small jobs to defragment, and topology-aware admission that knows an 8-GPU TP job wants one NVLink island. The metric to alert on is stranded-GPU count, not raw free-GPU count.

Q3: What is gang scheduling and what does it prevent? A: All-or-nothing allocation for a multi-GPU job: the scheduler places every GPU the job needs simultaneously or none. It prevents the partial-allocation deadlock — two 8-GPU jobs on 16 GPUs each grabbing 6, neither able to start its synchronous collective (Phase 08), neither releasing. Gang scheduling (with reservation/backfill so the cluster isn't idled while assembling a gang) is mandatory for distributed training; serving (single-GPU or already-placed replicas) usually doesn't need it.

Q4: How does Kubernetes schedule onto GPUs? A: Via the device-plugin API. A per-node plugin registers a resource name (nvidia.com/gpu) with the kubelet, streams the healthy device list via ListAndWatch (the node now advertises that many extended-resource units), and the default scheduler treats it like CPU/memory for placement. On binding, the kubelet calls the plugin's Allocate with chosen device IDs; the plugin returns the device nodes, mounts, and env (NVIDIA_VISIBLE_DEVICES) to inject — the Phase 04 container-injection mechanism. Critically the plugin does allocation, not isolation: fractional GPUs are just advertising one device as N units (time-slicing, no isolation) or exposing MIG instances as separate resource names (real isolation).

Q5 (leadership): Design the GPU-sharing strategy for a multi-tenant inference platform with both enterprise and free-tier customers. A: Tier by isolation need. Enterprise/regulated: dedicated GPUs or MIG instances (3g.40gb-class) — hardware isolation is contractual; advertise as distinct k8s resources; predictable QoS. Standard paid: MIG smaller profiles or MPS within a trusted pool with per-client memory limits and SM provisioning, SLO-backed. Free/dev tier: time-sliced oversubscription — cheap, explicitly best-effort, isolated only at the container level (and never co-scheduling free tenants with paid on the same physical GPU, to bound interference and the shared-fault blast radius). Across the cluster: best-fit placement to limit stranding, gang+topology scheduling for any training jobs, and a fragmentation dashboard (stranded GPUs, MIG-profile mix utilization). The platform's job is to expose these as a small menu of honest tiers — each customer picks their point on the isolation/cost curve, and we never silently put them somewhere weaker than they paid for. (This is also where Phase 11's tenant-isolation security story attaches.)

References

  • NVIDIA Multi-Process Service docs — https://docs.nvidia.com/deploy/mps/
  • NVIDIA MIG User Guide — https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
  • NVIDIA GPU time-slicing in Kubernetes — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
  • Kubernetes Device Plugin API — https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
  • NVIDIA k8s device plugin (read the real one) — https://github.com/NVIDIA/k8s-device-plugin
  • Ghodsi et al., "Dominant Resource Fairness" (NSDI 2011)
  • "Gang Scheduling" — Feitelson & Rudolph; and the Kubernetes "coscheduling" plugin / Volcano — https://volcano.sh
  • Google Borg paper (Verma et al., EuroSys 2015) — the cluster-scheduling bible
  • Cross-track: Phase 04 WARMUP Ch. 8–9, Phase 08 WARMUP (why topology matters)