Lab 02 — Kubernetes Device Plugin + Fractional GPU (Go)

Phase: 06 — GPU Scheduling | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Go (stdlib) | Hardware: none — a model of the device-plugin contract + a mock kubelet

Concept primer: ../WARMUP.md Ch. 8.

Run

go test ./...        # mock kubelet drives registration -> list -> allocate
go run .             # prints a narrated end-to-end allocation walkthrough

0. The mission

Implement the Kubernetes device-plugin contract for a fake GPU — the real mechanism by which a node advertises nvidia.com/gpu: 8 and a pod gets device nodes injected. Then extend it to advertise fractional GPUs (time-sliced replicas) and MIG profiles as distinct resources, so the allocation ≠ isolation boundary (WARMUP Ch. 8) is concrete code, not a slogan.

We model the three plugin operations that matter (the real API is gRPC over a Unix socket; we keep the semantics and the data flow, in plain Go interfaces, so it runs anywhere with no proto toolchain):

OperationReal APIWhat it does
RegisterRegistration.Registertell the kubelet the resource name
ListAndWatchstreams Device[]advertise healthy device IDs (→ node capacity)
AllocateAllocateResponsereturn device nodes + mounts + env to inject

1. What the walkthrough shows

mode=whole       resource=nvidia.com/gpu        advertised units=4   (4 physical GPUs)
  allocate [GPU-0 GPU-1] -> env NVIDIA_VISIBLE_DEVICES=GPU-0,GPU-1
                            devices /dev/nvidia0 /dev/nvidia1
  isolation: STRONG (dedicated physical devices)

mode=timeslice   resource=nvidia.com/gpu        advertised units=16  (4 GPUs x 4 replicas)
  allocate [GPU-0::3] -> env NVIDIA_VISIBLE_DEVICES=GPU-0  (replica 3 shares GPU-0!)
  isolation: NONE (replicas share memory + faults — WARMUP Ch. 4)

mode=mig         resource=nvidia.com/mig-1g.10gb advertised units=28  (4 GPUs x 7 instances)
  allocate [MIG-0-2] -> env NVIDIA_VISIBLE_DEVICES=MIG-GPU-0/2/0
  isolation: STRONG (hardware-partitioned instance)

The same plugin API advertises all three — but the isolation each delivers is completely different. Time-slicing's "16 GPUs" are 4 physical devices wearing 4 hats each; the plugin happily allocates replica 3 of GPU-0 to a pod with no memory or fault isolation from replicas 0–2. That's the lesson the lab burns in: the device plugin allocates; it does not isolate.

2. Reading order

  1. device.go — the Device, Plugin interface, and the three modes (whole / timeslice / mig) that change only what gets advertised.
  2. plugin.go — ListAndWatch (build the advertised list) and Allocate (pick devices, build the injection response with NVIDIA_VISIBLE_DEVICES).
  3. kubelet_mock.go — the mock kubelet: registers the plugin, reads the device list, and requests an allocation, exactly as the real kubelet does.
  4. plugin_test.go — the contract tests.

3. Extensions

  1. MIG geometry: advertise mixed profiles (1g.10gb, 2g.20gb, 3g.40gb) as separate resource names from the same physical GPUs, respecting that a GPU's instances are mutually exclusive (allocating a 3g consumes capacity the 1gs would have used) — the real MIG accounting problem.
  2. Health transitions: mark a device Unhealthy (model an Xid error from Phase 04/10); ListAndWatch must re-advertise and the kubelet must stop scheduling onto it.
  3. gpu-feature-discovery: emit node labels (nvidia.com/gpu.product=H100, nvidia.com/gpu.memory=80) so the scheduler can do affinity.
  4. Preferred allocation: implement GetPreferredAllocation to pick devices that share NVLink (topology — Phase 08) for multi-GPU requests.

4. Common pitfalls

  1. Conflating advertised units with isolation — the entire point. A test asserts that two timeslice replicas of the same physical GPU resolve to the same NVIDIA_VISIBLE_DEVICES value (proof they share hardware).
  2. Forgetting the env/mount injection — without it the container sees the resource accounted but can't actually reach the GPU (Phase 04 Ch. 9).
  3. Letting Allocate hand out more replicas than the device has, or replicas of an unhealthy device.

5. What this lab proves about you

You can implement the exact integration point between Kubernetes and GPUs, and you can articulate — with running code — why "fractional GPU" means very different isolation guarantees depending on the mechanism. When a customer or a PM asks for "GPU sharing in k8s," you can design the honest answer (which resource, which isolation, which tier — Phase 12).