Lab 02 — Device Plugin & Accelerator Scheduling

Phase: 06 — Orchestration | Difficulty: ⭐⭐⭐☆☆ | Time: 4–5 hours Language: Python 3 (sim) + real manifests | Hardware: none (kind/GPU for extensions)

Concept primer: ../WARMUP.md Ch. 5–6, ../HITCHHIKERS-GUIDE.md §4–§5.

Run

python3 solution.py     # place pods across nodes; capacity, taints, topology, Pending

Files

  • solution.py — a mini scheduler (filter → score → bind) for accelerator pods.
  • manifests.yaml — the real device-plugin DaemonSet + a pod requesting an accelerator.

0. The mission

Understand how an accelerator goes from hardware to a scheduled pod, and why a pod ends up Pending — by building the scheduler's two-phase placement (filter then score) with capacity, taints/tolerations, and topology (NUMA/PCIe from Phase 04).

1. What the sim shows

  • Capacity: too-big (16 accelerators) fits nowhere → Pending with the per-node reasons (like kubectl describe pod).
  • Taints/tolerations: a maintenance-tainted node only accepts pods that tolerate it (maint-tool can; infer-b can't) — the basis of cordon/drain.
  • Topology: want_topology pods are scored to land where accelerators share the NIC's NUMA node (Phase 04 performance).
  • Binding consumes capacity: placement mutates node usage, so later pods see less room.

2. The real manifests

manifests.yaml shows the device-plugin DaemonSet (one per node, privileged, tolerates everything so it runs even on tainted nodes), a tenant pod requesting qualcomm.com/ai-accelerator: 1 with anti-affinity to spread replicas across PDUs (Phase 01 failure domains), and the taint/cordon/drain commands for maintenance.

3. Extensions (do them)

  1. Run a real device plugin: on a kind cluster with a fake-device plugin (or a real GPU node + NVIDIA's plugin), kubectl describe node to see the advertised resource, then schedule a pod that requests it.
  2. MIG-style fractional resources: advertise accel-slice units and schedule sub-device workloads — the utilization/isolation tradeoff (Phase 09).
  3. Gang scheduling: require all N accelerators of a multi-GPU job to be placed together or none (avoid partial deadlock) — a real AI-training need.
  4. Topology Manager: read /sys NUMA from Phase 04 Lab 02 and feed real adjacency into the scorer.

4. What this lab proves

You can reason about accelerator scheduling end to end — the device-plugin path, why a pod is Pending, and topology-aware placement — which is exactly how rack-management software gets tenant AI workloads onto the right hardware. Combined with Lab 01's operator, you have the two halves of "integrate rack management with Kubernetes."