Phase 06 — Orchestration: Kubernetes & RKE2
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks
Roles supported: Rack Management SWE (Senior/Staff), Platform/Infra SWE, SRE
Hardware needed: none for the core labs (simulators + real manifests); optional kind/k3d for the extensions
Why This Phase Exists
The JD: "Integrate rack management software with orchestration platforms (e.g., Kubernetes, Terraform, Foreman)" · "Implement... software solutions for... workload deployment" · "Knowledge of AI workload orchestration tools (e.g., K8s, RKE2)." Once a rack's nodes are provisioned (Phase 05), the orchestrator is what schedules tenant AI workloads onto them and manages their lifecycle. Rack-management software both runs on and integrates with Kubernetes — and the operator pattern (a controller running a reconcile loop) is the modern way to encode rack-management logic into the cluster itself.
The deep idea you take from this phase is the reconcile loop: a controller continuously
drives actual state toward declared (CRD) state — the same declared-vs-discovered
convergence you met in inventory (Phase 01), provisioning (Phase 05), and config management
(Phase 05). You'll build a RackNode operator that reconciles, and walk the device-plugin +
scheduling path that gets accelerators to pods.
Concepts
Kubernetes core (enough to be dangerous, then deep where it matters)
- The control plane: API server, etcd, scheduler, controller-manager; the declarative API
- Nodes, kubelet, the container runtime (containerd/CRI), pods, the pod lifecycle
- Workload objects: Deployment/ReplicaSet, DaemonSet (the rack-agent pattern), StatefulSet, Job
- Scheduling: requests/limits, node selectors, taints/tolerations, affinity/anti-affinity, topology spread
- Services, the networking model (CNI), namespaces, RBAC, resource quotas (multi-tenancy)
The operator / controller pattern (the heart of the phase)
- CRDs (CustomResourceDefinitions) — extend the API with your own objects (a
RackNode) - The reconcile loop: observe → diff (desired vs actual) → act → requeue; level-triggered, idempotent
- Controllers, informers/watches, the work queue, status vs spec, finalizers, owner references
- Why operators encode operational knowledge as software (the "automated SRE" idea)
Accelerators in Kubernetes
- The device plugin API — how a node advertises accelerators as schedulable resources
- Node Feature Discovery; extended resources; the GPU/accelerator operator pattern
- Topology-aware scheduling (NUMA/PCIe adjacency — Phase 04); MIG-style partitioning concepts
RKE2 and production distros
- What RKE2 is (Rancher's secure, hardened, CIS-compliant k8s distro) and why hardened distros exist
- RKE2 vs upstream/k3s/OpenShift; air-gapped install; embedded etcd; the security posture
- Cluster lifecycle: node join/cordon/drain/upgrade; the relationship to provisioning (Phase 05)
Multi-tenancy & integration
- Namespaces + RBAC + quotas + network policy as tenant boundaries (pairs with Phase 09)
- How rack-management software integrates: as a DaemonSet agent, an operator, and an API consumer
Labs
Lab 01 — A RackNode Kubernetes Operator (reconcile loop)
| Field | Value |
|---|---|
| Goal | Build the reconcile loop at the heart of every operator: a controller that drives RackNode objects from desired (spec) to actual (status), idempotently and level-triggered, handling drift and failures. |
| Concepts | CRDs, spec vs status, the observe→diff→act→requeue loop, idempotency, level- vs edge-triggered, finalizers, requeue/backoff. |
| Steps | 1) python3 solution.py — a controller reconciles RackNodes (provision, drift, delete w/ finalizer). 2) Read the reconcile function. 3) Inject drift and a failed action; watch it requeue and converge. 4) Extensions (real kopf/CRD). |
| Stack | Python 3 stdlib (operator simulator) + a real CRD YAML |
| Output | A working reconcile loop demonstrating convergence, requeue-on-failure, and finalizer-based cleanup. |
| How to Test | Asserts verify convergence to desired state, idempotent re-reconcile (no-op), drift correction, requeue on transient failure, and finalizer cleanup ordering. |
| Talking Points | Why level-triggered beats edge-triggered; spec vs status; what a finalizer is for; why the loop must be idempotent; how this is the same loop as Phase 05 CM. |
| Resume Bullet | "Built a Kubernetes-style operator reconcile loop for a RackNode CRD (desired→actual convergence, requeue/backoff, finalizer cleanup); the control pattern behind rack lifecycle automation." |
| Extensions | Reimplement with kopf against a real kind cluster + a CRD; add a status condition history; add owner references/GC. |
Lab 02 — Device Plugin & Accelerator Scheduling
| Field | Value |
|---|---|
| Goal | Model how a node advertises accelerators and how the scheduler places pods on them, including taints/tolerations and topology (PCIe/NUMA) awareness from Phase 04. |
| Concepts | Device plugin API, extended resources, requests/limits, taints/tolerations, affinity, topology-aware placement, bin-packing vs spread. |
| Steps | 1) python3 solution.py — a mini scheduler places pods requesting accelerators across nodes, respecting capacity, taints, and topology. 2) Read the placement logic. 3) Over-subscribe and watch pods go Pending. 4) Read the real DaemonSet/device-plugin manifests. 5) Extensions. |
| Stack | Python 3 stdlib (scheduler sim) + real k8s manifests |
| Output | A mini accelerator scheduler honoring capacity, taints, and topology; real device-plugin/DaemonSet YAML. |
| How to Test | Asserts verify capacity enforcement, taint/toleration filtering, topology-preferred placement, and Pending when oversubscribed. |
| Talking Points | How device plugins expose hardware; why a pod goes Pending; topology-aware placement (GPU+NIC same NUMA); taints for dedicated/maintenance nodes. |
| Resume Bullet | "Modeled the Kubernetes device-plugin + scheduling path for accelerators with topology-aware (NUMA/PCIe) placement and taint/toleration filtering." |
| Extensions | Run NVIDIA's device plugin on a real GPU node; add MIG-style fractional resources; add gang scheduling for multi-accelerator jobs. |
Deliverables Checklist
- You can describe the k8s control plane (API server, etcd, scheduler, controllers) and the declarative model
- Operator reconcile loop converges desired→actual, is idempotent, and requeues on failure
- You can explain level- vs edge-triggered and why operators are level-triggered
- You can explain spec vs status and what a finalizer does
- Scheduler sim enforces capacity, taints/tolerations, and topology preference
- You can explain the device-plugin path from hardware to a schedulable resource
- You can say what RKE2 is and why a hardened distro exists; node drain/upgrade lifecycle
- You can connect the reconcile loop to Phase 05 CM and Phase 01 declared-vs-discovered
Interview Relevance
- "What happens when you
kubectl applya Deployment?" (control plane → scheduler → kubelet) - "Explain the operator pattern and the reconcile loop. Why level-triggered?"
- "What's the difference between spec and status? What is a finalizer for?"
- "How does a GPU/accelerator get scheduled to a pod?" (device plugin → extended resource)
- "Why might a pod be stuck Pending?" (capacity, taints, affinity, no matching node)
- "What is RKE2 and why use it over upstream k8s?" (hardened, CIS, air-gap)
- "How would you safely upgrade k8s on a 100-node rack?" (cordon/drain/upgrade, canary)