Warmup Guide — Orchestration: Kubernetes & RKE2

Zero-to-expert primer for Phase 06. You need no Kubernetes background: this guide starts at "what does an orchestrator do" and ends with you building an operator reconcile loop and reasoning about accelerator scheduling. The through-line is the reconcile loop — declared-vs-discovered convergence, the same idea from Phases 01 and 05.

Table of Contents

Chapter 1: Why Orchestration, and Where Rack Software Fits

Once nodes are provisioned (Phase 05), something must decide what runs where, restart failed work, scale it, and manage its lifecycle across the fleet. That's orchestration, and Kubernetes (k8s) is the de-facto standard, including for AI workloads (often via hardened distros like RKE2 and add-ons for accelerators).

Rack-management software relates to k8s in three ways — keep them distinct:

  1. It runs on k8s: your agents/exporters deploy as pods/DaemonSets (one per node).
  2. It integrates with k8s: it consumes the API to cordon/drain a node for maintenance, label nodes with hardware facts, or react to events.
  3. It extends k8s: you write operators (controllers + CRDs) that teach the cluster rack-specific operations — "a RackNode should be at firmware baseline X and healthy" — so rack lifecycle becomes declarative, self-healing software.

The mental anchor: Kubernetes is a declarative, level-triggered control system. You declare desired state (objects); controllers continuously reconcile reality to it. If that sounds like config management (Phase 05) and inventory reconciliation (Phase 01) — it is the same idea, which is why this curriculum keeps returning to it.

Chapter 2: The Kubernetes Control Plane

The control plane is a set of components around a database:

  • etcd: the consistent key-value store holding all cluster state (the source of truth).
  • API server (kube-apiserver): the only thing that talks to etcd; a REST API where everything is an object (validated, versioned, RBAC-controlled). All reads/writes go through it. kubectl apply is an API write.
  • Scheduler (kube-scheduler): watches for unscheduled pods and assigns each to a node (Chapter 5). It only decides; it doesn't run anything.
  • Controller manager: runs the built-in controllers (Deployment, ReplicaSet, Node, …), each a reconcile loop driving actual toward desired.
  • On each node: kubelet (runs/monitors the node's pods via the container runtime over CRI) and kube-proxy (service networking).

The flow for kubectl apply -f deploy.yaml: API server validates and writes the Deployment to etcd → the Deployment controller creates a ReplicaSet → the ReplicaSet controller creates Pods (desired count) → the scheduler binds each Pod to a node → that node's kubelet pulls the image and starts the container → controllers keep watching and correcting. No component gives orders imperatively; each watches state and reconciles. That's the whole design.

Chapter 3: Workloads, Nodes, and the DaemonSet Agent

The objects you'll use:

  • Pod: the smallest deployable unit (one or more co-located containers). Usually you don't create pods directly; a controller does.
  • Deployment → ReplicaSet → Pods: stateless apps; declare replicas: N, get N pods, self-healed and rolling-updatable.
  • DaemonSet: one pod per node (optionally a node subset). This is the rack-agent pattern — your node-level rack-management agent, the metrics exporter (Phase 07), the device plugin, and the firmware agent all deploy as DaemonSets, so every node runs exactly one copy automatically as nodes join/leave.
  • StatefulSet: stable identity/storage for stateful workloads (databases).
  • Job/CronJob: run-to-completion / scheduled tasks (a one-shot firmware update job).

Each pod declares requests (guaranteed resources, used for scheduling) and limits (caps). Getting requests right is how the scheduler bin-packs without overcommitting — and it's where accelerators plug in (Chapter 6).

Chapter 4: The Operator Pattern and the Reconcile Loop

This is the chapter that matters most for this role, because it's how you encode rack operations as cluster-native software.

CRD (CustomResourceDefinition): extends the API with your own object kind. Define a RackNode CRD and now kubectl get racknodes works and the API server stores/validates them like built-ins. The CRD has a spec (desired state, written by users/automation) and a status (observed state, written by your controller).

The reconcile loop (the operator's core):

loop forever:
  obj = observe(current state of the RackNode + the real world)
  diff = desired(spec) - actual(status/world)
  if diff: act to reduce it (idempotently)
  update status
  requeue (after a delay, or on the next event)

Properties that make it robust (and are interview gold):

  • Level-triggered, not edge-triggered: it reacts to current state, not to the event that changed it. So a missed event isn't fatal — the next reconcile sees reality and fixes it. Edge-triggered (act on the change notification) breaks if you miss an event; level- triggered is self-healing. This is why Kubernetes is reliable.
  • Idempotent: reconcile may run many times; each run converges and is a no-op once matched (Phase 05's idea, exactly).
  • Requeue + backoff: a failed action requeues for retry rather than crashing.
  • Finalizers: a marker on an object that blocks deletion until your controller does cleanup (e.g., decommission the physical node, remove it from inventory) — then removes the finalizer to allow deletion. Without finalizers, deleting the object would orphan real-world resources.
  • Owner references: child objects owned by a parent are garbage-collected with it.

A RackNode operator might reconcile: "spec says firmware baseline 1.2 and role=compute; actual says firmware 1.1 → trigger a firmware job (Phase 08), update status, requeue until matched; on delete, run the decommission finalizer." That's rack lifecycle as self-healing software — the "automated SRE" promise of operators. Lab 01 builds this loop.

Chapter 5: Scheduling — How a Pod Lands on a Node

The scheduler runs two phases per unscheduled pod:

  1. Filter (predicates): eliminate nodes that can't run it — insufficient requests-vs-allocatable, missing required labels (nodeSelector/affinity), or taints the pod doesn't tolerate.
  2. Score (priorities): rank the survivors — spread vs bin-pack, affinity preferences, topology spread, image locality — and pick the best.

Key levers a rack operator uses:

  • requests/limits: requests drive filtering/bin-packing; a pod requesting more than any node's allocatable stays Pending forever (a classic "why is my pod Pending" answer).
  • Taints & tolerations: taint a node (maintenance:NoSchedule, or dedicated=tenantA) and only pods tolerating it land there. This is how you drain a node for maintenance (cordon = unschedulable; taint+drain evicts), and how you dedicate accelerator nodes.
  • Affinity/anti-affinity & topology spread: co-locate or spread pods (e.g., spread replicas across failure domains — Phase 01 — so one PDU/ToR loss doesn't take all replicas).
  • Topology-aware placement: for accelerators, place a pod where its GPU and NIC share a NUMA node / PCIe root (Phase 04) for performance.

"Why is this pod Pending?" → walk the filter phase: capacity, taints, affinity, no matching node. Lab 02 makes this concrete.

Chapter 6: Accelerators — Device Plugins & Topology

Kubernetes doesn't natively know about GPUs/AI accelerators; the device plugin API teaches it:

  • A device-plugin DaemonSet runs on each node, discovers the accelerators, and advertises them to the kubelet as an extended resource (e.g., qualcomm.com/ai-accelerator: 4).
  • Pods request them like any resource: resources.limits["qualcomm.com/ai-accelerator"]: 1.
  • The scheduler now treats accelerators as schedulable capacity (filter/score), and the kubelet, via the plugin, allocates specific devices to the container (and sets up cgroups/device access).

Beyond basic counting:

  • Node Feature Discovery (NFD) labels nodes with hardware facts (PCIe IDs, capabilities) so scheduling can target them.
  • Topology-aware allocation places a pod's accelerators and NIC on the same NUMA node/PCIe root (Phase 04) — a real performance lever the Topology Manager handles.
  • Partitioning (MIG-style fractional accelerators) is exposed as multiple smaller resources — the multi-tenancy/utilization tradeoff.

The "accelerator operator" pattern bundles the device plugin, NFD, driver management, and monitoring into one operator — the kind of thing this role builds or integrates.

Chapter 7: RKE2 and Production Distros

Upstream Kubernetes is a kit; production clusters run a distribution. RKE2 (Rancher Kubernetes Engine 2, aka RKE Government) is a hardened, secure-by-default distro — relevant because the JD names it and because secure/sovereign deployments (think a government data center in the Kingdom) demand exactly its properties:

  • CIS-hardened out of the box; FIPS-capable; designed to pass security benchmarks.
  • Simple, self-contained install (a single binary, embedded etcd) — good for air-gapped deployments (no internet during install), a recurring requirement in sovereign/regulated environments.
  • containerd runtime, bundled CNI, no Docker dependency.

Context for the family: k3s (lightweight, edge), RKE2 (hardened, datacenter/gov), OpenShift (enterprise, opinionated), upstream/kubeadm (DIY). They're all Kubernetes; the differences are security posture, install model, and bundled components. Knowing why RKE2 (hardening + air-gap + CIS) signals you understand the deployment context, not just the API.

Chapter 8: Multi-Tenancy & Cluster Lifecycle

Multi-tenancy in k8s (pairs with Phase 09's network/VLAN isolation):

  • Namespaces partition objects; RBAC controls who can do what; ResourceQuotas cap a tenant's consumption; NetworkPolicies restrict pod-to-pod traffic; PodSecurity standards constrain what pods can do.
  • Soft multi-tenancy (namespaces + quotas + RBAC) suits trusted tenants; hard isolation (separate clusters/nodes, or strong sandboxing) suits untrusted ones. For accelerators, not co-placing competing tenants on one physical device (Phase 04/09) is part of this.

Cluster lifecycle (where rack-management software acts on k8s):

  • Node join: a freshly provisioned node (Phase 05) joins the cluster.
  • cordon: mark a node unschedulable (no new pods) without evicting.
  • drain: evict pods (respecting PodDisruptionBudgets) to free a node for maintenance — what you do before a firmware update (Phase 08) or a hardware swap.
  • upgrade: roll the k8s/OS/driver version node-by-node — cordon → drain → upgrade → uncordon, canaried (Phase 11), image-pinned for rollback.

These verbs are how your rack operator safely takes a node out of service and back. The maintenance flow (cordon → drain → act → validate → uncordon) is a core operational primitive and a frequent system-design detail.

Lab Walkthrough Guidance

Order: Lab 01 (operator) → Lab 02 (scheduling). The reconcile loop is the conceptual core; scheduling is how workloads (and your agents) land on the hardware.

Lab 01 (RackNode operator):

  1. python3 solution.py — a controller reconciles RackNodes: provisions to desired, corrects drift, and deletes with a finalizer-driven cleanup.
  2. Read the reconcile function; note it's level-triggered (acts on current state) and idempotent (re-reconcile is a no-op).
  3. Inject drift and a transient action failure; watch it requeue and converge.
  4. Extensions: reimplement with kopf against a real kind cluster + a CRD YAML.

Lab 02 (device plugin + scheduling):

  1. python3 solution.py — a mini scheduler places accelerator-requesting pods across nodes honoring capacity, taints, and topology.
  2. Over-subscribe accelerators and watch pods go Pending; add a maintenance taint and watch placement avoid that node.
  3. Read the real device-plugin DaemonSet + pod manifests.
  4. Extensions: run NVIDIA's device plugin on a real GPU; add MIG-style fractional resources; add gang scheduling.

Phase capstone question: "Design how rack-management runs and integrates with Kubernetes for an accelerator fleet." (Answer: node agent + exporter as DaemonSets; a RackNode operator encoding lifecycle/firmware/health as a reconcile loop; device plugin + NFD + topology-aware scheduling for accelerators; cordon/drain/upgrade for maintenance; namespaces+RBAC+quota+ NetworkPolicy for tenancy; RKE2 for a hardened, air-gappable base.)

Success Criteria

You're done with this phase when — without notes:

  • You can trace kubectl apply through the control plane to a running container (Ch. 2)
  • You can explain the DaemonSet rack-agent pattern (Ch. 3)
  • You can write/explain a reconcile loop and why it's level-triggered and idempotent (Ch. 4)
  • You can explain spec vs status, finalizers, and owner references (Ch. 4)
  • You can diagnose a Pending pod via the filter phase (capacity/taint/affinity) (Ch. 5)
  • You can explain the device-plugin path from hardware to schedulable resource + topology (Ch. 6)
  • You can say what RKE2 is and why hardened/air-gapped distros exist (Ch. 7)
  • You can describe the cordon→drain→upgrade maintenance flow (Ch. 8)

Interview Q&A

Q1: What happens when you kubectl apply a Deployment? A: The request hits the API server, which authenticates, validates, and writes the Deployment object to etcd. The Deployment controller (a reconcile loop) sees it and creates a ReplicaSet; the ReplicaSet controller creates the desired number of Pods. The scheduler watches for unscheduled Pods and, for each, filters nodes (capacity, taints, affinity) and scores the survivors, then binds the Pod to a node. That node's kubelet sees the binding, pulls the image via the container runtime (CRI/containerd), and starts the container, reporting status back. From then on the controllers keep watching and reconciling — if a pod dies, the ReplicaSet controller makes a new one. Nothing is imperative; every component watches state and converges it, which is the whole Kubernetes design.

Q2: Explain the operator pattern and the reconcile loop, and why it's level-triggered. A: An operator extends Kubernetes with a CRD (a custom object like RackNode) plus a controller that runs a reconcile loop: observe current state, diff desired (spec) against actual (status/world), act idempotently to close the gap, update status, and requeue. It's level-triggered — it reacts to current state, not to the event that changed it — so if a notification is missed or duplicated, the next reconcile still sees reality and corrects it. Edge-triggered logic (act on the change event) breaks when you miss an event; level-triggered is self-healing, which is why it's reliable. The loop must be idempotent (many runs converge, no-op once matched), requeue with backoff on failure, and use finalizers to do cleanup before an object is deleted. It's the same declared-vs-discovered convergence as configuration management — operators just apply it to cluster-native objects.

Q3: What's the difference between spec and status, and what is a finalizer for? A: Spec is the desired state, written by users or automation; status is the observed state, written by the controller. The controller's job is to make status (and the real world) match spec. A finalizer is a key in an object's metadata that blocks its deletion: when you delete the object, Kubernetes sets a deletion timestamp but keeps the object until all finalizers are removed. The controller sees the deletion timestamp, performs cleanup (e.g., decommission the physical node, remove it from inventory, release licenses), then removes its finalizer, allowing the object to be deleted. Without finalizers, deleting a RackNode object would orphan the real-world resources it represented.

Q4: How does an accelerator get scheduled to a pod, and why might a pod be Pending? A: A device-plugin DaemonSet on each node discovers the accelerators and advertises them to the kubelet as an extended resource (e.g., qualcomm.com/ai-accelerator: 4). A pod requests some via resources.limits, the scheduler treats them as schedulable capacity (filter + score), and the kubelet (via the plugin) allocates specific devices and wires up access. A pod goes Pending when the scheduler can't find a node that passes the filter phase: not enough free accelerators/CPU/memory anywhere, a required nodeSelector/affinity matches no node, or every candidate node has a taint the pod doesn't tolerate (e.g., maintenance, or dedicated to another tenant). For accelerators specifically, requesting more than any single node has, or topology constraints that can't be satisfied, leaves it Pending. The fix is reading the scheduler events and the node's allocatable vs requested.

Q5: What is RKE2 and why choose it over upstream Kubernetes? A: RKE2 is Rancher's hardened Kubernetes distribution — CIS-benchmark-hardened and FIPS-capable by default, with a simple single-binary, embedded-etcd install and bundled containerd/CNI. You choose it when security posture and deployment constraints matter: it passes security benchmarks out of the box and installs cleanly air-gapped (no internet), which is exactly what sovereign, regulated, or government data centers require — relevant for a data center in the Kingdom. Upstream Kubernetes is a kit you'd have to harden and assemble yourself; RKE2 gives you a secure, supportable base. It's still Kubernetes — same API and operators — so my rack operators and DaemonSets run on it unchanged.

Q6 (Staff): How do you safely upgrade Kubernetes (or a driver/firmware) across a 100-node accelerator rack with running tenant workloads? A: Treat it as a controlled, canaried, reversible rollout (Phase 11 thinking). First, ensure PodDisruptionBudgets so draining respects availability. Then per node, in waves: cordon (stop new scheduling) → drain (evict pods, honoring PDBs, so tenant work reschedules elsewhere) → perform the upgrade (k8s/driver/firmware — Phase 08) → validate (node healthy, accelerators enumerated at full PCIe width per Phase 04, smoke test) → uncordon. Start with a 1-node canary, bake, then widen the wave size, with automatic halt-and-rollback if health/ SLOs regress (image-pinned so rollback is fast). Respect failure domains (Phase 01) — don't drain a whole PDU/ToR's worth at once. Drive the whole thing from an operator/job so it's declarative, observable (Phase 07), and resumable. The Staff point is that the process — gated, canaried, domain-aware, reversible — matters more than the mechanics, because the cost of a bad fleet-wide upgrade is enormous.

References

  • Kubernetes docs — Concepts (control plane, controllers, scheduling) — https://kubernetes.io/docs/concepts/
  • Kubernetes — Operator pattern & the controller/reconcile model — https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
  • Kubebuilder Book (CRDs, controllers, finalizers, owner refs) — https://book.kubebuilder.io/
  • kopf (Kubernetes operators in Python) — https://kopf.readthedocs.io/
  • Kubernetes Device Plugins & Topology Manager — https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
  • NVIDIA GPU Operator / device plugin (the accelerator-operator pattern) — https://github.com/NVIDIA/k8s-device-plugin
  • RKE2 documentation (hardening, air-gap, CIS) — https://docs.rke2.io/
  • Cross-track: Head of SWE — GPU, Phase 06 (scheduling/virtualization)