🛸 Hitchhiker's Guide — Phase 06: Orchestration

Read this if: you can kubectl get pods but the control plane and operators are fuzzy. Compressed map of the parts, the reconcile loop, and how accelerators get scheduled.

0. The 30-second mental model

Kubernetes is a declarative, level-triggered control system: you write desired state (objects); controllers continuously reconcile reality to it. Rack software relates three ways — it runs on k8s (DaemonSet agents), integrates with it (cordon/drain/label), and extends it (operators encoding rack lifecycle). The one idea to carry: the reconcile loop is the same declared-vs-discovered convergence as Phases 01 and 05.

1. Control plane in one diagram

kubectl ─▶ API server ─▶ etcd (truth)
                │
   ┌────────────┼─────────────┐
 scheduler   controllers     (watch + reconcile)
   │              │
   ▼              ▼
  node: kubelet ─▶ containerd ─▶ pods

No imperative orders — every component watches state and converges it.

2. Workload objects (pick the right one)

ObjectUse
Deploymentstateless app, N replicas, rolling update
DaemonSetone pod per node — the rack-agent / exporter / device-plugin pattern
StatefulSetstable identity/storage
Job/CronJobrun-to-completion / scheduled (a firmware job)

3. The reconcile loop (memorize)

observe → diff(desired spec, actual status/world) → act idempotently → update status → requeue
  • level-triggered (acts on current state, not the event) → self-healing, survives missed events.
  • idempotent (no-op once matched) · requeue+backoff on failure · finalizers for delete-time cleanup · owner refs for GC.
  • spec = desired (users write) · status = observed (controller writes).

4. Why a pod is Pending (the filter phase)

capacity (requests > allocatable anywhere) · a taint it doesn't tolerate · nodeSelector/ affinity matches no node · topology can't be satisfied. Read scheduler events first.

5. Accelerators → pods

device-plugin DaemonSet discovers HW → advertises vendor.com/accel: N extended resource → pod resources.limits requests it → scheduler treats as capacity → kubelet allocates the device. Add NFD (labels), Topology Manager (GPU+NIC same NUMA — Phase 04), MIG (fractional).

6. Maintenance verbs (how rack software acts on k8s)

cordon (no new pods) → drain (evict, honor PDBs) → upgrade/firmware (Phase 08) → validate → uncordon. Canary + domain-aware (Phase 01) + image-pinned rollback (Phase 11).

7. RKE2 in one line

Hardened (CIS/FIPS), single-binary, embedded-etcd, air-gappable Kubernetes — the distro for secure/sovereign data centers. Same API; your operators run unchanged. (k3s=edge, OpenShift=enterprise, kubeadm=DIY.)

8. Multi-tenancy boundaries

namespaces + RBAC + ResourceQuota + NetworkPolicy + PodSecurity (soft); separate clusters/ nodes or strong sandboxing (hard). Pairs with Phase 09 VLAN/QoS isolation.

9. The "done this before" tells

"Is it level- or edge-triggered?" · "Is the reconcile idempotent?" · "Did you add a finalizer for cleanup?" · "Why is it Pending — capacity, taint, or affinity?" · "DaemonSet for the agent, right?" · "Did you drain before the upgrade?"

10. How this phase pays off later

The operator/reconcile loop is the Phase 13 capstone's control core. cordon/drain is the safe path for Phase 08 firmware updates. Tenancy pairs with Phase 09. DaemonSet is how the Phase 07 exporter ships.