Lab 01 — A RackNode Kubernetes Operator (reconcile loop)

Phase: 06 — Orchestration | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (sim) + a real CRD | Hardware: none (kind for the extension)

Concept primer: ../WARMUP.md Ch. 4, ../HITCHHIKERS-GUIDE.md §3.

Run

python3 solution.py     # provision -> idempotent re-reconcile -> drift -> delete (finalizer)

Files

  • solution.py — the reconcile loop (the operator's core), as a runnable simulator.
  • racknode-crd.yaml — the real CRD + an example RackNode (status subresource enabled).
  • operator.py — the real kopf operator (same logic, real Kubernetes handlers).
  • run-kind.sh + requirements.txt — stand up a real cluster and run the operator.

Run it for REAL on a live cluster (the extension — verified)

Needs Docker + kind + kubectl. The simulator's logic, as an actual operator:

./run-kind.sh up                                   # kind cluster + CRD
pip install -r requirements.txt
kopf run operator.py --verbose                     # the operator (leave running)
# in another shell:
kubectl apply -f racknode-crd.yaml                 # creates node-00
kubectl get racknodes -w                           # PHASE -> Ready, FIRMWARE -> desired
kubectl patch racknode node-00 --type merge \
  -p '{"spec":{"desiredFirmware":"2.2.0"}}'        # drift -> re-converges
kubectl delete racknode node-00                    # kopf finalizer cleanup runs
./run-kind.sh down

This was run end-to-end on a kind v1.31 cluster — observed output:

NAME      PHASE   FIRMWARE   DESIRED
node-00   Ready   2.1.0      2.1.0      # reconciled to baseline (with a level-triggered re-run)
node-00   Ready   2.2.0      2.2.0      # after a desiredFirmware patch: drift corrected
# on delete: "decommissioned node-00 (finalizer cleanup done)"

Note: kopf logs each TemporaryError("converging firmware") as failed temporarily — that's the requeue mechanism (level-triggered re-run), not a real failure. kopf manages its own finalizer because operator.py has an @kopf.on.delete handler.

0. The mission

Build the loop every operator runs and that the JD's "integrate with Kubernetes" and rack lifecycle automation depend on: observe → diff(spec, status/world) → act idempotently → update status → requeue. Make it level-triggered (self-healing) and prove it converges, no-ops when matched, corrects drift, requeues on failure, and cleans up via a finalizer.

1. The five behaviors to internalize

  1. Convergence: a RackNode with desiredFirmware: 2.1.0 drives the world to 2.1.0 and reaches Ready.
  2. Idempotency / level-triggered: re-reconciling a converged node adds nothing and asks for no requeue — it reacts to current state, not events.
  3. Drift correction: an out-of-band firmware downgrade is detected and remediated on the next reconcile (this is why level-triggered beats edge-triggered).
  4. Requeue on transient failure: a flaky firmware action raises, the controller requeues, and it eventually converges — never crashes, never half-applies.
  5. Finalizer cleanup: deleting the object runs the decommission cleanup before the object disappears, so the real node isn't orphaned.

2. Extensions (do them)

  1. Make it real with kopf + kind: kind create cluster, kubectl apply -f racknode-crd.yaml, then write a kopf handler whose @kopf.on.create/update/delete functions mirror reconcile(). Watch kubectl get racknodes show the status converge.
  2. Status conditions with timestamps (Kubernetes-style condition objects: type/status/ reason/lastTransitionTime).
  3. Owner references + GC: have the RackNode own a child Job (the firmware update) so deleting the RackNode garbage-collects it.
  4. Backoff: replace the naive retry with exponential backoff + a max-requeue policy (Phase 02).

3. What this lab proves

You understand the operator pattern at the level that lets you build one, not just use one — the modern way rack-management logic lives in the cluster. "Explain the reconcile loop and why it's level-triggered" becomes a demonstration. This loop is the control core of the Phase 13 capstone, and it's the same convergence pattern as Phase 05 CM and Phase 01 inventory.