Lab 01 — Reconciliation Orchestrator: a mini Kubernetes control loop

Phase 28 · Lab 01 · Phase README · Warmup

The problem

Kubernetes' single most important idea — the one an interviewer will keep circling back to — is declarative, desired-state reconciliation. You never tell the cluster "run this pod on node-3." You declare the desired state ("4 replicas, each requesting 500m CPU / 256Mi") and a controller — a control loop — continuously compares actual state to desired state and takes the actions that close the gap. Self-healing, scaling, and rolling updates are not three separate features; they are the same loop applied to different desired states.

If you can build that loop, you understand Kubernetes at the level that matters when a Deployment is stuck at 2/4 ready at 2 a.m. and you need to know why the controller isn't converging — is it a scheduling failure (no node fits), a capacity problem, or a rollout wedged because maxSurge=0 and there's nowhere to put the surge pod?

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• reconcile() converges actual pod count to replicas and is idempotent (a second call is a no-op).
• Bin-packing respects CPU and memory; a pod that fits no node is Pending, not an error.
• Deleting a running pod, then reconciling, recreates it (self-heal).
• Failing a node evicts its pods and the next reconcile reschedules them onto survivors — and never back onto the failed node.
• HPA.desired_replicas implements ceil(currentReplicas × value/target), clamps to [min, max], and holds steady inside the tolerance band.
• A rolling update converges to the new version while ready never drops below desired − maxUnavailable and total never exceeds desired + maxSurge.
• A rollout with maxSurge=0 and maxUnavailable=0 raises RolloutStalledError (no way to make progress) — the real "stuck rollout" failure.
• rollback() restores the prior revision; re-rolling returns every pod to the old version.
• All 32 tests pass under both lab and solution.

How this maps to the real stack

• The reconciliation loop is the real controller pattern. Every Kubernetes controller (Deployment, ReplicaSet, StatefulSet, Job) runs a for { actual := observe(); act(diff(desired, actual)) } loop against the API server's watch stream. Our reconcile() is that loop made synchronous and deterministic so you can assert on it. A real controller is level-triggered (it acts on the current observed state, not on an event), which is exactly why our idempotent reconcile() converges whether you call it once or ten times — the property that makes Kubernetes robust to missed events and restarts.
• The scheduler (kube-scheduler) really does a filter (which nodes have enough allocatable CPU/memory/GPU and satisfy affinity/taints) then score (spread, bin-pack, etc.) pass. Our first-fit-by-sorted-name is the filter step with a trivial deterministic scoring; the real one adds PodAffinity, NodeAffinity, Taints/Tolerations, PodTopologySpread, and pluggable scoring plugins. Crucially, real scheduling is request-based: a pod's resources.requests (not its live usage) is what the scheduler packs against — our _node_free does the same.
• Pending is the real pod phase when nothing fits (FailedScheduling, "0/N nodes are available: insufficient cpu"). This is the single most common "why isn't my pod running" incident, and the answer is almost always requests vs allocatable, a taint, or a missing node pool — exactly the mechanics this lab makes concrete.
• Self-healing is real: kill a pod and the ReplicaSet controller recreates it; a NotReady node past its eviction timeout has its pods deleted and rescheduled. Our fail_node collapses the node-controller eviction + rescheduling into one deterministic step.
• The HPA formula is verbatim from the Kubernetes HPA controller: desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue)), clamped to minReplicas/maxReplicas, with a default 10% tolerance (--horizontal-pod-autoscaler-tolerance). Real HPAs read the metric from the metrics.k8s.io / custom.metrics.k8s.io / external.metrics.k8s.io APIs (Metrics Server, Prometheus Adapter, KEDA); we inject the metric value directly, which is the same seam.
• The rolling update maxSurge/maxUnavailable semantics are exactly Kubernetes': total pods ≤ replicas + maxSurge, available pods ≥ replicas − maxUnavailable. Our step-and-snapshot loop is how you'd reason about (and test) a Deployment's RollingUpdate strategy; the real controller drives two ReplicaSets (old and new) up/down under the same budget.
• Rollback maps to kubectl rollout undo — the Deployment keeps a bounded revision history (.spec.revisionHistoryLimit) and rolling back is a normal rolling update toward a saved PodTemplate.

Limits of the miniature. There is no API server, no etcd, no watch stream, no controller manager — reconcile() is called by hand instead of running continuously. Readiness is a boolean phase, not a real readiness probe with initialDelaySeconds/periodSeconds. Scheduling ignores affinity, taints/tolerations, topology spread, and GPU/extended resources (the Warmup covers all of these). "Node failure" is instantaneous eviction; real Kubernetes waits out node-monitor-grace-period and the pod eviction timeout. And there is no PodDisruptionBudget gating voluntary disruptions — the rolling update's availability guarantee here is enforced purely by the maxUnavailable arithmetic.

Extensions (your own machine)

• Add taints/tolerations: a node carries taints, a pod carries tolerations, and _schedule filters out nodes whose taints the pod doesn't tolerate — then dedicate a "GPU" node pool to pods that tolerate nvidia.com/gpu.
• Add pod anti-affinity: refuse to place two replicas of the same Deployment on the same node, and watch how it changes the Pending math when you have fewer nodes than replicas.
• Add a PodDisruptionBudget object and make fail_node/rolling update respect minAvailable.
• Add a Cluster Autoscaler: when a pod is Pending for lack of capacity, "provision" a new node (append to self.nodes) and reschedule — then scale the node back down when it's idle.
• Replace first-fit with most-allocated (bin-pack) vs least-allocated (spread) scoring and compare the resulting placements.

Interview / resume signal

"Built a deterministic miniature of the Kubernetes control plane: a level-triggered reconciliation loop that bin-packs pods against CPU/memory requests (Pending when nothing fits), self-heals on pod deletion and node failure, an HPA implementing the real ceil(replicas × value/target) formula with a tolerance band, and a maxSurge/maxUnavailable rolling update proven to hold availability within budget — plus revision rollback."