« Phase 28 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 28 — Deep Dive: The Reconciliation Loop, the Gate Machine, and the Alert as Arithmetic

Three labs, three mechanisms, one shared discipline: compute the desired result from observed state and converge toward it, idempotently, so the same operation run twice is a no-op the second time. Kubernetes reconciliation is that idea as a control loop; a CI/CD pipeline is that idea as a monotone gate ladder; SLO alerting is that idea as arithmetic over counters. This doc traces the actual data structures, invariants, and worked steps each lab forces you to implement — because the mechanism is where the interview and the incident both live.

Reconciliation is a level-triggered loop, and idempotency is the load-bearing property

Lab 01's core is reconcile(), and the single most important fact about it is that it acts on current observed state, not on an event stream. State is a set of Node, Pod, and Deployment records. Desired is a number and a template on the Deployment; actual is the count of live pods matching its selector. Reconcile computes the gap and closes it: fewer pods than desired → create the difference; more → delete the surplus; unscheduled pods → run the scheduler.

The invariant the tests pin is idempotency: running reconcile() a second time with no external change must create and delete nothing. This is not a nicety — it is why the design survives its own components crashing. An edge-triggered system reacts to deltas ("a pod died"), so a missed event leaves it permanently wrong; a level-triggered system re-derives its work from state every pass, so a missed event self-corrects on the next tick. That is the deep reason self-healing is not a feature bolted on — it is the loop. delete_pod drops actual below desired; the next reconcile recreates it. fail_node evicts that node's pods; the next reconcile reschedules them onto survivors. Scaling is the same loop with a different desired number. Once you see it, half the Kubernetes API collapses into corollaries.

The scheduler packs requests, and Pending is arithmetic

_schedule is a filter-then-score pass reduced to its essence: for each unscheduled pod, filter to nodes whose free request capacity admits the pod, then score (the lab uses deterministic first-fit), then bind. The load-bearing subtlety is what "fits" means: the scheduler compares the sum of pod requests against the node's allocatable capacity — never live usage. A node can be 20% utilized and refuse your pod because its requests are fully booked; a node can run over 100% actual CPU and still be schedulable. This single fact explains most real "why isn't my pod running" incidents: the answer is FailedScheduling — 0/N nodes available: insufficient cpu, and the fix is arithmetic (shrink requests, add a node) or labels (a taint not tolerated), never a restart.

A pod that fits nowhere does not error — it sits in Pending. That is the correct behavior and the hinge the next mechanism (autoscaling) hangs on.

The HPA formula, exactly, with the anti-flap machinery

Lab 01 implements the real Horizontal Pod Autoscaler recomputation:

desiredReplicas = ceil( currentReplicas × currentMetricValue / targetMetricValue )

clamped to [minReplicas, maxReplicas]. Trace it: 4 replicas, current CPU 90%, target 50% → ceil(4 × 90/50) = ceil(7.2) = 8, clamped. Two anti-flap mechanisms make it usable in production and both are tested: a tolerance band (roughly 10%) so a ratio near 1.0 produces no action — a metric wobbling around target does not thrash the fleet — and, in the real controller, a scale-down stabilization window that scales down to the maximum recommendation over the window so replicas do not oscillate downward. One structural gotcha the mechanism exposes: CPU-utilization targets are computed relative to requests, so an HPA on a pod with no CPU request cannot compute utilization at all. And HPA cannot scale from zero — zero pods emit no metrics — which is the exact gap KEDA fills by reading the event source directly.

The rolling update is a two-ReplicaSet dance under an availability budget

A Deployment update does not mutate pods in place. It creates a new ReplicaSet for the new template and scales it up while scaling the old one down, bounded by two knobs: maxSurge (how many extra pods above desired may exist) and maxUnavailable (how many below desired may be ready). The invariant Lab 01 proves step by step is the whole point of the mechanism:

ready_count ≥ desired − maxUnavailable      (availability floor, every step)
total_pods  ≤ desired + maxSurge            (surge ceiling, every step)

The update proceeds in bounded increments that never violate either bound, which is why a rolling update is zero-downtime — not because Kubernetes is magic, but because the budget is enforced at every step and unready pods receive no traffic (readiness gates endpoint membership). kubectl rollout undo is not a special path: rollback re-scales a previous ReplicaSet up and the current one down under the same budget. The mechanism pointed backward.

The pipeline is a monotone gate ladder that fails fast

Lab 02's mechanism is a sequence of stages, each a predicate that can only BLOCK or pass, and execution stops at the first block (fail-fast). The stages: build produces a content-addressed digest (sha256:… over the manifest) — the artifact's identity, immutable, unlike a mutable tag; unit tests gate on green AND coverage ≥ threshold; the scan gate is severity ∈ {CRITICAL, HIGH} ⇒ block with accepted risks in a versioned waiver list, never a lowered global threshold; integration tests; then sign the digest (a deterministic signature over sha256:…). Promotion moves the same digest dev → staging → prod — never rebuild — with prod behind manual approval and a verify-before-deploy check that refuses any digest whose signature does not verify.

The three invariants the tests encode: promote artifacts, never rebuild (rebuilding for prod means prod runs bytes that never passed staging's gates); gates block, never skip silently (a gate you can bypass is not a gate); and signing makes "same artifact" a check, not a promise. Rollback is redeploying the previous known-good digest — a data-structure lookup, prepared before the incident.

The alert is arithmetic over counters, and the for: is what makes it quiet

Lab 03 builds the telemetry engine from the metric types up. A counter is monotonic cumulative — designed so a missed scrape loses resolution, not truth — which is exactly why a raw counter is meaningless and rate(x[5m]) (per-second increase over a window) is the only sensible reading. A histogram stores observations in le-bucketed cumulative counters plus _sum/_count; histogram_quantile(0.99, …) walks the cumulative buckets to the bucket containing the 99th percentile and linearly interpolates within it — so the p99 on your dashboard is an estimate whose accuracy you fixed the day you chose the bucket boundaries, not a measured truth. sum by (label) aggregates series sharing a label set.

The alert is a small state machine: an expression breaches → pending → held for the for: duration → firing → resolved when it clears. The for: debounce is the entire difference between an alert and a nuisance; Lab 03 proves a transient one-scrape spike enters pending and clears without ever firing.

Burn rate, and why one window is never enough

The SLO turns reliability into arithmetic. Error budget is 1 − SLO — for 99.9% over 30 days, 0.1%, about 43 minutes of downtime-equivalent. Burn rate is actual error rate ÷ allowed error rate: burn 1.0 spends the budget exactly by window's end; burn 14.4 exhausts a 30-day budget in ~2 days. A single-threshold alert is either too twitchy (fires on blips) or too slow (waits until the budget is gone). Lab 03's fast_burn_alert implements the SRE Workbook answer — a multi- window, multi-burn-rate rule that pages only when burn exceeds the factor over a long window (evidence it is real) AND a short window (evidence it is still happening). The tests prove the quiet case: an incident that already ended leaves the long window remembering and the short window clean, so the AND-of-two-windows does not page. That is the mechanism that is both fast and quiet, and it is impossible to express with a single window.

What to hold onto

Three labs, one mechanism: derive the desired result from observed state and converge idempotently. The controller converges pods to a replica count; the pipeline converges an artifact through gates that only ratchet forward; the alert converges a paging decision from rated counters under a two-window guard. Learn where the observed state, the desired result, and the convergence step live in each, and Deployments, GitOps, and burn-rate paging stop being three topics and become one.