« Phase 28 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 28 — Core Contributor Notes: How Kubernetes and Its Supply Chain Are Actually Built

This is the maintainer's-eye view of the real systems our miniatures imitate — the Kubernetes controller machinery, the scheduler framework, kube-proxy, Argo CD, Sigstore/cosign, and the Prometheus TSDB. The labs reproduce the shapes; the real systems earn their complexity in the seams. Where I describe a pattern rather than a documented internal, I say so — do not quote as fact a detail you cannot verify.

The controller runtime: informers, work queues, and the resync that makes it level-triggered

Our reconcile() scans state directly. Real controllers do not poll the API server in a hot loop — that would melt the control plane. The actual machinery (client-go / controller-runtime) is:

• An informer maintains a local cache of objects of a given kind, populated by a list- then-watch: list everything once, then open a long-lived watch to receive incremental deltas. The cache is what a controller reads, so reads are local and cheap.
• A watch event does not trigger business logic directly. It enqueues the object's key (namespace/name) onto a rate-limited work queue, deduplicated. A pool of workers pops keys and calls Reconcile(key), which reads the current cached state and acts. This is the deep reason reconciliation is level-triggered even though it is fed by an event watch: the event only says "look at this object again," and the handler re-derives its work from present state. A dropped or duplicated event is harmless — the key just gets re-queued, and the periodic resync re-enqueues everything anyway to catch anything missed.
• Writes are optimistic-concurrency via resourceVersion: a stale update gets a 409 Conflict and the controller re-reads and retries. Our lab has no conflict path because it is single-threaded; the real system's correctness under concurrent controllers depends on it.

A committer's takeaway: idempotency is not a style preference, it is a requirement the queue-and- resync architecture imposes — the same key will be reconciled many times, and every extra pass must be a no-op.

The scheduler is a plugin framework, not a for-loop

Lab 01's _schedule is deterministic first-fit. The real kube-scheduler is a framework with extension points that plugins register into: PreFilter, Filter (the predicates — does the node fit requests, tolerate taints, satisfy affinity, have the volume's zone?), Score (rank survivors — NodeResourcesFit, PodTopologySpread, InterPodAffinity, image locality), Reserve, Permit, and Bind. The scheduling cycle picks the highest-scoring node and issues a Binding; the binding cycle is asynchronous so a slow volume attach does not block the next pod. The property our lab keeps faithfully is the one that matters most: the fit check is against requests versus allocatable, not live usage — NodeResourcesFit sums pod requests. What the lab omits: the score phase's weighted plugins, preemption (evict lower-priority pods to make room for a higher-priority Pending pod), and the fact that Pending with a FailedScheduling event is exactly what the Cluster Autoscaler's simulation consumes to decide whether adding a node would help.

kube-proxy: a Service is a virtual IP nothing owns

Our miniature has no networking; the real seam is worth knowing because it is where health meets routing. A ClusterIP Service is a virtual IP — no interface anywhere owns it. kube-proxy on every node programs the kernel so packets to that IP are DNAT'ed to a current ready pod IP:

• iptables mode installs a chain per Service that randomly selects a backend via probability rules — correct but O(rules), so update cost grows with Service count.
• IPVS mode uses the kernel's in-kernel load balancer (hash tables, real scheduling algorithms), which scales to far more Services — the reason large clusters switch.

The backends come from EndpointSlices, and only pods passing their readiness probe are listed. That is the hinge: a rolling update is zero-downtime because unready new pods are absent from the slice and draining old pods finish in-flight requests after SIGTERM + preStop. Break readiness and you break the invariant our Deployment mechanism relies on.

Argo CD: the reconciliation loop lifted to Git

GitOps is §1's loop with Git as desired state, and Argo CD's implementation is instructive. Its unit is the Application (source: repo/path/revision; destination: cluster/namespace). The controller renders manifests (Helm/Kustomize/plain), computes a three-way diff — desired (Git), live (cluster), and last-applied — and reports Synced or OutOfSync. With automated sync plus selfHeal, a manual kubectl edit on prod is reverted within minutes; prune deletes live objects whose manifests were removed. The design decisions a contributor notices: it is a pull model (the agent runs inside the cluster and fetches from Git, so no external CI holds cluster credentials), and drift is a first-class computed state, not an afterthought. Flux ships the same model as toolkit- style controllers (source-controller, kustomize-controller); the ergonomics differ, the loop is identical. Lab 02's env-ladder promotion reappears here as directories/overlays per environment, where "promote to prod" is a PR bumping the image digest.

cosign / Sigstore: signing a digest, and keyless as the real innovation

Lab 02 signs a digest deterministically. Real cosign signs the image's sha256 digest and stores the signature in the registry alongside the image (as an OCI artifact). The genuinely novel part is keyless signing: instead of a long-lived private key, cosign obtains a short-lived certificate from the Fulcio CA bound to an OIDC identity (your CI's workload identity), signs, and records the signature in the Rekor transparency log — so "who signed this and when" is publicly auditable and there is no key to leak or rotate. Verification at deploy time is an admission webhook (sigstore policy-controller, Kyverno verifyImages) that rejects any digest whose signature does not verify against your trust policy. This closes the supply chain: an image that did not come out of your pipeline, or was tampered with after, cannot run. The SBOM (SPDX/ CycloneDX from syft or Trivy) is attached as a signed attestation — the ingredient list that answers "are we running anything with log4j" in minutes. Our lab's deterministic signature over the digest captures the contract (same-artifact is verifiable); it omits the PKI and the transparency log.

Prometheus: pull-based scraping and a TSDB built around counters

Lab 03 builds counter/gauge/histogram and PromQL-lite. The real system's design choices worth knowing: Prometheus is pull-based — it discovers targets via the Kubernetes API and scrapes each /metrics endpoint on an interval into a local TSDB of (metric name + label set) → time series. Counters are cumulative and monotonic on purpose: a missed scrape loses resolution, not truth, and rate() reconstructs the per-second value while tolerating counter resets (a process restart sends the counter to zero; rate detects the drop and corrects). histogram_quantile walks cumulative le buckets and linearly interpolates — the p99 is an estimate whose accuracy was fixed when the buckets were chosen, which is why the newer native (exponential) histograms exist to give better resolution at lower cardinality. The #1 operational failure a maintainer warns about is cardinality explosion: a label with unbounded values (user ID, request ID) multiplies series until the TSDB OOMs. Alerting rules evaluate with a for: duration (breach → pending → firing) and hand firing alerts to Alertmanager, which groups, inhibits, silences, and routes — the dedup and grouping our lab does not model.

What our miniatures deliberately simplify

• Reconciliation: direct state scan instead of informer cache + rate-limited work queue + resync + optimistic-concurrency retries.
• Scheduler: first-fit scoring instead of the filter/score/reserve/permit/bind plugin framework with preemption.
• Networking: absent — no kube-proxy, EndpointSlices, or CNI; readiness-gated endpoints are described, not built.
• CI/CD: deterministic in-process signature instead of cosign's Fulcio/Rekor PKI and an admission webhook enforcing verify-before-deploy.
• Observability: in-memory metric store and hand-driven clock instead of pull-scraping, a persistent TSDB, counter-reset handling, and Alertmanager routing.

Know the shape and the seams, and each system's real documentation reads as confirmation rather than surprise.