« Phase 28 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 28 — Principal Deep Dive: Operating an AI Platform on Kubernetes

The Deep Dive traced the loops. This doc is about the platform those loops run inside: a managed Kubernetes cluster (OKE) hosting an LLM inference service, shipped through a supply chain, watched by an SLO. The through-line is that on Kubernetes you do not keep services up by watching them — you compose control loops, gates, and budgets so the platform keeps itself up and pages a human only when the math says so. Everything below is the architecture that makes the three labs' mechanisms hold in production.

The managed control-plane / data-plane split is the architecture

The most important structural fact about a managed cluster is that Oracle operates the control plane (API server, etcd, controllers) and you own the data plane (node pools). This split governs every capacity, cost, and blast-radius decision. The control plane is where desired state lives and where the reconciliation loops run — you cannot break it, but you also cannot tune it; the data plane is where your money and your failures live. Designing to this split means: node pools are your unit of capacity and your upgrade mechanism (roll a new pool at the new version, cordon-drain-delete the old — nodes as cattle), and the cluster is multi-pool by design — a general CPU pool for stateless services, a tainted GPU pool for inference so your expensive silicon never fills with cron jobs, and maybe a spot pool for interruptible batch. The portable skill is the shape: every managed Kubernetes (OKE/EKS/GKE/AKS) is "managed control plane + your node pools + cloud-integrated LB/IAM/registry/secret-store." Learn one precisely and the others are renames.

The autoscaler chain is a composed capacity envelope, not one knob

There is no single "how does it scale" answer — the envelope is a composition of four autoscalers on four different axes, and confusing them is the classic failure. The chain that turns a traffic spike into new hardware with no human:

  • HPA changes replica count from pod metrics (ceil(current × metric/target)). It answers "how hot are the pods that exist" — and fails for queue-driven AI workloads, because a queue can be exploding while one worker's CPU looks calm, and HPA cannot scale from zero.
  • KEDA reads the event source itself (queue depth, consumer lag), activates 0 → 1, and drives an HPA for 1 → N. For GPU inference fed by a request queue, "KEDA on queue depth" is the standard cost-efficient signal.
  • Cluster Autoscaler does not watch utilization — it watches for Pending pods that failed scheduling, simulates whether a new node from some pool would let them schedule, and grows that pool; it scales down by draining underutilized nodes whose pods can be safely rehomed (respecting PodDisruptionBudgets).
  • VPA right-sizes a pod's own requests — best run in recommendation mode, and never fighting HPA over the same metric.

The chain: KEDA/HPA make pods → pods go Pending → Cluster Autoscaler makes nodes. That Pending state is not a failure; it is the signal the CA is designed to consume. A principal sizes the whole envelope — HPA/KEDA bounds, node-pool min/max, PDBs — as one system, not four toggles.

Capacity is arithmetic: requests, density, elasticity

Cost on Kubernetes is mostly three ratios you can move (Warmup §16), and a principal reasons in them:

  1. requests : actual usage (right-sizing). Requests are the currency of scheduling, so a pod requesting 4 CPU and using 0.5 strands 3.5 CPU of every node it lands on — invisible waste. Fix with p95 usage from Prometheus or VPA recommendations, shrunk toward reality with headroom.
  2. allocated : provisioned (bin-packing density). Even right-sized pods spread thin across half-empty nodes waste money; CA scale-down and consolidation raise density — bounded by blast radius (one huge node failing takes more with it) and PDB-safe drainability.
  3. provisioned : needed-at-peak (elasticity). Autoscale so provisioned tracks demand instead of peak-all-day; run interruptible work on spot pools; scale bursty inference to zero.

Capacity planning closes it: forecast peak QPS × per-request resource cost (from load tests) + failure headroom (N+1 across availability/fault domains — survive one domain down at peak) + growth, revisited against actuals. For GPU fleets the highest-leverage number is DCGM utilization — a 30%-utilized H100 pool is the single best cost-reduction target in an AI platform.

Failure modes and blast radius

The interesting failures are configuration and reasoning errors, not crashes:

  • Dependency check in a liveness probe. A liveness probe that pings the database restarts the whole healthy fleet when the database blips. Blast radius: every replica. Readiness checks dependencies; liveness checks only "is this process wedged."
  • The mutable-tag promotion. Promoting a :latest tag instead of a digest lets staging and prod silently diverge — prod runs bytes that never passed staging's gates. Blast radius: prod, invisibly. The cure is digests end to end and verify-before-deploy at admission.
  • CPU limits on a latency-sensitive service. CPU limits throttle via the kernel CFS quota — a common cause of tail latency, not a protection. Blast radius: p99 on that service. Requests always, memory limits yes, CPU limits only deliberately.
  • A single-window burn alert. Too twitchy pages on blips (alert fatigue → real pages ignored); too slow pages after the budget is gone. Blast radius: on-call trust. Multi-window is the only scheme both fast and quiet.
  • Over-broad RBAC. cluster-admin bound to humans for convenience, or workloads on the default ServiceAccount with leftover permissions. Blast radius: the whole API surface, until an audit.

The "looks wrong but is intentional" decisions

  • Level-triggered, not event-driven. Re-deriving work from state every pass looks wasteful next to reacting to events — and it is exactly why the system survives its own components restarting and missed events. Idempotent convergence beats a perfect event stream you will eventually drop.
  • The scheduler ignores live usage. Packing against requests rather than utilization looks like it wastes capacity — but it makes scheduling deterministic and capacity plannable, and it is why requests sizing is a first-class engineering act, not a guess.
  • NetworkPolicy defaults to allow-everything. Looks insecure — but the model is additive allows, so isolation is opt-in per namespace via a default-deny policy, and the CNI must actually enforce it. Explicit beats implicit-and-surprising.
  • GitOps pulls instead of pushing. A build server running kubectl apply with cluster-admin is simpler — and exactly the credential-blast-radius GitOps eliminates: the pull model puts a reconciling agent inside the cluster, so no CI system holds prod credentials, drift self-heals, and rollback is git revert.

Cross-cutting concerns

Security composes two orthogonal gates: RBAC decides who may write which API objects; admission (Pod Security Standards, Kyverno/Gatekeeper, cosign verify-before-deploy) decides what objects are acceptable. Pod security is an admission concern, not RBAC. Both build on Phase 09's primitives: restricted PSS forces non-root, dropped capabilities, seccomp RuntimeDefault — the cluster forcing every pod onto the isolation Phase 09 built for one process.

Observability is three legs answering three questions: metrics (aggregate — RED for services, USE for resources), logs (what exactly happened here — shipped by a DaemonSet to Loki/Elasticsearch because the kubelet keeps only a rotating window), traces (where in the fan-out the latency went — the only leg that localizes a regression across a gateway → retriever → three model calls). The #1 operational hazard is cardinality explosion: a label with unbounded values multiplies series until the TSDB falls over.

Reliability is the budget: an SLO turns "is it reliable" into a spendable resource — plenty left, ship faster; exhausted, freeze and fix — and burn-rate paging is the only scheme that is both fast and quiet. DR is two numbers agreed in advance (RPO/RTO); GitOps shrinks cluster-rebuild RTO to "recreate with Terraform, point Argo CD at the repo," and a DR plan is a hypothesis until a game day runs the restore.

Where this fits the platform

The AI-serving specialization sits on top of all of it: GPUs are extended resources (nvidia.com/gpu via the device plugin, MIG for real multi-tenant isolation, time-slicing only for non-prod); startup is heavy (multi-GB weights → startup probes, model-cache PVCs, so liveness does not kill a loading pod); readiness must mean "model loaded and warm," not "port open"; and rollout is a distribution problem, not a boolean — canary or shadow by traffic percentage, promoted on quality metrics an eval harness judges, not on 5xx alone. The senior architecture ships itself: control loops schedule and heal, gates promote signed artifacts, budgets decide when to page.