« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 28 — Kubernetes & Cloud-Native AI Infrastructure (SRE)

Answers these JD lines: the entire "AI Platform SRE — Kubernetes/OKE" role — "provision and operate Kubernetes (OKE) clusters: node pools, autoscaling, network policies, RBAC, secrets management, disaster recovery," "build CI/CD pipelines (Jenkins/GitLab CI/GitHub Actions/Argo CD) with static analysis, container image scanning, unit/integration tests, and staged artifact promotion with quality gates," "observability and incident response: Prometheus/Grafana metrics, alerts and dashboards, log aggregation, distributed tracing, on-call, SLOs and error budgets," "security and compliance: pod security standards, vulnerability remediation, image signing," "infrastructure as code (Terraform, Helm, GitOps)," and "performance tuning: capacity planning, resource quotas, cost-efficient scaling" — plus JD1's "Docker, Kubernetes, CI/CD pipelines, and model monitoring." This is the phase where the agent platform you built in Phases 00–17 gets a place to run and a team that keeps it running.

Why this phase exists

Everything else in this track produces software: an agent loop, a retriever, a gateway, an eval harness. This phase is about the discipline that keeps that software alive: the cluster it runs on, the pipeline that ships it, the telemetry that tells you it's sick, and the SLO math that tells you whether to page a human. Three ideas do most of the work:

1. Declarative desired state + reconciliation. Kubernetes is not a scheduler you command; it is a database of desired state (Deployments, Services, HPAs) plus controllers that relentlessly drive actual state toward it. Self-healing, scaling, and rolling updates are the same control loop applied three ways — and GitOps (Argo CD/Flux) is that identical loop lifted one level up, with Git as the desired state. Lab 01 makes you build the loop.
2. Ship through gates, promote artifacts, never rebuild. A production pipeline is a gate machine: tests, coverage thresholds, a Trivy-style CRITICAL/HIGH vulnerability gate, then a signed, content-addressed artifact promoted dev → staging → prod (prod behind a human approval), with rollback to last-good as a data-structure operation. Lab 02 makes you build the gates.
3. Reliability is arithmetic, not vibes. SLIs measure, SLOs promise, error budgets price the gap, burn rates decide when to page. Under every Grafana panel is rate(), histogram_quantile(), and sum by(); under every good page is a for: debounce and a multi-window burn-rate rule. Lab 03 makes you build all of it.

This phase deliberately complements two earlier ones rather than repeating them: Phase 09 built process-level isolation (namespaces, cgroups, seccomp — what a container is); this phase operates fleets of those containers (what a cluster does), and its pod-security material builds directly on Phase 09's primitives. Phase 14 built agent-runtime observability (token cost meters, OTel-style traces of an agent's reasoning); this phase builds infrastructure observability — Prometheus metrics, alert state machines, SLOs — the layer that tells you the pods running Phase 14's gateway are healthy at all.

Concept map

• The reconciliation model — desired vs actual state, level-triggered controllers, why idempotent convergence beats event scripts (Lab 01).
• Core objects — Pod, ReplicaSet, Deployment, Service, Ingress, ConfigMap, Secret, Namespace; scheduling — requests/limits, QoS classes, affinity, taints/tolerations (Lab 01 + Warmup).
• Autoscaling — HPA (the ceil(replicas × value/target) formula, Lab 01), VPA, Cluster Autoscaler, KEDA for event-driven/scale-to-zero AI workloads.
• Node pools & OKE — CPU vs GPU pools for inference, OCI-specific provisioning, DR across availability/fault domains (Warmup).
• Networking — Services and kube-proxy, CNI, NetworkPolicy default-deny, Ingress (Warmup).
• RBAC, ServiceAccounts & secrets — least-privilege API access, external secret stores (Warmup).
• Health & self-healing — liveness/readiness/startup probes, drain, PodDisruptionBudgets (Lab 01 + Warmup).
• CI/CD & supply chain — pipeline stages, quality gates, Trivy scanning, cosign signing, SBOMs, staged promotion (Lab 02); GitOps — Argo CD/Flux, drift reconciliation (Warmup).
• IaC — Terraform state/plan/apply, Helm charts and releases (Warmup).
• Pod security & admission — Pod Security Standards, admission controllers, policy engines (Warmup, building on Phase 09).
• Observability & SRE — counter/gauge/histogram, PromQL, alert lifecycle, RED/USE, log aggregation, tracing, SLO/error-budget/burn-rate, on-call, blameless postmortems, DR/RPO/RTO (Lab 03 + Warmup).
• Serving AI on K8s — KServe/Seldon, GPU scheduling and sharing, model rollout patterns, capacity and cost tuning (Warmup).

The labs

Integrated scenario (how this shows up at work)

Your team ships an LLM inference service — the model-serving backend for the agent platform — and you own its uptime on OKE. You provision the cluster with Terraform and two node pools — CPU for the API tier, GPU (tainted, tolerated only by inference pods) for the model servers. The Deployment declares requests/limits sized from load tests; an HPA scales the API tier on request rate while KEDA scales GPU workers on queue depth, and the Cluster Autoscaler grows the GPU pool when pods go Pending (Lab 01's mechanics, end to end). NetworkPolicies default-deny east-west traffic; the model bucket credentials live in an external secret store, mounted via a ServiceAccount with exactly one RBAC role. Every commit runs Lab 02's pipeline — static analysis, unit tests with a coverage gate, a Trivy scan that blocks on CRITICAL/HIGH, integration tests — then cosign-signs the image digest, and Argo CD promotes that digest through dev → staging → prod, prod gated on human approval, with the manifests in Git as the single source of truth. Prometheus scrapes RED metrics; Grafana shows p99 time-to-first-token from a histogram; a multi-window burn-rate alert on your 99.9% availability SLO pages on-call (Lab 03). When a bad model rollout burns budget fast, the pager fires within minutes, kubectl rollout undo (or the Argo CD revert) restores last-good, and the postmortem is blameless and produces a new canary gate. Every noun in this paragraph is a section of the Warmup and a mechanism in a lab.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py (32 tests).
• Lab 02 green (29 tests); you can explain why you promote a digest, never rebuild for prod.
• Lab 03 green (32 tests); you can compute a burn rate on a whiteboard.
• You can explain reconciliation (desired vs actual, level-triggered) in under a minute.
• You can name the four autoscalers (HPA/VPA/CA/KEDA) and what each one scales.
• You can walk a CRITICAL CVE from Trivy finding → waiver-or-remediation → redeploy.
• You can state your service's SLI, SLO, error budget, and paging windows without notes.

Key takeaways

• Kubernetes is one idea: declared desired state + controllers that converge actual state to it. Learn the loop once and Deployments, HPAs, GitOps, and operators are all the same story.
• The scheduler packs requests, not usage — which is why requests/limits sizing is a capacity-planning act, and why "it's Pending" is almost always arithmetic.
• A pipeline is only as strong as its gates, and a gate you can silently skip is not a gate. Sign after the gates, verify before every deploy.
• SLOs turn reliability into a budget you can spend on velocity or bank for stability — and burn-rate paging is the only alerting scheme that is both fast and quiet.
• The senior framing: "I don't keep services up by watching them; I build control loops, gates, and budgets so the platform keeps itself up — and pages me only when the math says so."