« Phase 28 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 28 Warmup — Kubernetes & Cloud-Native AI Infrastructure (SRE)

Who this is for: someone who has built the agent platform (Phases 00–17) and the framework internals (18–24) and now needs to operate all of it — the cluster it runs on, the pipeline that ships it, the telemetry that watches it, and the reliability math that decides when a human gets paged. By the end you will be able to explain, from first principles, how Kubernetes' reconciliation model works and why it beats imperative scripts; how scheduling, autoscaling, networking, RBAC, and self-healing actually behave under the hood; how a production CI/CD pipeline gates, signs, and promotes artifacts; how Terraform, Helm, and GitOps split the infrastructure-as-code job; and how Prometheus metrics, SLOs, and error budgets turn "is it reliable?" into arithmetic. Everything in the labs is mechanism — no cluster, no cloud account, no network.

Table of Contents

  1. The one idea: declarative desired state and reconciliation
  2. The core objects: Pod to Namespace
  3. Scheduling: requests, limits, QoS, affinity, taints
  4. Autoscaling: HPA, VPA, Cluster Autoscaler, KEDA
  5. Node pools and OKE specifics
  6. Networking: Services, kube-proxy, CNI, NetworkPolicy, Ingress
  7. RBAC, ServiceAccounts and secrets management
  8. Health and self-healing: probes, drain, PodDisruptionBudgets
  9. CI/CD: stages, gates, scanning, signing, SBOM, promotion
  10. GitOps: Git as desired state
  11. Infrastructure as Code: Terraform and Helm
  12. Pod security and admission control
  13. Observability: Prometheus, PromQL, Grafana, logs, tracing
  14. SRE: SLOs, error budgets, on-call, postmortems, DR
  15. Serving AI models on Kubernetes
  16. Cost and capacity tuning
  17. Common misconceptions
  18. Lab walkthrough
  19. Success criteria
  20. Interview Q&A
  21. References

1. The one idea: declarative desired state and reconciliation

Strip away every YAML file and CLI flag and Kubernetes is one idea: a database of desired state, plus a set of controllers that continuously drive actual state toward it.

You never command the cluster ("start container X on node 3"). You write a record into the API server — "there should exist a Deployment named inference, 4 replicas, this image, these resource requests" — and control loops do the rest. The Deployment controller sees a Deployment with no matching ReplicaSet and creates one. The ReplicaSet controller sees a ReplicaSet whose desired count is 4 but observes 0 pods and creates 4 Pod records. The scheduler sees pods with no node assignment and binds each to a node that fits. The kubelet on each node sees pods bound to it that aren't running and starts their containers. Five independent loops, none of which knows about the others, each doing the same thing: observe actual, compare to desired, act to close the gap.

Two properties of this design carry all the production weight:

  • Level-triggered, not edge-triggered. A controller acts on the current observed state, not on an event stream. If the controller crashes, misses an event, or the network partitions, the next reconciliation pass still converges — because it re-derives its work from state, not from a queue of deltas it might have dropped. This is why Kubernetes survives its own components restarting, and it is the deep reason idempotency is the cardinal virtue of a controller: running reconcile() twice must be a no-op the second time (Lab 01 tests exactly this).
  • Self-healing is not a feature — it's the loop. Delete a pod: actual (3) no longer matches desired (4), so the ReplicaSet controller creates a replacement. A node dies: its pods are eventually evicted, actual drops, the loop restores it on surviving nodes. Scaling is the same loop with a different desired number. A rolling update is the same loop with a second ReplicaSet. Once you see it, half the Kubernetes API collapses into corollaries.

The pattern generalizes upward, too: an operator is a controller you write for your own CRD ("there should exist a PostgresCluster with 3 replicas and daily backups"), and GitOps (§10) is the identical loop with Git as the desired-state store and the cluster as the actual state. Lab 01 makes you build the loop — schedule, heal, scale, roll — precisely because everything else in this phase leans on it.

2. The core objects: Pod to Namespace

The API objects you must know cold, and — more importantly — why each exists:

  • Pod — the atom: one or more containers sharing a network namespace (one IP, localhost between them) and volumes. Pods are mortal and disposable by design; nothing re-creates a bare pod. Multi-container pods exist for sidecars (log shipper, proxy) and init containers (fetch a model before the server starts — an AI-serving staple).
  • ReplicaSet — "keep N identical pods alive." Almost never written by hand; it is the Deployment's mechanism.
  • Deployment — the object you actually ship: a pod template + replica count + an update strategy. A rolling update works by creating a new ReplicaSet for the new template and scaling it up while scaling the old one down, under the maxSurge/maxUnavailable budget (Lab 01 §8 below). kubectl rollout undo re-scales a previous ReplicaSet — rollback is just the same mechanism pointed backward. (Siblings for other shapes: StatefulSet for stable identity/storage, DaemonSet for one-per-node agents like log shippers and GPU device plugins, Job/CronJob for run-to-completion work like batch inference.)
  • Service — a stable virtual IP and DNS name over an ever-changing set of pod IPs, selected by label. Pods die and get new IPs constantly; the Service is the indirection that makes that survivable (§6 for the mechanism).
  • Ingress — L7 (HTTP) routing from outside the cluster to Services: host/path rules, TLS termination. An Ingress is only data — an ingress controller (NGINX, Traefik, a cloud L7 LB) must be running to act on it; its successor API, Gateway API, splits the same job into role-oriented resources.
  • ConfigMap / Secret — configuration injected as env vars or mounted files, decoupled from the image (the twelve-factor split). A Secret is the same object with base64-encoded data and slightly stricter handling — base64 is not encryption (§7).
  • Namespace — a named partition of the cluster: the scope for names, RBAC grants, ResourceQuotas, LimitRanges, and NetworkPolicies. Multi-tenancy in most orgs is namespace-per-team/env with quotas and default-deny network policies — the same isolation discipline as Phase 13's multi-tenant platform, enforced at the infrastructure layer.

3. Scheduling: requests, limits, QoS, affinity, taints

The scheduler answers one question per pod: which node? It does a filter pass (remove nodes that can't take the pod) then a score pass (rank the survivors), and binds the pod to the winner. Everything you configure is input to those two passes.

Requests vs limits — the distinction that runs the whole cluster. resources.requests is what the scheduler reserves: it packs pods onto nodes by comparing the sum of requests against the node's allocatable capacity — never by live usage. A node can be 20% utilized and still refuse your pod because its requests are fully booked; conversely a node can be over 100% actual CPU while perfectly schedulable. resources.limits is what the runtime enforces: exceed a CPU limit and the kernel's CFS quota throttles you (latency spikes, no kill); exceed a memory limit and the kernel OOM-kills the container (restart, OOMKilled in the pod status). This asymmetry drives real sizing policy: always set requests (they're the capacity-planning signal), set memory limits (OOM is at least crisp), and be deliberate about CPU limits (many latency-sensitive teams omit them to avoid throttling).

QoS classes fall out of requests/limits mechanically: Guaranteed (every container has requests = limits for both CPU and memory), Burstable (some requests set, not Guaranteed), BestEffort (nothing set). Under node memory pressure the kubelet evicts in reverse order — BestEffort first, then Burstable pods exceeding their requests, Guaranteed last. Your inference server should be Guaranteed or high-request Burstable; your batch scraper should not be.

Placement controls, in increasing strength: nodeSelector (label equality), node affinity (expressive required/preferred rules), pod affinity/anti-affinity (place near / away from other pods — anti-affinity across zones is how you stop one rack from taking out all replicas), topology spread constraints (even spreading across zones/nodes), and taints & tolerations — the inverse of affinity: a taint on a node repels all pods that don't explicitly tolerate it. Taints are the canonical way to reserve a GPU pool: taint the GPU nodes, add tolerations only to inference pods, and your expensive silicon never fills up with cron jobs (§5, §15).

Lab 01's _schedule is the filter pass with deterministic first-fit scoring, packing against requests — and its Pending behavior is the real one: a pod that fits nowhere sits in Pending with a FailedScheduling event ("0/6 nodes are available: insufficient cpu"), which is the single most common "why isn't my pod running" incident in production.

4. Autoscaling: HPA, VPA, Cluster Autoscaler, KEDA

Four autoscalers, four different axes. Confusing them is a classic interview trap.

HPA — Horizontal Pod Autoscaler — scales replica count. The controller loop recomputes:

\[ \text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \frac{\text{currentMetricValue}}{\text{targetMetricValue}} \right\rceil \]

clamped to [minReplicas, maxReplicas], with a ~10% tolerance band so a metric wobbling around target doesn't flap the fleet, and a stabilization window (default 5 minutes on scale-down) so replicas don't thrash downward. CPU-utilization targets are relative to requests — an HPA on a pod with no CPU request cannot compute utilization at all. Metrics come from the Metrics Server (metrics.k8s.io) or, for custom/external metrics, an adapter (Prometheus Adapter, KEDA). Lab 01 implements the formula, clamp, and tolerance exactly.

VPA — Vertical Pod Autoscaler — scales the pod's own requests. It observes actual usage and recommends (or applies) better requests. Traditionally applying meant evicting and recreating the pod; newer Kubernetes adds in-place resize for some cases. Rule of thumb: run VPA in recommendation mode as a right-sizing advisor, and don't let VPA and HPA fight over the same metric (both reacting to CPU produces oscillation).

Cluster Autoscaler — scales the node count. Crucially, it does not watch utilization: it watches for Pending pods that failed scheduling, simulates whether adding a node from some node pool would let them schedule, and grows that pool; it scales down by finding underutilized nodes whose pods can be safely rehomed (respecting PodDisruptionBudgets), then draining them. HPA creates pods; when they go Pending, CA creates nodes — that chain is how a traffic spike turns into new hardware with no human involved.

KEDA — event-driven scaling, including to zero. HPA's standard metrics answer "how hot are the pods that exist," which fails for queue-driven and bursty AI workloads (a queue can be exploding while your one worker's CPU looks fine, and HPA can't scale from zero because zero pods emit no metrics). KEDA's ScaledObject reads the event source itself — queue depth, Kafka consumer lag, Postgres rows, cron — activates the workload from 0 → 1, and drives an HPA for 1 → N. For GPU inference workers fed by a request queue, "KEDA on queue depth + Cluster Autoscaler on the GPU pool" is the standard cost-efficient pattern (§15, §16).

5. Node pools and OKE specifics

A node pool is a group of identically-shaped nodes managed as a unit — same instance shape, same image, same labels/taints, its own autoscaling bounds. Real clusters are multi-pool by design: a general CPU pool for stateless services; a GPU pool (tainted, labeled, expensive) for inference; maybe a spot/preemptible pool for interruptible batch. Pools are also your upgrade mechanism: roll a new pool at the new Kubernetes version, cordon and drain the old one, delete it — infrastructure as cattle at the node-group level.

OKE (Oracle Kubernetes Engine) is OCI's managed Kubernetes — the control plane (API server, etcd, controllers) is Oracle-operated; you own the data plane through node pools. The pieces an OKE-focused SRE role expects you to know by name:

  • Cluster types: basic clusters (managed control plane) vs enhanced clusters (adds features like virtual nodes — a serverless data plane where OCI runs pods without you managing node VMs — plus stronger SLAs and add-on management).
  • Node pools on Compute shapes, including GPU shapes (NVIDIA A10/A100/H100, VM and bare metal) for inference pools; the NVIDIA device plugin advertises nvidia.com/gpu on them (§15).
  • Networking: pods attach to your VCN either via a flannel overlay or VCN-native pod networking (each pod gets a VCN IP — routable, security-listable, the option enterprises pick); a Service of type LoadBalancer provisions an OCI Load Balancer via annotations.
  • HA/DR topology: OCI regions are divided into availability domains (isolated datacenters) and fault domains (anti-affinity groups within an AD). Spreading a node pool across ADs/FDs plus pod topology spread is your zone-failure story; cross-region DR adds a standby cluster, replicated registry (OCIR), and restored state — measured by RPO/RTO (§14).
  • Ecosystem integration: OCIR (the container registry your pipeline pushes to), OCI Vault (external secret store, §7), IAM/workload identity for pod-level cloud permissions, and the OCI Terraform provider (oci_containerengine_cluster, oci_containerengine_node_pool) for provisioning it all as code (§11).

The portable skill is the shape, not the vendor: every managed Kubernetes (OKE/EKS/GKE/AKS) is "managed control plane + your node pools + cloud-integrated LB/IAM/registry/secret-store." Learn it once precisely and the others are renames.

6. Networking: Services, kube-proxy, CNI, NetworkPolicy, Ingress

Kubernetes networking rests on one radical simplification: every pod gets its own IP, and every pod can reach every other pod without NAT — a flat network. The CNI plugin (Calico, Cilium, flannel, or the cloud's VPC/VCN-native driver) is what makes that true on real infrastructure, wiring veth pairs, routes, and encapsulation or native routing.

Services and kube-proxy. A ClusterIP Service is a virtual IP — no interface anywhere owns it. kube-proxy on every node programs the kernel (iptables or IPVS) so that packets to the ClusterIP are DNAT'ed to one of the current ready pod IPs behind it (tracked via EndpointSlices). Only pods that pass their readiness probe are in that set — which is the hinge connecting networking to health (§8): a deploy is zero-downtime only because unready pods receive no traffic. Service types stack outward: ClusterIP (internal), NodePort (a port on every node), LoadBalancer (cloud LB in front — the OCI LB on OKE), plus headless Services (DNS straight to pod IPs, for stateful peers). CoreDNS gives every Service the svc.namespace.svc.cluster.local name.

NetworkPolicy is the pod-level firewall — and the default is allow-everything. A policy selects pods and whitelists ingress/egress by pod selector, namespace selector, or CIDR; policies are additive allows (there is no deny rule — isolation comes from being selected by any policy, after which only listed traffic passes). The production baseline is a default-deny policy per namespace plus explicit allows — the same default-deny egress discipline Phase 09 applied to a single sandboxed process, enforced here at the cluster level. Note the dependency: NetworkPolicy is data; the CNI must implement it (Calico/Cilium do; classic flannel alone does not).

Ingress (§2) handles L7: one entry point routing by host/path to Services, TLS termination, usually cert-manager for certificates. For gRPC/streaming inference endpoints, L7 idle timeouts and body-size limits on the ingress controller are recurring production foot-guns worth knowing about in advance.

7. RBAC, ServiceAccounts and secrets management

RBAC answers "who may do what to which API objects." Four objects: a Role (a list of allowed verbs on resources, within a namespace), a ClusterRole (same, cluster-wide), and RoleBinding/ClusterRoleBinding (attach a role to subjects — users, groups, or ServiceAccounts). Two properties matter more than the syntax: RBAC is additive-only (there is no deny; your effective permission is the union of your bindings, so least privilege means granting narrowly, not blocking broadly), and it protects the API server, not the network (a pod with no RBAC at all can still call another pod's HTTP endpoint — that's NetworkPolicy's job, §6). The classic audit findings are cluster-admin bound to humans for convenience, and workloads running under the default ServiceAccount with leftover permissions.

A ServiceAccount is a workload's identity: the pod gets a projected, expiring token, and RBAC bindings against that ServiceAccount define what the pod may do to the API. Cloud workload identity (OKE's IAM integration and its EKS/GKE analogues) extends the same idea to cloud APIs — the pod exchanges its ServiceAccount identity for cloud credentials, eliminating long-lived keys in the cluster.

Secrets management in layers, weakest to strongest: (1) Kubernetes Secrets are base64-encoded, not encrypted — anyone with get secret RBAC or raw etcd access reads them; (2) enable encryption at rest for etcd (a KMS-backed EncryptionConfiguration) so the etcd file isn't plaintext; (3) keep secrets out of Git — Sealed Secrets (encrypt into a Git-safe CRD a cluster controller can decrypt) or, more commonly, (4) an external store of record — Vault, OCI Vault, cloud secret managers — synced into the cluster by the External Secrets Operator or mounted via the Secrets Store CSI driver, giving you rotation, audit, and central revocation. The JD phrase "secrets management" means: RBAC-restrict get secret, encrypt etcd, externalize the store, rotate, and never bake credentials into images or Git.

8. Health and self-healing: probes, drain, PodDisruptionBudgets

Kubernetes can only heal what it can see, and probes are its eyes. Three probes, three distinct consequences — mixing them up causes real outages:

  • Liveness probe fails → the kubelet restarts the container (with exponential CrashLoopBackOff). Liveness answers "is this process wedged beyond recovery?" A liveness probe that checks a dependency (database, downstream API) is a classic self-inflicted outage: the dependency blips and Kubernetes helpfully restarts your entire healthy fleet.
  • Readiness probe fails → the pod is removed from Service endpoints (§6) — no restart, no traffic. Readiness answers "can this pod serve right now?" and is supposed to fail during startup, overload, or dependency loss. Rolling updates are only zero-downtime because new pods join the endpoint set when ready and old pods finish draining after SIGTERM + preStop.
  • Startup probe holds liveness/readiness off until the app finishes booting — essential for slow starters, and inference servers loading multi-GB model weights are the canonical case (§15): without it, liveness kills the pod mid-load, forever.

Voluntary vs involuntary disruption. A node crash is involuntary — the node controller marks the node NotReady, waits out grace periods, evicts, and the ReplicaSet loop (Lab 01's fail_node + reconcile) restores replicas elsewhere. A drain (kubectl drain — cordon, then evict every pod) is voluntary: upgrades, scale-down, patching. PodDisruptionBudgets gate voluntary evictions only: "at least minAvailable: 2 of these pods must remain" — a drain that would violate it blocks and retries. PDBs are what let the Cluster Autoscaler and node upgrades proceed safely at 3 a.m. without paging you; the deliberate rollout analogue is maxUnavailable in the Deployment strategy, whose "ready never drops below desired − maxUnavailable" invariant Lab 01 proves step by step.

9. CI/CD: stages, gates, scanning, signing, SBOM, promotion

Whatever the tool — Jenkins (pipeline-as-Groovy, self-hosted, infinitely pluggable), GitLab CI (.gitlab-ci.yml, tight repo integration), GitHub Actions (workflow YAML, marketplace ecosystem), Argo Workflows/CD (Kubernetes-native) — a production pipeline is the same gate machine: ordered stages, each a check that can BLOCK, and only artifacts that pass every gate move forward. Lab 02 builds it. The canonical stages:

  1. Build — produce a container image; the output identity is its content-addressed digest (sha256:… over the manifest). Digests, not tags: tags are mutable pointers, and mutable pointers are how staging and prod silently diverge.
  2. Static analysis — linters, type checkers, SAST (SonarQube, Semgrep); findings above a severity threshold fail the job. Same gate shape as everything else.
  3. Unit tests + coverage gate — all green AND coverage ≥ threshold. A coverage gate exists to stop erosion, not to worship a number.
  4. Image scanning — Trivy (or Grype/Snyk) matches the image's OS and language packages against vulnerability databases and reports findings by severity. The gate is policy: trivy image --severity HIGH,CRITICAL --exit-code 1 blocks the pipeline on any CRITICAL/HIGH. Accepted risks go in a .trivyignore-style waiver list — versioned, reviewed, auditable — never in a lowered global threshold. Vulnerability remediation as an SRE duty is the loop around this gate: triage findings, bump the base image or package, rebuild, rescan, redeploy; for running workloads, scheduled rescans of deployed images catch CVEs published after ship.
  5. Integration tests — against a real-ish environment (compose, ephemeral namespace).
  6. Sign + attest — cosign (Sigstore) signs the image digest, storing the signature in the registry; keyless mode ties it to CI's OIDC identity with the Rekor transparency log. Attach an SBOM (SPDX or CycloneDX, generated by syft or Trivy) as an attestation — the ingredient list that lets you answer "are we running anything with log4j?" in minutes.
  7. Staged promotion — the same digest moves dev → staging → prod, each environment requiring the previous one's gates and soak, prod additionally requiring manual approval (GitLab protected environments / when: manual, GitHub environment required-reviewers). At deploy time an admission controller (Kyverno verifyImages, sigstore policy-controller) verifies the signature — verify-before-deploy closes the loop: an image that didn't come out of your pipeline, or was tampered with after it, cannot run.
  8. Rollback — redeploy the previous known-good digest (or revert the GitOps commit, §10). Prepared before the incident, executed in one operation.

The one-sentence philosophy Lab 02's tests encode: promote artifacts, never rebuild; gates can only block, never be skipped silently; signing makes "same artifact" a check instead of a promise.

10. GitOps: Git as desired state

GitOps is §1's reconciliation loop lifted one level: Git holds the desired state of the whole system (manifests, Helm values, Kustomize overlays), and an in-cluster agent — Argo CD or Flux — continuously compares the live cluster against the repo and converges it.

Mechanically, Argo CD's unit is the Application: a source (repo, path, revision) and a destination (cluster, namespace). The controller renders the manifests, diffs them against live objects, and reports Synced or OutOfDrift — with automated sync + selfHeal, a manual kubectl edit on prod gets reverted automatically within minutes, and prune deletes live objects whose manifests were removed. This is drift reconciliation as a running control loop, and it changes the operational contract: the cluster is no longer the source of truth about itself; Git is.

Why teams adopt it, in order of real value: audit and rollback for free (every change is a commit; rollback is git revert); no CI system holds prod credentials (the pull model — the agent inside the cluster fetches from Git, instead of a build server pushing kubectl apply with cluster-admin); drift can't accumulate silently; and promotion becomes data — Lab 02's env ladder reappears as directories/overlays per environment, where "promote to prod" is a PR bumping the image digest in envs/prod, reviewed and merged like any code. Argo CD and Flux differ in ergonomics (Argo has the UI and app-of-apps pattern; Flux is toolkit-style controllers) — the model is identical.

11. Infrastructure as Code: Terraform and Helm

Two tools, two layers, one principle: infrastructure is declared in versioned text, and applying it is reproducible.

Terraform provisions the platform — the cluster itself and everything under it (VCN, subnets, node pools, registries, IAM). Core mechanics you must be able to explain: providers translate HCL resources into cloud API calls (the OCI provider's oci_containerengine_cluster / oci_containerengine_node_pool provision OKE, §5); state is Terraform's record of what it believes exists, mapping resource addresses to real IDs — kept in a remote backend (object storage) with locking so two applies can't interleave; plan → apply is the two-phase contract — terraform plan diffs desired (code) against recorded (state) and refreshed (real) infrastructure and prints exactly what would change; apply executes that plan. Drift — someone hand-edits the console — shows up at the next plan as an unexpected diff; the discipline is the GitOps one: fix the code, or revert the hand edit, never let them diverge quietly. Modules give you reuse (one node-pool module, N pools). Terraform is deliberately not a great app-deployment tool — reconciliation runs only when a human runs it.

Helm deploys the applications — it is Kubernetes' package manager. A chart is a directory of Go-templated manifests plus a values.yaml of defaults; helm install renders templates with your value overrides and applies them as a named, versioned release; helm upgrade renders and diffs the new version in; helm rollback re-applies a previous revision (release history is stored in-cluster as Secrets) — noting it restores manifests, not your data. Charts are how you install the ecosystem (Prometheus stack, ingress-nginx, cert-manager, KEDA) and how you template your own service across environments with per-env values files. In a GitOps shop the division of labor is: Terraform makes the cluster exist; Argo CD + Helm charts make the right software run on it; nobody runs kubectl apply by hand.

12. Pod security and admission control

Phase 09 built what a container is — namespaces for view, cgroups for consumption, seccomp/capabilities for the syscall surface. This section is how a cluster forces every pod to use those protections.

Pod Security Standards (PSS) define three profiles: privileged (anything goes — node agents only), baseline (blocks the known dangerous knobs: privileged: true, hostNetwork, hostPath, added capabilities), and restricted (the hardening target: run as non-root, drop ALL capabilities, seccomp RuntimeDefault, no privilege escalation, read-only root filesystem encouraged). They're enforced by Pod Security Admission, the built-in admission controller configured per namespace via labels — with three modes you can mix: enforce (reject), audit (log), warn (tell the client). The standard migration is warn/audit at restricted first, fix the violations, then flip enforce. (The old PodSecurityPolicy object was removed in Kubernetes 1.25; saying "PSP" in an interview dates your knowledge.)

The general mechanism underneath: admission control — after authentication and RBAC authorization, every API write passes through mutating then validating admission webhooks. That is the extension point where policy engines live: Kyverno (policies as Kubernetes resources — require signed images §9, require resource limits, block :latest tags) and OPA Gatekeeper (Rego constraints), plus the built-in CEL ValidatingAdmissionPolicy. Admission is also where the supply chain meets the runtime: the cosign verify-before-deploy gate is a validating webhook rejecting unsigned digests. The principle to say out loud: RBAC decides who may write objects; admission decides what objects are acceptable — pod security is an admission concern, not an RBAC one.

13. Observability: Prometheus, PromQL, Grafana, logs, tracing

Three legs — metrics, logs, traces — answering three questions: what is happening in aggregate, what exactly happened here, where in the call chain did it happen. Phase 14 built these for the agent runtime (token costs, reasoning traces); this section is the infrastructure layer beneath it, and Lab 03 builds its mechanisms.

Prometheus is pull-based: it discovers targets via the Kubernetes API and scrapes each pod's /metrics endpoint on an interval into a local time-series database. Every series is identified by a metric name plus label set — which gives you sum by (service) power and also the #1 operational hazard: a label with unbounded values (user ID, request ID) multiplies series until the TSDB falls over (cardinality explosion). The metric types that matter: counter (monotonic cumulative — designed so a missed scrape loses resolution, not truth), gauge (point-in-time value), histogram (observations in le-bucketed counters plus _sum/_count — aggregatable across pods, which is why it beats the summary type in practice). The PromQL core, all built in Lab 03: rate(x[5m]) (per-second counter increase over a window — raw counters are meaningless unrated), histogram_quantile(0.99, …) (walk cumulative buckets, linearly interpolate — your p99 is an estimate whose accuracy you chose when you chose buckets), and sum by (label)(…). Recording rules precompute expensive expressions into new series; alerting rules evaluate an expression with a for: duration — breach → pending → (held for the duration) → firing → resolved — and firing alerts go to Alertmanager, which groups, silences, inhibits, and routes them to PagerDuty/Slack. The for: debounce is the difference between an alert and a nuisance; Lab 03 proves a transient spike never fires.

Grafana is the query-and-render layer over it. Two dashboard disciplines beat a hundred random panels: RED for every service — Rate, Errors, Duration (p50/p95/p99 from histograms) — and USE for every resource — Utilization, Saturation, Errors (node CPU, memory pressure, GPU utilization via DCGM). If a dashboard can't answer "is it broken, for whom, how badly," it's decoration.

Logs: containers write to stdout/stderr; the kubelet keeps only a small rotating window (kubectl logs dies with the pod). Production runs a DaemonSet shipper (Fluent Bit, Promtail) forwarding every container's stream — enriched with pod/namespace labels — to a store: Loki (indexes labels only, cheap, grep-like queries) or Elasticsearch/OpenSearch (full-text, heavier). Emit structured JSON with request IDs so logs join to traces.

Distributed tracing: instrument services with OpenTelemetry, propagate context via the W3C traceparent header, sample, and send spans through the OTel Collector to Jaeger/Tempo. Tracing is the only leg that answers "where in the fan-out did the latency go" — for an agent platform whose single request may touch a gateway, a retriever, and three model calls, it's the difference between knowing p99 regressed and knowing which hop regressed.

14. SRE: SLOs, error budgets, on-call, postmortems, DR

Site Reliability Engineering's core move is turning reliability from an argument into arithmetic. Lab 03 implements every formula here.

SLI → SLO → error budget. An SLI is a measured ratio: good events / valid events (requests under 300 ms and non-5xx / all requests). An SLO is a target for that SLI over a window: 99.9% over 30 days. (An SLA is the SLO's legal cousin — a contract with penalties; never confuse them in an interview.) The error budget is what the SLO does not promise: \(1 - 0.999 = 0.1\%\) — about 43 minutes of full downtime per 30 days. The budget is a spendable resource: plenty left → ship faster, take risks; exhausted → freeze features and fix reliability. That policy — agreed before the incident — is what makes SLOs a management tool and not a dashboard ornament.

Burn rate is how fast you're spending it: actual error rate ÷ allowed error rate. Burn 1.0 spends the budget exactly by window's end; burn 14.4 exhausts a 30-day budget in ~2 days. Naive threshold alerts are either too twitchy or too slow; the SRE Workbook's answer is multi-window, multi-burn-rate alerts: page only when the burn rate exceeds the factor over a long window (evidence it's real) and a short window (evidence it's still happening). Canonical 30-day/99.9% setup: page at 14.4× over 1 h AND 5 m; page at 6× over 6 h AND 30 m; ticket at 1× over 3 days. Lab 03's fast_burn_alert implements exactly this AND-of-two-windows, and its tests prove the quiet case: incident over → short window clean → no page, even while the long window still remembers.

On-call and incident response: a rotation with primary/secondary, paged only by budget-threatening alerts (everything else is a ticket); every page maps to a runbook; incidents get roles (incident commander coordinating, others operating, someone communicating) and severity levels with defined response expectations. After: the blameless postmortem — timeline, contributing causes framed as systems ("the deploy pipeline allowed an unsigned image") not people ("Alice pushed a bad image"), and tracked action items. Blameless is not a courtesy; it's an incentive design — punish honesty and your next postmortem will be fiction.

Disaster recovery is measured by two numbers agreed in advance: RPO (Recovery Point Objective — how much data you may lose; your backup/replication interval bounds it) and RTO (Recovery Time Objective — how long restoration may take; your automation bounds it). For a cluster: etcd/state backups (managed by the provider on OKE), Velero for workload/PV backup, manifests already in Git (GitOps shrinks cluster-rebuild RTO to "recreate with Terraform, point Argo CD at the repo"), registry replication, and node pools spread across availability/fault domains (§5) so a datacenter failure is degradation, not disaster. A DR plan you haven't exercised is a hypothesis — game days exist because restores fail in ways backups don't reveal.

15. Serving AI models on Kubernetes

Everything above, specialized for the workload this track cares about: model inference.

GPUs are extended resources. The NVIDIA device plugin (a DaemonSet) advertises nvidia.com/gpu on GPU nodes; pods request whole GPUs in resources.limits. GPUs are not natively divisible like CPU — three sharing mechanisms exist when whole-GPU-per-pod wastes money: time-slicing (the plugin advertises N virtual slots per GPU; no memory isolation — one tenant's OOM can kill another's process; fine for dev, risky for prod), MIG (A100/H100-class hardware partitioning into isolated instances with dedicated memory — the real multi-tenant answer), and MPS (concurrent CUDA contexts, partial isolation). Pool design follows §3/§5: taint the GPU pool, tolerate only inference pods, and let the Cluster Autoscaler grow it — because idle H100s are the most expensive idle anything in your fleet.

Model servers and serving platforms. The servers: Triton, vLLM, TGI — processes that load weights and expose HTTP/gRPC inference. The platforms above them: KServe (the InferenceService CRD: point it at a model URI and it manages the server Deployment, autoscaling — optionally scale-to-zero via Knative — and canary rollout by traffic percentage) and Seldon Core (inference graphs: transformers, ensembles, A/B routers as composable steps). What the platforms actually buy you is rollout semantics for models: a new model version is a new artifact whose quality is a distribution, not a boolean, so you shift 5% of traffic, compare quality/latency metrics against the stable version, then step up — a canary. The alternatives: blue/green (two full stacks, instant switch, instant rollback, double cost) and shadow (mirror traffic to the candidate, compare offline, user-invisible — ideal for models because quality regressions don't 500; they answer plausibly and wrong, which only evaluation catches — Phase 11's harness is the judge you'd wire in).

The operational quirks of model workloads, versus stateless web services: startup is heavy (pulling a multi-GB image plus tens of GB of weights takes minutes — use startup probes §8, init containers or a model-cache PVC, and pre-pulled images); readiness must mean "model loaded and warm," not "HTTP port open," or your canary receives traffic it answers with 503s; autoscaling signals differ (GPU-bound servers saturate on queue depth and token throughput long before CPU — KEDA on queue depth §4, or custom metrics like batch queue time); and rollback must include the model, so version model artifacts with the same digest-and-promote discipline as images (Lab 02's ladder applies unchanged).

16. Cost and capacity tuning

Cost on Kubernetes is mostly three ratios you can move:

  1. Requests : actual usage (right-sizing). Requests are the currency of scheduling (§3), so over-requesting strands capacity invisibly: a pod requesting 4 CPU and using 0.5 wastes 3.5 CPU of every node it lands on. Fix with usage data — VPA in recommendation mode, or p95 usage from Prometheus — and shrink requests toward reality with headroom.
  2. Allocated : provisioned (bin-packing density). Even right-sized pods can be spread thinly across half-empty nodes. Cluster Autoscaler scale-down, consolidation-aware autoscalers, and fewer/larger nodes raise density — bounded by blast radius (one huge node failing takes more with it) and PDB-safe drainability.
  3. Provisioned : needed-at-peak (elasticity). Autoscale (HPA/KEDA/CA) so provisioned tracks demand instead of being sized for peak all day; run interruptible work (batch inference, evals, training-lite) on spot/preemptible pools at steep discounts with graceful-eviction handling; scale bursty inference to zero with KEDA/Knative when idle.

The governance layer: ResourceQuota per namespace (caps total requests/limits/objects — how platform teams stop one team from eating the cluster) and LimitRange (per-pod defaults and bounds so "I forgot requests" is impossible). Attribution: Kubecost/OpenCost-style tooling maps node cost → pod requests → team labels, which is what makes ratio 1 visible per team. For GPU fleets, all of this applies with higher stakes plus one extra metric: DCGM GPU utilization — a 30%-utilized H100 pool is the single best cost-reduction target in an AI platform, attacked with batching (continuous batching in vLLM), MIG partitioning (§15), and queue-driven scaling. Capacity planning closes the loop: forecast peak QPS × per-request resource cost (from load tests) + failure headroom (N+1 across ADs — survive one domain down at peak) + growth, revisited quarterly against actuals.

17. Common misconceptions

  • "Kubernetes schedules pods onto the least-loaded node by live usage." It schedules by requests vs allocatable, not live usage. A cluster can be simultaneously "empty" (low CPU use) and "full" (requests booked). This one misconception explains half of all Pending-pod confusion.
  • "A failing readiness probe restarts the pod." No — readiness only removes the pod from Service endpoints. Liveness failures restart. Wiring a dependency check into liveness turns a dependency blip into a fleet-wide restart storm.
  • "CPU limits protect my service." CPU limits throttle (CFS quota) and are a common cause of tail latency. Memory limits OOM-kill. Requests always; memory limits yes; CPU limits only deliberately.
  • "Kubernetes Secrets are encrypted." Base64-encoded by default. Encryption at rest is an opt-in etcd configuration; real secret hygiene externalizes the store (§7).
  • "NetworkPolicy is default-deny." The default is allow-everything; isolation begins only when a policy selects the pod — and only if the CNI enforces policies at all.
  • "The Cluster Autoscaler scales on CPU utilization." It scales on unschedulable (Pending) pods and simulated placements — utilization is HPA/VPA's department.
  • "HPA works out of the box on any pod." CPU-utilization targets are computed relative to requests — no CPU request, no utilization math.
  • "A rolling update is automatically zero-downtime." Only with honest readiness probes and graceful shutdown (SIGTERM handling, preStop); otherwise you get maxSurge'd pods serving errors and draining pods dropping connections.
  • "GitOps = CI pushing kubectl apply." GitOps is a pull-based reconciliation loop with drift detection and self-heal; the push model is exactly what it replaces (and the reason no CI runner needs prod credentials).
  • "terraform plan was clean, so apply is safe." Plan is a point-in-time diff; state drift, concurrent changes, and eventually-consistent cloud APIs mean apply can still surprise — locking and short plan-to-apply windows exist for a reason.
  • "The p99 on the dashboard is the real p99." From a histogram it's a bucket-interpolated estimate (Lab 03) — its accuracy was fixed the day someone chose the bucket boundaries.
  • "We target 100% availability." Then your error budget is zero and no release, node drain, or dependency blip is ever acceptable — 100% is not an SLO, it's a refusal to do the math.
  • "Pod Security Standards will fix insecure images." PSS constrains pod spec (privilege, host access, capabilities). Vulnerable contents are the scanner's job (§9); the two gates compose, neither substitutes.

18. Lab walkthrough

Build the three miniatures in order; each isolates one third of the SRE job and injects every effectful dependency (metrics, scan reports, test results, clocks-as-ticks) so everything is deterministic and offline.

  1. Lab 01 — Reconciliation Orchestrator. The Kubernetes control loop: Node/Pod/Deployment objects, a deterministic first-fit scheduler that packs against requests (Pending when nothing fits), an idempotent reconcile() that converges actual to desired, self-healing after delete_pod and fail_node, the real HPA formula with clamp and tolerance, and a maxSurge/maxUnavailable rolling update whose availability invariants the tests check at every step — plus rollback. 32 tests.
  2. Lab 02 — CI/CD Pipeline & Gates. The gate machine: build (content-addressed digest + SBOM) → unit-test (coverage gate) → image-scan (Trivy-style CRITICAL/HIGH severity gate with waiver list) → integration-test → sign (deterministic signature over the digest), fail-fast; then dev → staging → prod promotion with verify-before-deploy, prod manual approval, and rollback to last-good. 29 tests.
  3. Lab 03 — Observability, SLOs & Alerting. The telemetry engine: a label-keyed counter/gauge/histogram store, PromQL-lite (rate, histogram_quantile with bucket interpolation, sum by), the pending → firing → resolved alert state machine with for: debounce, and SLO arithmetic — SLI, error budget, burn rate, multi-window fast-burn paging. 32 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

19. Success criteria

  • You can explain level-triggered reconciliation and why idempotency makes it crash-proof.
  • You can list what the scheduler packs against (requests) and diagnose a Pending pod in three questions.
  • You can state the three probe types and the different consequence of each failing.
  • You can write the HPA formula from memory and name what KEDA adds that HPA can't do.
  • You can describe the OKE-specific pieces: node pools, GPU shapes, VCN-native networking, ADs/FDs, OCIR, OCI Vault.
  • You can walk a commit through every gate to a signed digest running in prod, and roll it back two ways (pipeline redeploy, GitOps revert).
  • You can explain what Argo CD does when someone hand-edits prod.
  • You can define SLI/SLO/error budget/burn rate and derive the 14.4× fast-burn factor's meaning.
  • You can name the three GPU-sharing mechanisms and when each is safe.
  • All three labs pass under both lab and solution (93 tests total).

20. Interview Q&A

Q: Walk me through what happens when you kubectl apply a Deployment. A: kubectl sends the manifest to the API server, which authenticates, authorizes (RBAC), runs admission (mutating then validating webhooks — defaults injected, policy enforced), and persists the object to etcd. Then the controllers take over, each a level-triggered loop: the Deployment controller creates a ReplicaSet for the pod template; the ReplicaSet controller creates Pod objects to match the replica count; the scheduler binds each Pending pod to a node whose free requests capacity and constraints (affinity, taints) fit; the kubelet on that node sees the binding and starts the containers, running probes; once ready, the pod's IP joins the Service's endpoints. Nothing in that chain is a direct command — every step is a controller converging observed state to desired state, which is why it survives restarts and missed events.

Q: Liveness vs readiness vs startup probes? A: Liveness failure restarts the container — it means "wedged beyond recovery." Readiness failure removes the pod from Service endpoints, no restart — it means "can't serve right now" and is expected during startup and overload. Startup probes suspend the other two until the app boots — essential for slow starters like inference servers loading model weights. Two classic mistakes: dependency checks in liveness (a database blip restarts your whole healthy fleet) and no startup probe on a slow loader (liveness kills it mid-load forever).

Q: What's the difference between requests and limits, and what actually happens at the limit? A: Requests are the scheduler's currency — pods are packed against the sum of requests vs node allocatable, never live usage — and they define QoS class. Limits are runtime enforcement, and the two resources behave differently: exceeding a CPU limit throttles via the kernel CFS quota (latency, no kill); exceeding a memory limit OOM-kills the container. That asymmetry is why the common production stance is: always set requests, set memory limits, and treat CPU limits with suspicion for latency-sensitive services. Under node memory pressure, eviction order follows QoS: BestEffort, then Burstable over its requests, then Guaranteed.

Q: How does the HPA decide the replica count, and how do you stop it flapping? A: desired = ceil(current × currentMetric/targetMetric), clamped to min/max. Anti-flap comes from three mechanisms: the ~10% tolerance band (no action when the ratio is near 1), the scale-down stabilization window (default 5 minutes — it scales down to the maximum recommendation over the window), and sane target values. For CPU utilization the metric is relative to requests, so an HPA without CPU requests can't function. And HPA can't scale from zero — no pods, no metrics — which is the gap KEDA fills by reading the event source (queue depth, consumer lag) directly.

Q: HPA vs VPA vs Cluster Autoscaler vs KEDA — one sentence each. A: HPA changes replica count from pod metrics; VPA changes a pod's own requests (best run as a right-sizing recommender); Cluster Autoscaler changes node count, triggered by Pending pods and simulated scheduling — not utilization; KEDA scales from external event sources, including zero→one, and drives an HPA for one→N. They compose: KEDA/HPA make pods, pods go Pending, CA makes nodes.

Q: A pod is stuck Pending. Debug it. A: kubectl describe pod and read the events — FailedScheduling tells you which filter failed on how many nodes. Three families: resources (requests don't fit any node's free allocatable — check requests math, check whether the Cluster Autoscaler can/should add a node, check quota), constraints (nodeSelector/affinity matches no node, a taint isn't tolerated — the GPU-pool case, topology spread unsatisfiable), and non-scheduler causes (volume can't bind or attach in the right zone). The fix is almost always arithmetic or labels, not restarts.

Q: Design a CI/CD pipeline for a service that ships to Kubernetes. A: Stages as gates, fail-fast: build a content-addressed image; static analysis; unit tests with a coverage threshold; Trivy scan blocking on CRITICAL/HIGH with a versioned waiver list for accepted risks; integration tests; then sign the digest with cosign and attach an SBOM attestation. Promotion moves the same digest dev → staging → prod — never rebuild — each env gated on the previous, prod behind manual approval, and the cluster's admission controller verifies the signature before anything runs. Rollback is redeploying the previous known-good digest, or in a GitOps setup, reverting the commit that bumped it. The principles: gates block and can't be skipped silently; artifacts are immutable and promoted; signing turns "same artifact" into a verifiable check.

Q: What is GitOps, actually — beyond "manifests in Git"? A: A pull-based reconciliation loop: an in-cluster agent (Argo CD, Flux) continuously diffs live cluster state against the rendered manifests at a Git revision and converges — drift from a manual kubectl edit is detected and, with self-heal on, reverted automatically. Git becomes the source of truth, so every change is a reviewed commit, rollback is git revert, cluster rebuild is "point the agent at the repo," and no CI system holds production credentials. That last property — push CD gives a build server cluster-admin; pull CD doesn't — is the security argument that usually wins the adoption debate.

Q: Your SLO is 99.9% over 30 days. Design the alerting. A: Error budget is 0.1% — roughly 43 minutes of downtime-equivalent per 30 days. Alert on burn rate — actual error rate divided by allowed — with multi-window rules: page when burn exceeds 14.4× over both 1 hour and 5 minutes (that rate spends ~2% of the monthly budget in an hour, and the 5-minute window guarantees it's still happening — no pages for incidents that already ended); page on 6× over 6 h + 30 m; open a ticket at 1× over 3 days for slow leaks. Every paging rule carries a for: duration so scrape blips never page. This is exactly the SRE Workbook design, and it's what Lab 03's fast_burn_alert implements.

Q: How would you run GPU inference on Kubernetes cost-efficiently? A: A dedicated, tainted GPU node pool (only inference pods tolerate it) sized by the Cluster Autoscaler; nvidia.com/gpu requests via the device plugin; MIG partitioning where models don't need a full GPU (time-slicing only for non-prod — no memory isolation); scaling driven by the real bottleneck signal — queue depth or token throughput via KEDA, not CPU; scale-to-zero for bursty models; startup probes and a model cache so cold starts don't fight liveness; canary or shadow rollout for new model versions with quality metrics, not just 5xx, deciding promotion; and DCGM GPU utilization on the dashboard, because a 30%-utilized GPU pool is the biggest line item you can fix.

Q: RTO vs RPO — and what's your Kubernetes DR story? A: RPO is how much data you may lose (bounded by backup/replication frequency); RTO is how long recovery may take (bounded by automation). For the cluster: control-plane/etcd backups (provider-managed on OKE), Velero for workload and volume backup, everything declarative in Git so rebuild is Terraform + pointing Argo CD at the repo, registry replication, and node pools spread across availability and fault domains so an AD failure degrades rather than destroys. And a DR plan is a hypothesis until you've run the restore in a game day — untested backups fail at the worst possible moment.

Q: What do Pod Security Standards actually enforce, and what don't they? A: They constrain the pod spec at admission time — three profiles (privileged/baseline/restricted), applied per namespace by Pod Security Admission in enforce/audit/warn modes. Restricted means non-root, capabilities dropped, seccomp on, no privilege escalation — forcing workloads onto the namespace/cgroup/seccomp primitives from Phase 09. They do not inspect image contents: a fully restricted pod can still run a CVE-riddled image, which is the scanner gate's job in CI, plus rescans of deployed images. Spec security and supply-chain security are separate gates that compose.

21. References

  • Kubernetes documentation — Concepts (controllers and the reconciliation model, Workloads, Services/Ingress, Configuration, Security). https://kubernetes.io/docs/concepts/
  • Kubernetes documentation — Horizontal Pod Autoscaling (the algorithm, tolerance, stabilization). https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
  • Kubernetes documentation — Pod Security Standards and Pod Security Admission. https://kubernetes.io/docs/concepts/security/pod-security-standards/
  • Kubernetes documentation — Network Policies. https://kubernetes.io/docs/concepts/services-networking/network-policies/
  • Oracle Cloud Infrastructure documentation — Container Engine for Kubernetes (OKE): clusters, node pools, GPU shapes, VCN-native pod networking. https://docs.oracle.com/en-us/iaas/Content/ContEng/home.htm
  • Cluster Autoscaler — Kubernetes Autoscaler project documentation (FAQ explains the Pending-pods trigger and scale-down simulation). https://github.com/kubernetes/autoscaler
  • KEDA documentation — ScaledObjects, scalers, scale-to-zero. https://keda.sh/docs/
  • Trivy documentation — vulnerability scanning, severity filtering, ignore files. https://trivy.dev/
  • Sigstore cosign documentation — container signing, verification, attestations (SBOM). https://docs.sigstore.dev/
  • Argo CD documentation — Applications, sync, automated self-heal and pruning (drift reconciliation). https://argo-cd.readthedocs.io/
  • Flux documentation — the GitOps toolkit controllers. https://fluxcd.io/flux/
  • Terraform documentation — state, backends, plan/apply; and the OCI provider. https://developer.hashicorp.com/terraform/docs and https://registry.terraform.io/providers/oracle/oci/latest/docs
  • Helm documentation — charts, templates, releases, rollback. https://helm.sh/docs/
  • Prometheus documentation — metric types, PromQL basics, rate(), histogram_quantile(), recording and alerting rules. https://prometheus.io/docs/
  • Grafana documentation — dashboards and Prometheus data source. https://grafana.com/docs/
  • Google SRE Book (Site Reliability Engineering) — SLOs, error budgets, being on-call, postmortem culture. https://sre.google/sre-book/table-of-contents/
  • Google SRE Workbook — Chapter 5, "Alerting on SLOs" (the multi-window multi-burn-rate method). https://sre.google/workbook/alerting-on-slos/
  • The RED Method (Tom Wilkie) and the USE Method (Brendan Gregg) — service and resource dashboard disciplines. https://www.brendangregg.com/usemethod.html
  • KServe documentation — InferenceService, canary rollout, serverless inference. https://kserve.github.io/website/
  • NVIDIA device plugin / GPU Operator documentation — GPU scheduling, time-slicing, MIG. https://docs.nvidia.com/datacenter/cloud-native/
  • OpenTelemetry documentation — traces, context propagation, the Collector. https://opentelemetry.io/docs/