« Phase 28 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 28 — Staff Engineer Notes: Owning Reliability on Kubernetes

Anyone can kubectl apply a Deployment. The gap between using Kubernetes and being trusted to own an AI platform's uptime is judgment about things nobody hands you a default for: what your SLO actually is and what you will spend its budget on, which autoscaler answers which signal, when a Pending pod is a bug versus a capacity purchase order, and — the one that gets escalated to you — how to design an alert that pages for real incidents and stays silent for everything else. This doc is about that judgment and the signal that proves you have it.

The decisions a staff engineer actually owns here

The SLO and its budget policy are yours, agreed before the incident. The junior instinct is "target 100%." A 100% SLO means a zero error budget — no release, node drain, or dependency blip is ever acceptable — which is not a target, it is a refusal to do the math. You own picking the SLI (good events / valid events: requests under 300 ms and non-5xx), the SLO (99.9% over 30 days → ~43 minutes budget), and the policy: budget left → ship faster; budget exhausted → freeze features and fix reliability. That policy is what turns an SLO from a dashboard ornament into a management tool.

Capacity is a portfolio of pools and autoscalers, not one setting. You segment traffic by shape: a CPU pool for the stateless API tier on HPA by request rate; a tainted GPU pool for inference scaled by KEDA on queue depth (because a GPU server saturates on queue depth and token throughput long before CPU); the Cluster Autoscaler growing the GPU pool when pods go Pending; VPA in recommendation mode as a right-sizing advisor. You own the requests sizing (the currency of scheduling), the pool min/max, and the PDBs that let the CA and node upgrades proceed at 3 a.m. without paging you.

Requests/limits sizing is a capacity-planning act. Always set requests (they are the scheduler's input and the capacity signal); set memory limits (OOM is crisp); be deliberate about CPU limits (they throttle and cause tail latency — many latency-sensitive teams omit them). Getting this wrong is invisible: a pod requesting 4× what it uses strands capacity on every node it touches.

The supply chain's gate policy is yours. Which coverage threshold; which scan severities block (CRITICAL/HIGH) and what goes in the versioned waiver list versus a lowered global threshold (never the latter); whether prod requires human approval; whether admission verifies signatures before deploy. These are the company's risk posture expressed as pipeline config, and you make the safe path the easy one.

The decision framework, out loud

  • Interactive traffic needing p99 guarantees → sized requests, HPA/KEDA headroom, and readiness that means "warm," not "port open."
  • Bursty, queue-driven, or scale-to-zero AI workloads → KEDA on the event source; HPA alone cannot scale from zero.
  • New model version, quality is a distribution not a boolean → canary (traffic %) or shadow (mirror, compare offline) with an eval harness judging quality, not just 5xx; blue/green only when instant rollback justifies double cost.
  • A pod is Pending → it is arithmetic or labels: requests don't fit (shrink or let the CA add a node), or a constraint/taint isn't satisfied (the GPU-pool case). Almost never "restart it."
  • Change management on prod → GitOps pull (Argo CD/Flux), so rollback is git revert, cluster rebuild is "point the agent at the repo," and no CI runner holds prod credentials.

Code-review red flags

  • A dependency check inside a liveness probe. A database blip will restart the entire healthy fleet. Dependency checks belong in readiness.
  • A slow-loading pod (multi-GB weights) with no startup probe. Liveness kills it mid-load, forever. Startup probes exist exactly for this.
  • Promotion of a mutable tag (:latest) instead of a digest. Staging and prod silently diverge. Promote the digest; verify the signature at admission.
  • Rebuilding the image for prod. Prod then runs bytes that never passed staging's gates. Promote the artifact, never rebuild.
  • A single-threshold burn alert, or an alert with no for:. The first is too twitchy or too slow; the second pages on scrape blips. Multi-window burn rate with a for: debounce.
  • cluster-admin bound to a human, or a workload on the default ServiceAccount. Least privilege means granting narrowly; RBAC is additive-only, so there is no "deny" to save you.
  • A NetworkPolicy assumed to be default-deny, on a CNI that doesn't enforce policy. Default is allow-everything, and flannel alone ignores policy. Verify both.
  • CPU limits added "to be safe" on a latency-sensitive service — they throttle and cause tail latency. README.md chapter links and un-pinned image tags are the small tells.

War stories worth carrying

  • The liveness restart storm. A liveness probe checked a downstream API. The API had a 30-second blip; Kubernetes dutifully restarted every replica, turning a minor dependency hiccup into a fleet-wide outage. Readiness would have removed pods from rotation and let them recover. Liveness answers "wedged beyond recovery," nothing more.
  • The canary that answered wrong, not with errors. A new model version shipped to 5% of traffic and its 5xx rate was clean — so it was promoted. It was also confidently hallucinating, which does not raise a 5xx; only the eval harness caught it, after full rollout. Model rollouts need quality gates, not just error gates.
  • The throttle "outage" that was a purchase order. Two weeks of sustained GPU-pool Pending pods, treated as an incident to code around, was a capacity-threshold signal. The fix was raising the node-pool max and the KEDA bounds, not a retry loop.
  • The p99 that lied. A latency dashboard's p99 looked healthy through an incident because the histogram's largest bucket was +Inf and everything slow fell into it — the estimate's accuracy was fixed the day someone chose the buckets. Bucket boundaries are a design decision.

The interview / architecture-review signal

What a reviewer listens for: do you keep the loops, the gates, and the budget straight, and can you turn reliability into arithmetic? Can you explain level-triggered reconciliation and why idempotency makes it crash-proof in under a minute? Name the four autoscalers and what each scales? Diagnose a Pending pod in three questions (requests, constraints, volume)? Walk a commit through every gate to a signed digest running in prod and roll it back two ways? Derive the 14.4× fast-burn factor and explain why one window is never enough? Candidates who can run kubectl are common; the person who explains why the scheduler packs requests not usage, why a blocked input gate means the model never runs the bytes, and why a good page needs two windows is who gets handed "own the platform's uptime."

Closing takeaways

  1. Kubernetes is one idea — declared desired state plus controllers that converge actual to it. Learn the loop once and Deployments, HPAs, GitOps, and operators are corollaries.
  2. The scheduler packs requests, not usage — so sizing is capacity planning and Pending is almost always arithmetic, not a restart.
  3. A pipeline is only as strong as its gates, and a gate you can silently skip is not a gate. Promote signed artifacts, verify before deploy, never rebuild for prod.
  4. SLOs turn reliability into a budget you spend on velocity or bank for stability — and multi-window burn-rate paging is the only alerting that is both fast and quiet.
  5. You don't keep services up by watching them. You build control loops, gates, and budgets so the platform keeps itself up and pages you only when the math says so — that is the staff signal.