« Phase 28 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 28 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the mechanism; this is the stuff you say in the incident channel.

30-second mental model

Kubernetes is one idea worn four ways: a store of desired state plus controllers that drive actual state toward it (self-heal, scale, roll — all the same loop; Lab 01). A CI/CD pipeline is a gate machine: build → test → scan → sign, then promote the same signed digest dev → staging → prod (never rebuild), prod behind a human, rollback to last-good ready before you need it (Lab 02). Observability is three legs — metrics/logs/traces — and SRE turns reliability into arithmetic: SLI measures, SLO promises, error budget prices the gap, burn rate decides when to page (Lab 03). GitOps is the reconciliation loop again, with Git as desired state. Learn the loop, the gate, and the budget and the whole role is corollaries.

The things to tattoo on your arm

The distinctions that signal seniority

• Requests vs usage → the scheduler packs against requests; a cluster can be "empty" (low usage) and "full" (requests booked) at once. Half of all Pending-pod confusion dies here.
• Liveness vs readiness → liveness restarts, readiness de-endpoints. Dependency check in liveness = one blip restarts your whole fleet. Never mix them.
• CPU limit vs memory limit → CPU throttles (latency), memory OOM-kills. That asymmetry is why "always requests, memory limits yes, CPU limits carefully" is the real policy.
• CA vs HPA trigger → Cluster Autoscaler reacts to Pending pods, not utilization; and HPA can't scale from zero (no pods → no metrics), which is the gap KEDA fills by reading the event source. Queue-driven GPU workers need KEDA.
• Push CD vs GitOps → CI running kubectl apply hands a build server cluster-admin; GitOps pulls, so no external system holds prod creds, and drift self-heals.
• SLO vs SLA → SLO is your internal target; SLA is the contract with penalties. Never conflate them, and never target 100% (that's an error budget of zero — a refusal to ship). And a histogram p99 is a bucket-interpolated estimate, decided the day someone picked the buckets.

The CLI one-liners to know cold

# reconciliation & debugging
kubectl get deploy,rs,pods -o wide            # the whole chain at a glance
kubectl describe pod <p>                       # READ THE EVENTS — FailedScheduling lives here
kubectl rollout status deploy/<d>              # is the roll converged?
kubectl rollout undo deploy/<d>                # rollback = re-scale a previous ReplicaSet
kubectl top pod / kubectl top node             # actual usage (needs Metrics Server)
kubectl drain <node> --ignore-daemonsets       # cordon + evict (respects PodDisruptionBudgets)
kubectl auth can-i create pods --as system:serviceaccount:ns:sa   # RBAC, answered

# supply chain
trivy image --severity HIGH,CRITICAL --exit-code 1 <img>   # the CI scan gate
cosign sign <img@sha256:...>  ;  cosign verify <img@...>   # sign the digest / verify

# IaC & GitOps
terraform plan   ;  terraform apply                # diff desired vs state, then execute
helm upgrade --install <rel> <chart> -f values.yaml  ;  helm rollback <rel> <rev>
argocd app sync <app>  ;  argocd app get <app>     # converge / show Synced-or-Drifted

sum by (service) (rate(http_requests_total[5m]))                       # RED: request rate
histogram_quantile(0.99, sum by (le) (rate(latency_seconds_bucket[5m]))) # RED: p99 duration

War stories

• The liveness probe that restarted the whole fleet. Someone pointed liveness at GET /health which pinged Postgres. Postgres failed over for 20 seconds; Kubernetes read every pod as dead and restarted the entire deployment into a thundering-herd cold start. Readiness checks dependencies; liveness checks only the process.
• "Just add more retries" for the throttling that was actually a capacity decision. API kept throwing 429s under load; the team bolted on retries, which made it worse. The real answer was right-sizing requests and letting the HPA + Cluster Autoscaler add capacity — the retries were hammering a fleet that arithmetic said was already Pending.
• The :latest tag that made staging and prod silently different. Two deploys of "the same image" pulled two different builds hours apart. Digests, not tags — promote the digest that passed the gates, don't rebuild.

Vocabulary

Reconciliation / level-triggered / idempotent · Pod / Deployment / Service / Ingress / ConfigMap / Secret / Namespace · requests / limits / QoS · affinity / taint / toleration / topology spread · Pending / FailedScheduling · liveness / readiness / startup probe · drain / PodDisruptionBudget · HPA / VPA / Cluster Autoscaler / KEDA · node pool / GPU shape / MIG / time-slicing · CNI / NetworkPolicy (default-allow) · RBAC (additive) / ServiceAccount / workload identity · Trivy / SBOM / cosign / verify-before-deploy · staged promotion / quality gate / rollback · GitOps / Argo CD / Flux / drift / self-heal · Terraform (state/plan/apply) / Helm (chart/release/rollback) · Pod Security Standards / admission / Kyverno · counter / gauge / histogram / rate / histogram_quantile / sum by · RED / USE · SLI / SLO / SLA / error budget / burn rate · RPO / RTO / blameless postmortem · OKE / OCIR / OCI Vault / availability domain / fault domain.

Beginner mistakes

1. Reading "Pending" as a bug instead of scheduling arithmetic — describe the pod, read the events, check requests vs allocatable.
2. Putting a dependency check in a liveness probe (self-inflicted restart storm).
3. Setting CPU limits on a latency-sensitive service and blaming the tail latency on the network.
4. Assuming NetworkPolicy or "Secrets" are secure by default — both are wide open until you configure them (default-allow; base64).
5. Expecting the Cluster Autoscaler to react to CPU — it reacts to Pending pods.
6. Rebuilding the image "for prod" instead of promoting the tested digest.
7. Targeting 100% availability, then having no budget to ship, drain, or survive a dependency blip.