👨🏻 Brother Talk — Phase 07, Off the Record

No slides, no certification blueprint. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. Containers are the part of the stack everyone thinks they understand because they've run docker build and kubectl apply a hundred times. And that's exactly why this phase matters: the gap between "I can run the commands" and "I understand the mechanism" is invisible until a senior architect asks you one question and watches you reach for the portal instead of the model. Let me save you that moment.

Here's the secret the whole phase turns on: an image is content named by a hash, and a tag is just a sticky note pointing at it. Once that clicks, you stop being surprised. You stop being surprised that two big images barely grow the registry (same layers, stored once). You stop being surprised that deleting a thousand tags frees nothing (you deleted the sticky notes, not the boxes). You stop being surprised that :latest "changed by itself" (someone moved the sticky note). The engineers who panic at these are the ones who memorised commands. The ones who shrug and say "yeah, content addressing, that's expected" are the ones who get the staff offer. The difference is one mental model, and you're building it this week.

So let me give you the mindset shifts, brother to brother.

Stop deploying tags. Deploy digests. I mean it. app:latest in a production manifest is a loaded gun pointed at a future incident, because a tag is mutable and someone will re-push it — a teammate, a CI job, a "quick fix" at 4 p.m. on a Friday. When you pin app@sha256:1d4f…, you've turned "what's running" from a question into a fact. The first time you debug an outage and can say "the running digest is X, the tag now points at Y, here's exactly when it diverged" — while everyone else is guessing — you'll feel the room shift. Digests are how you make "what's deployed" legible. Legibility is seniority.

Garbage collection is where careful people get burned, so respect it. The instinct "delete old stuff to save money" is right; the execution is where it goes wrong. Deleting tags doesn't reclaim bytes — GC does, and GC only sweeps what no live image references. The flip side is the scarier one: run an over-eager purge without understanding reachability and you can delete a base layer ten current images depend on, and now nothing pulls. The principal move is to know the closure: which blobs are exclusive to the thing you're deleting, and which are shared. The lab makes you compute that closure by hand. Do it until it's boring, because the day you run a real acr purge you want it to be boring.

Fall in love with the boring force again: a Pending pod is not a mystery. This is the single most common thing you'll debug on a cluster, and ninety percent of engineers treat it like weather — they stare at the dashboard, restart things, scale up randomly. Don't be them. A Pending pod is arithmetic: the scheduler packs by requests against allocatable capacity, under taints. kubectl describe pod, read the Events, and it literally tells you: Insufficient cpu (the requests don't fit) or untolerated taint (the node pool is reserved). That's it. Those are the two answers. When the room is panicking about "why won't it schedule," and you calmly read the describe output and say "the committed requests on every node already sum to allocatable — we're not out of CPU, we're out of requestable CPU; right-size the requests or add a node" — you're not smarter than them. You just read the machine instead of the dashboard. That's the whole flex.

Learn the requests-vs-limits distinction cold, because almost nobody has. Requests are what the scheduler uses to place; limits are what the kubelet uses to enforce. CPU over the limit gets throttled; memory over the limit gets OOM-killed — silently, brutally, at 3 a.m. The relationship between them sets your QoS class, which decides who gets evicted first when a node runs out of memory. If you set limits but not requests, you've quietly opted into BestEffort and you'll be the first one killed. Most "random restarts" and "the pod just disappeared" tickets are an OOM-kill or an eviction nobody understood. Understand it, and you fix in five minutes what costs other teams a week.

Don't put everything on Kubernetes because it's the cool one. This is a career-shaping judgment call, not a tech preference. AKS is a platform — you own node pools, upgrades, the whole sprawling Kubernetes surface, and the on-call that comes with it. For a bursty queue worker that's idle half the day, Azure Container Apps with scale-to-zero costs nothing overnight and you run no cluster. For a nightly batch, ACI spins up one container and bills by the second. The engineer who reflexively reaches for AKS for everything is the one whose team is perpetually busy operating infrastructure that a managed service would have run for free. Naming the operability/cost dial out loud — "this doesn't need a cluster, it needs scale-to-zero" — is exactly the principal-level judgment the interviewer is listening for.

The honest truth about this phase's difficulty: it's not deep, it's deceptively simple. The whole thing is a hash function and some set operations and a bin-packing loop. There's no scary math. The difficulty is that the simplicity hides behind a mountain of tooling — Docker, BuildKit, OCI, ACR, AKS, kubelet, scheduler, KEDA — and most people learn the tools and never extract the model. You're doing it backwards on purpose: build the 30-line registry and the 15-line scheduler, watch dedup and Pending emerge from nothing, and then every product is just "oh, this is that model with a billing page and a portal."

One more thing, and it's the one that lasts. The reason we obsess over digests and immutability and supply chain isn't pedantry — it's that containers are how code gets from your laptop to production, and that path is exactly where attackers and accidents live. A poisoned base image, a re-pushed tag, an unsigned manifest — these are real breaches that have hit real companies. When you pin digests, sign manifests, scan for CVEs, pull with a managed identity instead of a password, and rebuild automatically when a base CVE drops, you're not checking compliance boxes. You're being the person who made the supply chain defensible. That person is who the org trusts with production. That's the ladder.

Go build the registry and the scheduler. Then come find me in Phase 08 — that's where we take these digests and wire them into a CI/CD pipeline with approval gates and OIDC federation, and the "secure supply chain" stops being a buzzword and becomes a thing you can draw.

— your brother 👨🏻