🛸 Hitchhiker's Guide — Phase 07: Containers, Docker, ACR & AKS

Read this if: you can write a Dockerfile and kubectl apply but "why did :latest change?", "why did deleting tags free no space?", and "why is my pod Pending?" still send you to the portal. This is the compressed tour the WARMUP derives slowly. Skim it, read the WARMUP, then build the registry + scheduler.

0. The 30-second mental model

An image is content named by a hash: an ordered stack of read-only layers + a config, each a sha256 digest; a manifest lists those digests; a tag is a mutable pointer to a manifest digest. From that one model, everything falls out:

push   = upload only the blobs the registry is MISSING        (dedup)
pull   = download manifest's blobs MINUS your local cache      (layer reuse)
GC     = mark from the TAGS, sweep unreferenced blobs          (shared layers survive)
replicate = copy the manifest + only the destination's MISSING blobs

And AKS is one more model: the scheduler bin-packs pods onto nodes by requests, under taints, against allocatable capacity. One sentence to tattoo: tags are mutable, digests are immutable — pin prod by digest; and a Pending pod is requests-vs-capacity arithmetic, not a mystery.

1. The image model, one breath

Dockerfile  →  layers (read-only, content-addressed) + config
            →  manifest (lists config + layer digests, itself a digest)
            →  tag  (mutable name → manifest digest)        e.g. app:1.0 → sha256:1d4f…

Bottom layers = rarely change (OS, deps) → cached & shared. Top layer = your app → changes every build. Layer boundaries, not total size, decide pull cost.

2. The numbers and rules to tattoo on your arm

3. The one-liners you'll actually type

Docker / OCI

docker pull app@sha256:1d4f…             # pin by digest (prod), not :latest
docker buildx build --platform linux/amd64,linux/arm64 -t app:1.0 --push .   # multi-arch
docker inspect --format '{{.RepoDigests}}' app:1.0    # the digest a tag resolves to
crane digest myacr.azurecr.io/app:1.0    # resolve tag → digest without pulling

Azure Container Registry

az acr build -r myacr -t app:1.0 .                      # build IN the registry, no local Docker
az acr login -n myacr                                   # token via your Entra identity
az acr repository show-tags -n myacr --repository app   # list tags
az acr manifest list-metadata -n myacr -r app           # see untagged manifests (GC fodder)
az acr run --cmd "acr purge --filter 'app:.*' --ago 30d --untagged" /dev/null  # purge task
az acr replication create -r myacr -l westeurope         # geo-replicate (Premium)
az role assignment create --assignee <mi> --role AcrPull --scope <acr-id>   # pull rights

AKS / kubectl

kubectl describe pod <p> | sed -n '/Events/,$p'         # the WHY of Pending lives here
kubectl get pod <p> -o jsonpath='{.spec.containers[*].resources}'   # requests/limits
kubectl describe node <n> | grep -A6 'Allocated resources'          # what's reserved
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl top node                                        # USAGE (not what scheduler packs by!)
az aks nodepool add -g rg --cluster-name c -n gpu --node-taints gpu=true:NoSchedule

4. Budget the registry, spend the cache

Two 900 MB images that share a 700 MB base store 1.1 GB, not 1.8 GB — the base lands once. A node that already cached the base pulls only the 200 MB that differs. So: put slow-changing layers at the bottom of your Dockerfile, keep base images shared across your fleet, and geo-replicate so the 200 MB delta is a local pull, not a transatlantic one. Juniors optimise image size; principals optimise layer reuse and cache locality.

5. War story shapes you'll relive

• ":latest broke prod and nobody deployed." → someone re-pushed the tag to a new digest. The fix is pinning @sha256:… and immutable tags — not "be more careful."
• "We deleted thousands of tags, the bill didn't move." → deleting tags ≠ deleting blobs; GC only sweeps what no tagged manifest references, and your old images shared layers with current ones. Run GC after untagging, and measure the exclusive closure.
• "Pod's Pending but the dashboard shows the node 20% busy." → the scheduler packs by requests, not usage; the node's CPU is already reserved. describe pod says Insufficient cpu. Right-size requests; don't stare at kubectl top.
• "GPU pod won't schedule even though the GPU node is empty." → the node is tainted (gpu=true:NoSchedule) and the pod lacks the toleration. Add the toleration (and a nodeSelector), don't remove the taint — the taint is doing its job.
• "Cold starts in Europe take forever." → images pulled cross-region from a US registry. Geo-replicate to a Premium ACR replica in the cluster's region.
• "It pulled exactly-once, we're fine." → you re-earn correctness at every boundary; same energy as exactly-once in messaging — pin the digest or you didn't pin anything.

6. Vocabulary that signals you've held the pager

• Digest — sha256:…; the immutable, content-derived name of a blob or manifest.
• Tag — a mutable pointer to a manifest digest (the thing that "changes under you").
• Manifest / image index — the list of blob digests / the multi-arch list of manifests.
• Blob — a content-addressed object (a layer or a config); stored once, referenced many.
• Dedup / layer reuse — the same bytes stored/transferred once across images and pulls.
• Garbage collection — mark-and-sweep from the tags; reclaims unreferenced blobs.
• Geo-replication — Premium ACR copying content to regional replicas (missing-only).
• Requests vs limits — placement floor (scheduler) vs runtime ceiling (kubelet).
• Taint / toleration — node repels pods / pod opts in; the node-pool isolation knob.
• Allocatable — capacity minus kube/system reserved; what the scheduler packs against.
• QoS class — Guaranteed / Burstable / BestEffort; sets memory-pressure eviction order.
• Pending / Unschedulable — no node passed the filter (capacity or taint).
• Managed identity + AcrPull — how a pod pulls without a secret.

7. Beginner mistakes that mark you in interviews

1. Saying "an image is a binary/VM" — it's content-addressed layers + a config + a manifest.
2. Treating :latest as a deployment contract instead of a mutable pointer. Pin the digest.
3. Thinking deleting tags reclaims storage. Untag and GC; measure the exclusive closure.
4. Believing pull cost ∝ image size. It's ∝ the layers you don't have cached.
5. Reading kubectl top to explain a Pending pod. The scheduler packs by requests.
6. Confusing requests and limits — or setting limits without requests (BestEffort surprise).
7. Removing a taint to schedule a pod, instead of adding the toleration the taint demands.
8. Pulling with the ACR admin password in prod instead of a managed identity + AcrPull.
9. Defaulting every workload to AKS when ACA (scale-to-zero) or ACI (one-shot) would do.

8. How this phase pays off later

• Digests + signing + scanning are the spine of P08 (CI/CD & secure supply chain) — the deploy gate pins a digest and verifies a signature.
• Managed identity + AcrPull is P03/P04 (Entra + RBAC) made concrete for a pod.
• Requests/limits/taints are the substrate for any autoscaling, cost, and reliability story (P14) — and the most common real incident you'll debug on a cluster.
• The registry engine and scheduler you build here are the mechanical models you'll run in your head during a 2 a.m. "why won't it pull / why won't it schedule" page.

Now read the WARMUP slowly, then build the registry + scheduler. You'll reach for its pull-as-set-difference and Pending-as-arithmetic models every time a cluster misbehaves.