Phase 07 — Containers, Docker, ACR & AKS Internals

Difficulty: ⭐⭐⭐☆☆ (the digest/manifest model) → ⭐⭐⭐⭐⭐ (the judgment about immutability, supply-chain trust, and why a pod is Pending) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (control plane vs data plane; the resource-ID shape), Phase 03 (Entra ID — the managed identity token an AKS pod uses to pull from ACR), Phase 04 (RBAC — AcrPull / AcrPush are role assignments). No Docker daemon or live cluster required for the lab; everything is offline and deterministic.

Why This Phase Exists

The JD names it plainly: "Build, deploy, and manage containerized applications using Docker and Azure Container Registry (ACR)." Almost every engineer can write a Dockerfile and run docker push. Almost none can answer the questions that actually decide a production incident:

• ":latest was fine yesterday and broke prod today and nobody deployed — how?" (a tag is a mutable pointer; someone repushed it).
• "Two images are 900 MB each but the registry only grew by 200 MB on the second push — why?" (content-addressed layer dedup; the base layer was already stored).
• "We deleted 4 000 old tags and reclaimed nothing — why?" (deleting tags doesn't delete blobs; garbage collection sweeps unreferenced blobs, and yours were still referenced by a tag you missed).
• "The pod is Pending and there's plenty of CPU on the dashboard — why won't it schedule?" (the scheduler packs by requests, not usage, and your node has a taint the pod doesn't tolerate).

A senior operates the tools. A principal holds the mechanism: an image is an ordered stack of read-only layers + a config, each named by a sha256 digest; a manifest lists those digests; a tag is a mutable name that points at a manifest digest. Once you hold that, ACR (a managed OCI registry), AKS (a managed control plane that bin-packs pods onto node pools), and the serverless container paths (ACA/ACI) stop being three products and become one content-addressed storage model and one scheduling model.

So this phase makes you build both engines: a content-addressable registry with dedup, missing-blob pulls, garbage collection, and geo-replication; and an AKS-style scheduler that bin-packs pods by requests under taints and tells you, arithmetically, why a pod can't be placed. After you've written the algorithm, "manifest digest" and "Pending pod" stop being portal mysteries and become functions you can run in your head at 2 a.m.

What "Principal-Level" Means Here

A senior pushes images and applies YAML. A principal understands the machinery well enough to:

• Reason in digests, not tags. Explain why a tag is mutable and a digest is immutable, why production deploys pin app@sha256:… (not app:latest), and how a content digest gives you both deduplication and tamper-evidence for free.
• Predict registry storage and pull cost. Given two images sharing a base layer, say how many blobs the registry stores (the shared layer once) and how many blobs a client with a warm cache must download (only the missing ones).
• Run garbage collection safely. Explain why deleting tags ≠ reclaiming storage, why GC is mark-and-sweep from the tagged manifests, and why a shared base layer survives until the last image referencing it is gone.
• Secure the supply chain. Pull with a managed identity (AcrPull), not an admin password; pin by digest; sign manifests (content trust / Notation); scan for CVEs; and use geo-replication so a region failure doesn't stall every pod start.
• Debug a Pending pod from first principles. Map kubectl describe pod output to the scheduler's filter predicates — Insufficient cpu/memory is requests-vs-allocatable arithmetic; had untolerated taint is a node-pool isolation choice — and fix it without guessing.
• Pick the right runtime. Know when AKS (full Kubernetes), Azure Container Apps (serverless Kubernetes-less, scale-to-zero, KEDA), or ACI (a single bare container) is the right tool — and defend it with the operability and cost dials, not taste.

Concepts

• The image model — layers (read-only, content-addressed) + a config; the digest (sha256:…) as the universal name; the manifest as the list of digests; the tag as a mutable pointer; the manifest list / image index for multi-arch.
• OCI distribution — pull = resolve tag → manifest → fetch only the MISSING layer blobs; content-addressed deduplication across images; the /v2/ API shape ACR speaks.
• ACR (Azure Container Registry) — a managed OCI registry; SKUs (Basic/Standard/ Premium); geo-replication; ACR Tasks (build-on-push, base-image-update triggers); content trust / signing; soft-delete; garbage collection of untagged manifests and unreferenced blobs; auth via managed identity + AcrPull.
• AKS (Azure Kubernetes Service) — a managed control plane (API server, etcd, scheduler, controller-manager — Microsoft-run) + customer node pools; the scheduler bin-packs pods onto nodes by requests (cpu/mem), bounded by limits, respecting taints/ tolerations and node allocatable capacity; QoS classes (Guaranteed/Burstable/ BestEffort) that decide eviction order; why a pod is Pending.
• Requests vs limits — requests are the floor the scheduler guarantees and places by; limits are the ceiling the kubelet enforces at runtime (CPU is throttled, memory is OOM-killed); the gap defines the QoS class.
• The serverless container paths — ACA (Container Apps: scale-to-zero, KEDA-driven, Dapr, no cluster to run) and ACI (a single container, per-second billing) as the non-Kubernetes contrast that frames when AKS is overkill.

Labs

Lab 01 — OCI Registry Engine + AKS Scheduler (flagship, implemented)

→ Lab folder: lab-01-oci-registry-engine/

Integrated-Scenario Suggestions (carried through the whole track)

These tie Phase 07 to the rest of the platform you are building across the track. Each is a design you should be able to defend with the dials from this phase:

1. A golden-image supply chain. ACR Tasks builds on base-image update; images are signed (content trust) and scanned (Defender/Trivy); deploys pin by digest via a CI/CD gate (P08); pull uses a managed identity with AcrPull (P03/P04). The deliverable: no human ever types a registry password, and a CVE in the base image triggers an automatic rebuild of everything FROM it.
2. A multi-region, fast cold-start cluster. ACR geo-replication puts blobs in every region an AKS cluster runs in, so a node in westeurope pulls locally, not across an ocean; node-pool taints isolate system / spot / GPU workloads; pod requests are tuned so the cluster autoscaler scales the right pool.
3. Cost-driven runtime selection. A bursty, event-driven worker runs on ACA (scale-to-zero, KEDA on queue depth) instead of an always-on AKS node pool; a one-shot batch job runs on ACI; a stateful, multi-team platform runs on AKS. The deliverable: an ADR that names the operability/cost dial for each.
4. A storage-reclamation program. Untagged manifests pile up from CI; an acr purge Task plus GC reclaims them on a schedule; soft-delete gives a recovery window. The deliverable: the registry bill stops growing linearly with build count.

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• A container image is content (layers + config) named by digests, listed by a manifest, pointed at by a mutable tag. That one model explains dedup, pulls, GC, and replication — they are all set operations on digests.
• Tags drift; digests don't. Pin production by digest. Treat :latest as a convenience for humans, never as a deployment contract.
• Deleting tags is not reclaiming storage. GC is mark-and-sweep from the tagged manifests; a shared layer survives until the last image using it is gone.
• A Pending pod is arithmetic, not magic. The scheduler places by requests under taints against allocatable capacity; Insufficient cpu and untolerated taint are the two answers, and both are fixable without guessing.
• Choose the runtime by the operability/cost dial: AKS for a platform, ACA for serverless/scale-to-zero, ACI for a single short-lived container.

Deliverables Checklist

• Lab 01 implemented; all tests pass against solution.py and against your lab.py
• You can draw the image model (layer → digest → manifest → tag) from memory
• You can explain why a warm-cache pull downloads only the new layers
• You can explain why GC after untagging reclaims exclusive blobs but keeps shared ones
• You can debug a Pending pod from kubectl describe output to a root cause
• You can pick AKS vs ACA vs ACI for a workload and defend it with the dials