« Phase 13 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. The dependency graph, mechanically
• 2. State: locking, corruption, and moves
• 3. What ForceNew really costs
• 4. Refresh, and the drift you cannot see
• 5. Kubernetes admission
• 6. The GPU stack, layer by layer
• 7. Scheduling, and why bin-packing is hard here
• 8. Private endpoint DNS, precisely
• 9. Routing: UDRs and the path a packet takes
• 10. NSGs and service tags
• 11. Reachability as a search problem
• 12. ext_authz and the fail-open switch
• 13. Sidecar versus ambient
• 14. Workload identity federation, step by step
• 15. Supply chain: what each control stops
• 16. Failure modes

1. The dependency graph, mechanically

Terraform builds the graph from two sources:

Implicit — an interpolation creates an edge:

resource "azurerm_private_endpoint" "openai" {
  subnet_id = azurerm_subnet.endpoints.id     # ← edge: endpoints -> openai
}

Explicit — depends_on, for ordering the data flow does not express:

depends_on = [azurerm_role_assignment.kv_reader]   # the app needs the role first,
                                                   # but never references it

The implicit form is preferred because it cannot go stale. The explicit form is the escape hatch, and it is over-used — a depends_on that duplicates an existing reference adds nothing, and one that papers over an eventual-consistency race hides a real problem.

Ordering is a topological sort, and the frontier must be sorted for determinism. Without that, two runs over the same graph produce different orders, and a plan that reorders between runs cannot be reviewed — a reviewer cannot tell a reordering from a change.

Parallelism falls out: independent subtrees apply concurrently (-parallelism=10 by default). Which is also a failure mode — ten concurrent creates against a subscription with an API rate limit produces throttling that looks like a provider bug.

Blast radius is the transitive dependent set. It is worth computing before a change to a shared resource, because "I am editing the VNet" and "I am editing five things downstream of the VNet" are different conversations.

2. State: locking, corruption, and moves

State is a JSON document mapping addresses to real resource ids plus last-known attributes. Three operational realities:

Locking. Two concurrent applies would interleave writes and corrupt it, so backends take a lock — a blob lease on Azure, a DynamoDB item on AWS. Which produces the familiar failure: a killed apply leaves the lock held, and terraform force-unlock exists for it. Before force-unlocking, establish that no apply is actually running, because unlocking a live apply is how state genuinely gets corrupted.

State contains secrets. Any attribute a provider returns is stored, including connection strings and generated passwords. So the state backend needs encryption at rest, restricted access and audit logging — it is a secrets store whether or not anyone treats it as one.

Refactoring the config means moving state. Renaming a resource or extracting a module changes the address, and Terraform sees a destroy plus a create. moved blocks (or terraform state mv) record the rename:

moved {
  from = azurerm_subnet.aks
  to   = module.network.azurerm_subnet.aks
}

Without it, a refactor that changed no infrastructure destroys and recreates everything it touched. This is the single most common way a "cleanup" PR becomes an incident.

And splitting state is a real design decision. One state for everything means one lock, one blast radius and slow plans. Many states mean cross-state references via terraform_remote_state or data sources, and a dependency order between applies that nothing enforces. The usual split is by lifecycle: network (rarely changes), platform (sometimes), workloads (constantly).

3. What ForceNew really costs

A provider marks attributes ForceNew when the API has no in-place update. Changing one is destroy + create:

Three defences:

prevent_destroy. A lifecycle block that turns a replacement into a plan error. Correct for anything stateful, and it means a genuine move requires deliberately removing the guard — which is the point.

create_before_destroy. Reverses the order, so the new one exists before the old is removed. Only works when the two can coexist — a subnet with a fixed CIDR cannot, a VM scale set can.

Read the plan. # forces replacement is the string. In a fifty-resource plan it is one line, and a CI check that greps for it and requires an explicit approval label is fifteen minutes of work.

4. Refresh, and the drift you cannot see

   terraform plan                  # diffs config against STATE, refreshing by default
   terraform plan -refresh=false   # diffs against state only — fast, and blind
   terraform plan -refresh-only    # diffs STATE against REALITY — this is drift detection

The third is the one that matters and the one nobody runs. It answers "has anything changed outside Terraform?", which is the question the other two cannot.

What refresh cannot see:

Unmanaged resources are the dangerous category, and finding them needs a different tool: Azure Resource Graph queries or driftctl-style comparison of the whole subscription against all state files. Worth running quarterly, and the first run always finds something.

ignore_changes deserves care. It is necessary — an autoscaler changes node_count and you do not want to fight it — and it creates blind spots by design:

lifecycle {
  ignore_changes = [node_count]      # necessary, and now invisible
}

The discipline is to ignore the narrowest possible attribute, never all, and to write down why.

5. Kubernetes admission

Every object passes through a chain before it is persisted:

   request ──► authn ──► authz ──► MUTATING admission ──► schema validation
           ──► VALIDATING admission ──► etcd

Two webhook types, and the order matters:

Mutating runs first and can change the object — inject a sidecar, add default limits, add labels. Istio's sidecar injection is a mutating webhook.

Validating runs after and can only accept or reject. Gatekeeper and Kyverno policies are validating webhooks (Kyverno also mutates).

Three configuration properties with sharp edges:

failurePolicy: Fail vs Ignore. Fail means an unreachable webhook blocks every matching request — including, potentially, the pods that are the webhook, which is a cluster-wide deadlock after a full outage. Ignore means the policy silently stops applying. The standard answer is Fail plus an exemption for kube-system plus at least two webhook replicas across zones.

timeoutSeconds. Default 10, max 30. A slow webhook adds that latency to every object creation, which surfaces during a large rollout as inexplicable slowness.

Ordering is not guaranteed among webhooks at the same stage, so two mutating webhooks that both edit the same field produce a result that depends on registration order. Rare and extremely confusing when it happens.

And the thing admission cannot do: it validates at write time only. A policy added today does not evaluate yesterday's workloads. Gatekeeper's audit mode scans existing objects and reports violations, which is a separate mechanism and is how you find out what you already have.

6. The GPU stack, layer by layer

   pod: resources.limits."nvidia.com/gpu": 8
     │
   kubelet ──► device plugin (nvidia-device-plugin)     ← advertises the resource
     │
   containerd ──► nvidia-container-runtime              ← injects devices + libraries
     │
   driver (host)  ──► CUDA ──► the GPU

The GPU Operator installs and manages driver, container toolkit, device plugin, DCGM exporter and MIG manager as a bundle, and it is the right default — hand-installing drivers on node images is a maintenance obligation with no upside.

Three things worth knowing:

GPUs are not divisible by default. nvidia.com/gpu: 1 gets a whole GPU; there is no 0.5. Sharing needs MIG (hardware) or time-slicing (software, no isolation, and one workload can starve another).

Node start is dominated by the image pull. An LLM serving image with CUDA, PyTorch and the engine is 5–20 GB. Which points at the fixes: pre-pull on the node image, use a registry with a local cache, or keep nodes warm.

Monitor two numbers, not one. DCGM exports GPU memory and SM occupancy, and they disagree: a model can fill VRAM at 15% utilization (memory-bound decode) or run at 95% SM with memory to spare. Alerting on one of them alone produces confident wrong conclusions (Phase 05).

7. Scheduling, and why bin-packing is hard here

The default scheduler: filter feasible nodes, score them, place the highest. Per pod, independently.

That model fails here for three reasons.

Gangs. Covered in the warmup. One pod at a time plus all-or-nothing requirements equals deadlock. Volcano's PodGroup with minAvailable makes the group the scheduling unit; Kueue does it with quota-aware admission.

Fragmentation. Two nodes with 8 GPUs each, four 2-GPU jobs placed badly:

   node A: [job1][job1][job2][job2][ ][ ][ ][ ]
   node B: [job3][job3][job4][job4][ ][ ][ ][ ]
   → 8 GPUs free, and an 8-GPU gang cannot be placed

Which is why the lab places largest-first: it is the same first-fit-decreasing heuristic that bin-packing has always used, and it is a heuristic — it does not eliminate fragmentation, it makes it less likely.

Topology. Eight GPUs on one node connected by NVLink is very different from eight across two nodes over Ethernet, for a tensor-parallel model where every layer synchronizes. The scheduler needs to prefer single-node placement, and expressing that needs topology-aware scheduling or explicit node-level requests.

Preemption interacts badly with gangs. Preempting one pod of a running gang gains you one GPU and kills a whole job. A preemption policy that does not understand gangs will do exactly that, which is another reason the batch scheduler is not optional.

8. Private endpoint DNS, precisely

The mechanism, end to end:

   1. create the private endpoint       → a NIC, private IP 10.10.2.4
   2. create privatelink.openai.azure.com  (a private DNS zone)
   3. add an A record: myaccount → 10.10.2.4
   4. LINK the zone to the VNet          ← the step that is missed
   5. the workload resolves myaccount.openai.azure.com
        → CNAME to myaccount.privatelink.openai.azure.com   (public DNS returns this)
        → the linked private zone answers 10.10.2.4

Step 5 is worth reading twice. Public DNS returns a CNAME to the privatelink name — that part is automatic. The private zone is what resolves that CNAME to a private IP. Without the link, the CNAME resolves through public DNS to the public IP, and the connection succeeds over the internet.

Which produces the failure signature: it works, and it should not. Nothing errors. The only symptoms are a residency claim that is false and, later, a connection that breaks the day somebody sets public_network_access_enabled = false.

Where it goes wrong in practice:

The last three are the hub-and-spoke reality, and they are why "we have private endpoints" is a statement about intent rather than about packets.

9. Routing: UDRs and the path a packet takes

Azure's effective route table, in precedence order:

1. User-defined routes (UDRs) — highest
2. BGP routes (ExpressRoute, VPN)
3. System routes — default

A UDR forcing egress through a firewall:

   Route: 0.0.0.0/0 → VirtualAppliance → 10.0.1.4 (Azure Firewall)

Now everything leaving the subnet goes through the firewall, and the firewall's FQDN rules apply. This is the mechanism behind §9 of the warmup, and it is the thing the lab's model omits — which is its most significant honest limitation, because a UDR is how most real topologies control egress.

Three subtleties that produce real incidents:

Longest prefix wins. A /32 route beats a /0. So one specific route can bypass the firewall for one destination, and that is exactly how an exception is granted and then forgotten.

Asymmetric routing. Traffic out through the firewall, return traffic direct — the firewall drops the return because it never saw the request. Classic, and the symptom is a connection that hangs rather than fails.

Peering does not carry routes by default. Hub-and-spoke needs allow_forwarded_traffic and UDRs in the spokes pointing at the hub firewall. Without both, a spoke's traffic to another spoke takes the direct peering path and never passes the firewall — which means the control exists and is not in the path.

10. NSGs and service tags

First match by priority wins, 100–4096, evaluated low to high. This is a genuine trap for anyone arriving from Phase 09, where deny beats allow regardless of order:

   priority 100: ALLOW  *          → *         ← matches everything
   priority 200: DENY   *          → internet  ← never evaluated

The deny is dead code. A rule set's meaning depends entirely on the numbers, and reviewing one requires sorting it — which is why NSG rules are usually generated rather than hand-written.

Service tags are named IP sets Microsoft maintains: Internet, Storage, Storage.UAENorth, AzureCloud, AzureActiveDirectory. They make rules writable and they are much broader than they read:

• Storage is every storage account in the cloud, not yours.
• AzureCloud is essentially the whole Azure IP space.

So ALLOW → Storage permits egress to any storage account anywhere — an exfiltration channel with a reassuring name. Regional tags (Storage.UAENorth) narrow it; private endpoints remove the need for it entirely, which is the better answer.

Application security groups (ASGs) let rules reference a logical group of NICs rather than CIDRs, which is how you write maintainable rules in a dynamic environment. Neither service tags nor ASGs are in the lab's model, and both are in every real one.

11. Reachability as a search problem

The formalization: a graph where nodes are network locations and edges are permitted transitions.

   nodes: subnets, private endpoints, service endpoints
   edges: intra-VNet routing, peerings, UDR next-hops, egress paths
   guards: NSG rules, firewall policy, DNS resolution

Then: is there a path from the workload to the service, and does any such path leave the region?

Three properties that make an implementation useful:

Return paths, not booleans. A counter-example is a finding somebody can fix; "denied" is an opinion, and "allowed" is useless without knowing how.

Explain the negatives too. When there is no path, which guard blocked each attempt? Without that, an unreachable result sends somebody on a hunt. This is why the lab accumulates blocked_by.

Be honest about completeness. The lab visits each subnet once, so it finds a path per subnet rather than every path — fine for a counter-example, not a proof of the negative. Real tools have the same class of limitation, and a claim of "no path exists" is only as strong as the model's coverage of route tables, firewall rules and DNS.

Which is worth saying to an examiner rather than hiding: "this tool proves the positive — here is a path. Its negative result is bounded by these modelling assumptions."

Production tools: Azure Network Watcher connectivity check (live, actually sends packets), AWS Reachability Analyzer (static, over the config), and batfish (static, multi-vendor, the most thorough).

12. ext_authz and the fail-open switch

Envoy's external authorization filter calls a service before forwarding:

http_filters:
- name: envoy.filters.http.ext_authz
  typed_config:
    grpc_service:
      envoy_grpc: { cluster_name: opa-sidecar }
    failure_mode_allow: false          # ← the switch
    with_request_body:
      max_request_bytes: 8192          # ← the body the PDP can see

failure_mode_allow is exactly the fail-open/fail-shut dilemma from Phase 09, at the network layer:

Neither is right, and the resolution is the Phase 09 one: make the PDP local (an OPA sidecar with a pushed bundle), so it cannot be unreachable independently of the workload. Then false costs nothing, because a dead sidecar means a dead pod, which the mesh routes around.

Two other properties worth knowing:

with_request_body is bounded. The PDP sees at most max_request_bytes. A policy that needs to inspect a large body silently sees a truncated one, which is a class of bug that only appears on large requests.

Latency is on every request. A sidecar PDP is sub-millisecond; a remote one is 5–20 ms on everything. Same conclusion.

13. Sidecar versus ambient

For an AI platform the arithmetic is direct: a fleet of many small agent pods pays the sidecar tax per pod. Two hundred pods at 80 MB is 16 GB of memory doing nothing but proxying. Ambient moves that to one ztunnel per node.

The counter-consideration: L7 features need a waypoint, so if every namespace needs ext_authz you have reintroduced a proxy per namespace — cheaper than per pod, not free.

And the upgrade property matters more than it looks. Upgrading sidecars means restarting every pod in the mesh, which for a stateful agent runtime with long-running tasks is a genuine operational event (Phase 01). Ambient decouples that.

14. Workload identity federation, step by step

   1. the pod's service account is annotated with a client id
   2. the kubelet projects a signed SA token into the pod (audience: api://AzureADTokenExchange)
   3. the SDK reads the token from the projected volume
   4. it presents the token to Entra as a client assertion
   5. Entra validates it against the cluster's OIDC ISSUER (a public JWKS endpoint)
   6. Entra returns an access token for the managed identity
   7. the app calls Azure with it

What this removes: any secret that could leak. There is no client secret, no certificate, no credential file. The projected token is short-lived (default one hour), refreshed by the kubelet, and bound to a specific service account in a specific namespace in a specific cluster.

The federated credential's subject is the binding, and it is where the security lives:

   subject: system:serviceaccount:ai-platform:agent-kernel
   issuer:  https://uaenorth.oic.prod-aks.azure.com/<tenant>/<cluster>/

Two consequences:

Any pod using that service account gets the identity. The boundary is the service account, not the pod — so a debugging pod in the same namespace with the same SA has the agent's permissions. One service account per workload, and namespace-level RBAC on who can create pods with it.

The OIDC issuer is public. That is fine — it publishes only a JWKS — but it means the cluster's issuer URL is a name in Entra's trust configuration, and recreating a cluster changes it. Which is a migration step people forget.

This is the same RFC 8693-adjacent machinery as Phase 08, delivered by the platform: an assertion about a workload, exchanged for a scoped credential, short-lived.

15. Supply chain: what each control stops

The order matters. Signing plus admission verification is the one that changes the threat model — without it everything else is advisory, because an unsigned image can still run. An SBOM on an unverified image tells you what was in an image, not the one running.

Keyless signing with Cosign is worth understanding because it removes the key-management problem:

   cosign sign --yes registry.bank.ae/agent-kernel@sha256:abc...
   # → an ephemeral key, an OIDC identity from the CI, a certificate from Fulcio,
   #   and the signature recorded in Rekor (a transparency log)

There is no key to store or rotate. The signature's identity is the CI workload's OIDC identity, and verification checks that identity rather than a public key:

   cosign verify --certificate-identity-regexp '^https://github.com/bank/.*' \
                 --certificate-oidc-issuer https://token.actions.githubusercontent.com

And digest pinning is what makes signing meaningful. Verify myimage:v1.2, then somebody repushes that tag, and you are running something you never verified. Pin @sha256:... and the reference is immutable.

16. Failure modes