« Phase 13 · Lab 01 · Track Overview

Warmup — The Cloud & Infrastructure Backbone, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: reconciliation
• 2. Terraform's model
• 3. Drift
• 4. Kubernetes is the same idea, continuously
• 5. GPUs break every assumption
• 6. Gang scheduling
• 7. MIG, and when to partition
• 8. Private endpoints, and what DNS does
• 9. Egress control
• 10. Proving a topology claim
• 11. The service mesh
• 12. Where the gateways sit
• 13. No long-lived secrets
• 14. Policy-as-code, at two layers
• 15. CI/CD and the supply chain
• 16. Numbers worth carrying
• 17. Interview questions, answered
• 18. References

0. Where this sits

This phase is the substrate every other phase runs on, and it is deliberately near the end of the track rather than the beginning — because the infrastructure decisions that matter are the ones the earlier phases generate requirements for:

And the phase's own question, which none of the others can answer: how do you make a topology claim you can prove?

Note what this phase is not. The Principal Azure Cloud Engineer track covers landing zones, ARM, RBAC and Azure networking in general. Here we take that as given and cover only the AI platform's slice — which is the GPU, residency and egress-control slice.

1. From first principles: reconciliation

One idea underlies Terraform, Kubernetes, Argo CD, Flux and every operator anyone has written:

    loop:
        desired = read_config()
        actual  = observe_reality()
        for difference in diff(desired, actual):
            act_to_close(difference)

That is it. The variations are when the loop runs and what it observes.

The interesting consequence is what happens between runs. Terraform's loop is manual, so between applies, reality drifts and nothing notices. Kubernetes' loop is continuous, so a deleted pod is recreated in seconds — but also, a change you make by hand is reverted in seconds, which surprises people the first time.

Neither is better. The point is that "desired state" is only meaningful with a stated reconciliation frequency, and for infrastructure that frequency is usually "when somebody remembers" — which is why §3 exists.

2. Terraform's model

Four concepts, and the first is the one that matters.

The resource graph. Resources with dependencies, forming a DAG. This is Terraform's real contribution — not HCL, which is a detail. Because infrastructure is a graph:

• ordering falls out (topological sort);
• parallelism falls out (independent subtrees apply concurrently);
• blast radius falls out (transitive dependents of a change).

Dependencies are mostly inferred from references — subnet_id = azurerm_subnet.aks.id creates an edge — with depends_on as the escape hatch for ordering the data flow does not express.

State. A file recording what Terraform believes exists, mapping addresses to real resource ids and their last-known attributes. State is the source of most Terraform pain and the reason is structural: it is a third thing, alongside the config and reality, and any two of the three can disagree.

Plan. A diff of desired against state — which is worth saying precisely, because people assume it is a diff against reality. It is not, unless you refresh. Actions:

Replace is the dangerous one. Changing a subnet's address prefix, or a cluster's location, destroys and recreates. For a stateful resource that is data loss, and it appears in the plan as one line among fifty that says # forces replacement. Reading for that line is a discipline, and prevent_destroy in a lifecycle block is the mechanism.

Apply. Walk the graph in dependency order, calling the provider. And the safety property that matters: a failure skips every dependent. Creating a private endpoint whose subnet failed to create produces a resource in an undefined state and a state file that disagrees with reality — much worse than stopping. State is written per resource as it succeeds, so a crash mid-apply leaves a state file describing what actually got built.

3. Drift

Divergence between state and reality. Three categories, and they need three different responses:

Unmanaged resources are the dangerous category because they are invisible to every process you have: they will not be destroyed when the environment is torn down, they will not be updated when a policy changes, they will not appear in a cost report attributed to anything, and nobody will notice when their certificate expires.

Drift happens for ordinary reasons: an incident where somebody scaled a node pool by hand at 3 a.m., a portal change, another team's automation, a cloud provider changing a default. So the useful framing is not "prevent drift" — it is drift detection is a run-state responsibility, scheduled and reviewed, exactly like reconciliation in Phase 12.

And the choice, each time: revert (the config is right) or absorb (reality is right, update the config). Doing neither is how an estate becomes un-reproducible, one emergency at a time.

4. Kubernetes is the same idea, continuously

A controller watches its resource type and reconciles toward the spec, forever. The Deployment controller sees replicas: 3, counts 2, creates one.

For an AI platform, the pieces that matter:

Namespaces as the tenancy boundary, with resource quotas and network policies.

Workload identity — the pod gets an Azure AD identity via a projected service-account token exchanged for an Entra token. This is Phase 08's "no long-lived secrets" expressed as infrastructure, and it is the mechanism that removes the last secret from the cluster.

Requests and limits. Requests drive scheduling; limits drive enforcement. Two consequences people get wrong:

• A pod with no limits can starve its neighbours. Hence the admission policy in the lab.
• A pod with requests == limits gets the Guaranteed QoS class and is evicted last, which for a stateful agent runtime is usually what you want.

And the memory/CPU asymmetry: exceeding a CPU limit throttles; exceeding a memory limit is an OOMKill. So a too-low CPU limit is a latency bug and a too-low memory limit is a crash loop, which is why the two need different tuning discipline.

5. GPUs break every assumption

Everything you know about autoscaling a stateless service is wrong here. Five reasons:

The first row is the one that reshapes the design. A node that takes nine minutes to serve traffic — a seven-minute node start plus two minutes of engine warm-up — cannot be provisioned reactively. By the time it is ready, the spike is over and you have paid for a node nobody used, or the spike is still going and you have had nine minutes of queueing.

So the answer is a warm pool: keep N idle nodes, sized from the arrival distribution rather than from current load, and scale the pool on a slow signal. That is a standing cost and it is the correct one, and being able to state it as a deliberate trade — "we pay $X/month to remove a nine-minute cold start" — is the difference between a design and an accident.

The other rows produce the rest of the design: taints and tolerations so only GPU workloads land on GPU nodes, node selectors per GPU type, and a scheduler that understands gangs.

6. Gang scheduling

The property that genuinely breaks Kubernetes' default model.

A 70B model with tensor parallelism across 8 GPUs needs all eight simultaneously. Not eight eventually — eight at once, because the model's layers are sharded across them and no shard can do anything alone.

Kubernetes' default scheduler places pods one at a time. So:

   8 GPUs free → schedule pod 1..6 → 2 GPUs left, another job takes them
   → pods 1-6 are RUNNING, HOLDING 6 GPUs, making no progress
   → they wait for pods 7-8, which cannot be scheduled
   → deadlock

Worse, the six held GPUs are unavailable to anything else, so a second gang job arrives, takes what is left, and now two jobs are deadlocked holding the whole cluster.

The fixes:

And the scheduling-order property from the lab: place the largest gangs first. A gang of 8 scheduled after four gangs of 2 may find no contiguous capacity even though the total is sufficient — fragmentation, and it is why bin-packing order matters here in a way it does not for stateless pods.

7. MIG, and when to partition

Multi-Instance GPU partitions one physical A100/H100 into hardware-isolated instances with their own memory, cache and SMs.

When it helps: many small models. Seven embedding models on one GPU, each isolated, each with predictable performance. Isolation is hardware, so a neighbour cannot degrade you — which is genuinely different from time-slicing.

When it does not: one large model. A slice's memory is a hard ceiling, and a 70B model in fp16 needs ~140 GB. Seven 10 GB slices do not help; they are seven places it does not fit.

Two operational facts: reconfiguring MIG requires draining the node (it is a device-level change), and MIG is not available on every GPU generation. Which makes the profile a node-pool-level decision, taken in advance — so a real estate has a MIG pool and a whole-GPU pool, and the routing between them is a capacity-planning exercise (Phase 05).

8. Private endpoints, and what DNS does

A private endpoint is a network interface in your subnet with a private IP that maps to a PaaS service. Traffic reaches it over the Microsoft backbone instead of the public internet.

Setting one up is three things, and only the first is what people remember:

1. The endpoint — the NIC and the private IP.
2. The private DNS zone — e.g. privatelink.openai.azure.com, with an A record pointing the service's name at the private IP.
3. The VNet link — linking that zone to the VNet that needs to resolve it.

Miss (2) or (3) and the FQDN still resolves to the public IP. Traffic takes the public path. Everything works. Nothing errors. The diagram is correct and the residency claim is false — and it stays false until an auditor asks or until somebody disables public access and it breaks for reasons nobody can find.

That is why the lab's prover treats an unlinked DNS zone as not a path: the endpoint exists and it is not being used.

And the second half, which is a separate setting entirely:

A private endpoint does not disable public access.

The service keeps its public FQDN and its public listener. The private endpoint adds a private route; it removes nothing. Closing the public path is public_network_access_enabled = false, and it is a different line in a different resource. Both are needed, and checking only one is the most common misconfiguration in this whole area.

9. Egress control

For an agent platform this is a security control, not hygiene — it is the enforcement point for the exfiltration defence in Phase 11.

The mechanisms, in increasing strength:

The strongest posture is the one to start from and relax deliberately: no outbound route. The subnet has no NAT gateway, no public IP and no default route. The only things reachable are what has a private endpoint. Then add a firewall with an explicit FQDN allow-list for what genuinely needs the internet.

Which is a real constraint you should anticipate: package installs, container pulls and telemetry all need egress. The answer is private registries, private package mirrors and private endpoints for telemetry — all of which are work, and all of which are the reason "we'll lock down egress later" turns into never.

10. Proving a topology claim

The phase's question, and its answer.

"No inference call on customer data leaves the UAE" is not a configuration. It is a property of the whole topology, and it is false if any path exists — which means checking one hop proves nothing.

The naive check is "does the workload's subnet have internet egress?" It misses three things, and each has been a real finding:

Peerings. Platform VNet → shared services VNet → legacy VNet in West Europe, which has had a NAT gateway since 2021. A per-VNet review clears the platform VNet and is correct and useless.

Intra-VNet routing. Azure routes between subnets in a VNet implicitly. So "is the private endpoint in my subnet?" is the wrong question — it only has to be somewhere in the VNet. A check that looks at the workload's own subnet reports a false negative, which trains people to ignore it.

DNS. §8. The endpoint exists, the FQDN resolves publicly, every packet takes the public path.

So the tool has to search the topology, and — the part that makes it useful — return a counter-example path when it finds one:

   subnet:aks -> peering:platform->shared -> subnet:shared-svc
              -> peering:shared->legacy -> subnet:legacy-app
              -> egress:legacy-app (internet) -> service:openai-global (westeurope)

A control that says "denied" is an opinion. A control that says "denied, and here is the path" is a finding somebody can fix.

Azure Network Watcher and AWS Reachability Analyzer do exactly this, and the reason to build a small one is to understand what they are checking — and what they are not.

11. The service mesh

A sidecar (or ambient) proxy alongside every workload, intercepting all traffic. What it buys an AI platform:

mTLS between every workload, with no application change. Which is Phase 08's workload identity, delivered by infrastructure — the mesh issues and rotates the certificates, and the application never sees one.

Retries, timeouts, circuit breaking, centrally configured. Note this overlaps Phase 10 and the overlap needs a decision: the mesh does not know a payment is non-idempotent, so mesh-level retries must be off for anything side-effecting. That is a real trap — a helpful platform default that double-executes payments.

Consistent-hash load balancing, which gives session affinity for the agent kernel's session store (Phase 01) without a sticky-session cookie.

ext_authz — an external authorization filter calling a PEP before forwarding. This is where Phase 09's policy engine plugs into the data plane, and it comes with a sharp edge worth naming: failure_mode_allow. Set true, an unreachable PEP means every request is allowed. Set false, an unreachable PEP means an outage. It is exactly the fail-open/fail-shut dilemma from Phase 09, and the fail-static answer is a local PDP (a sidecar), so the question never arises.

Sidecar versus ambient: sidecars cost ~50–100 MB and ~1 ms per pod and are mature; ambient mode moves L4 to a per-node ztunnel and L7 to an optional waypoint, cutting the per-pod cost substantially. For a large fleet of small agent pods, ambient is increasingly the right answer.

12. Where the gateways sit

Four things are called a gateway and they are at different layers:

   internet
      │
   [ APIM / Front Door ]        north-south: TLS, WAF, rate limit, subscription keys
      │
   [ ingress / mesh gateway ]   into the cluster
      │
   [ Envoy sidecars ]           east-west: mTLS, retries, ext_authz
      │
   [ LLM gateway ]              ← an APPLICATION (Phase 04), not network infrastructure
      │
   model endpoints

The one people get wrong is the last. The LLM gateway is an application, not a network gateway. It does model routing, fallback, token accounting, semantic caching and tenant quota — none of which an API gateway can express, because all of them need to understand the content of the request.

The temptation is to implement it as APIM policy. It ends as thousands of lines of unreviewable XML that nobody can test, and it cannot do the parts that matter (token counting, semantic caching) because those need a model's tokenizer.

So: APIM for transport concerns, the LLM gateway for model concerns, and they compose.

13. No long-lived secrets

Phase 08's requirement, expressed as infrastructure. Three mechanisms remove three classes of secret:

Managed identity. A resource gets an Entra identity; the platform issues tokens to it. No client secret exists to leak — which also means no rotation.

Workload identity federation. A Kubernetes service account token is exchanged for an Entra token via OIDC. The pod holds a projected token valid for an hour, and it is refreshed by the kubelet.

OIDC in CI/CD. GitHub Actions or Azure DevOps present a signed OIDC token; Entra trusts the issuer, scoped to a repository and branch. This removes stored cloud credentials from CI entirely — which matters, because a CI system with a stored service-principal secret is the highest-value target in the estate: it can deploy anything, anywhere.

What is left after all three: Key Vault, holding the secrets that genuinely cannot be federated — third-party API keys, mostly — accessed via managed identity, with rotation and access logging.

14. Policy-as-code, at two layers

Two layers, two tools, two moments:

Azure Policy stops a storage account being created without a private endpoint. Gatekeeper stops a pod being admitted with an unsigned image. Neither substitutes for the other, and a design with only one has a gap somebody will find.

The policies worth having for an AI platform, and each has a story behind it:

1. Restricted data requires a private endpoint and public access disabled.
2. Images must be signed by a trusted signer.
3. Images must be pinned to a digest — a tag is mutable, so :latest today is not :latest tomorrow, and the signature you verified was for a different artifact.
4. Every workload declares CPU and memory limits.
5. No public IPs on compute.
6. Every resource has an owner tag.
7. GPU pools carry taints.

And two implementation properties that matter more than the rule list:

Deny by default, with every deny naming its rule — "denied" is unactionable.

A policy that errors must deny. A policy engine that fails open when its own code breaks is a policy engine that will fail open during an incident, which is precisely when it is needed.

15. CI/CD and the supply chain

The pipeline is a control surface, and for an AI platform it carries the supply-chain risk that is OWASP's LLM03.

Authentication: OIDC federation, per §13. No stored credentials.

Supply chain, in order of how much each buys:

Signing plus admission verification is the one that changes the threat model: without it, everything else is advisory, because an unsigned image can still run.

Deployment: staged, and reversible. Dev → staging → canary → production, with an automated rollback trigger. Which connects to Phase 14: the rollback trigger is an SLO burn rate, not a human noticing.

And the phase's own framing: config is compliance. A routing rule that sends restricted data offshore is one commit away and takes effect everywhere at once. So infrastructure changes get the same treatment as code — review, staging, signing, reversibility, and a record of who changed what.

16. Numbers worth carrying

17. Interview questions, answered

Q1. "Explain Terraform's model."

A resource graph, state, plan and apply — and the graph is the part that matters. Because infrastructure is a DAG, ordering, parallelism and blast radius all fall out of one structure instead of being managed by hand in a script.

State is where the pain is, and the reason is structural: it is a third thing alongside the config and reality, so any two of the three can disagree. Which is why plan is a diff against state, not against reality — a plan can be empty while the estate is wrong, and that is what drift detection is for.

The two things I check in a plan. Anything marked forces replacement — that is a destroy and recreate, and for a stateful resource it is data loss hiding in one line among fifty. And the apply semantics: a failed resource must skip its dependents, because creating a private endpoint whose subnet failed produces something in an undefined state and a state file that disagrees with reality.

Q2. "How do you autoscale GPU inference?"

Mostly, you don't — not reactively. A GPU node takes five to eight minutes to start, mostly image pull and driver init, plus one to three minutes for the engine to load the model. That is about nine minutes from "we need capacity" to "it is serving", and reactive autoscaling against that is a nine-minute outage with a graph attached.

So: a warm pool, sized from the arrival distribution rather than from current load, scaled on a slow signal. That is a standing cost and it is the correct trade, and I would state it that way — we pay X per month to remove a nine-minute cold start.

The other thing that breaks is gang scheduling. A tensor-parallel deployment needs all eight GPUs simultaneously, and Kubernetes' default scheduler places pods one at a time — so it will give you six and leave them running, holding the GPUs, making no progress, waiting for two more that may never come. That is a deadlock, and it is why Volcano and Kueue exist.

Q3. "When would you use MIG?"

For many small models. MIG partitions an A100 or H100 into hardware-isolated instances — seven at 10 GB, three at 20, two at 40 — each with its own memory and SMs, so a neighbour cannot degrade you. Seven embedding models on one GPU with predictable performance is exactly the case.

Not for one large model. A slice's memory is a hard ceiling and a 70B model in fp16 needs about 140 GB, so seven 10 GB slices are seven places it does not fit.

Two operational facts that make it a node-pool-level decision rather than a per-workload one: reconfiguring MIG requires draining the node, and it is not available on every generation. So a real estate has a MIG pool and a whole-GPU pool, and the routing between them is a capacity-planning exercise.

Q4. "Prove that no customer data leaves the region."

That is not a configuration question, and it is the interesting thing about it. It is a property of the entire topology, and it is false if any path exists — so checking one hop proves nothing.

I would build, or use, a reachability prover. It searches from the workload across intra-VNet routing, peerings, private endpoints and egress policy, and returns a counter-example path when it finds one.

The three things a naive check misses, all of which I have seen be real. Peerings — the platform VNet peers to shared services, which peers to a legacy VNet in another region with a NAT gateway from 2021; a per-VNet review clears the platform VNet and is correct and useless. Intra-VNet routing — Azure routes between subnets implicitly, so "is the private endpoint in my subnet" is the wrong question. And DNS — if the private DNS zone is not linked to the VNet, the FQDN still resolves publicly and every packet takes the public path while everything appears to work.

The output has to be a path, not a boolean. A control that says "denied" is an opinion; one that says "denied, and here is the path" is a finding somebody can fix.

Q5. "What does a private endpoint actually do?"

It puts a NIC with a private IP in your subnet, mapped to a PaaS service, so traffic goes over the backbone instead of the internet.

Two things people miss, and both are how residency claims turn out false.

It does not disable public access. The service keeps its public FQDN and its public listener; the endpoint adds a private route, it removes nothing. Closing the public path is public_network_access_enabled = false, a separate setting on a separate resource. Both are needed.

DNS is the half that breaks. The endpoint needs a private DNS zone with an A record, and that zone needs to be linked to the consuming VNet. Miss either and the FQDN resolves to the public IP, so traffic takes the public path — and nothing errors, so it stays that way until an auditor asks.

Q6. "Where does the LLM gateway sit relative to APIM and the mesh?"

They are at different layers and the LLM gateway is not one of them. APIM is north-south — TLS, WAF, rate limiting, subscription keys. The mesh is east-west — mTLS, retries, ext_authz. The LLM gateway is an application: model routing, fallback, token accounting, semantic caching, tenant quota.

None of those are expressible in an API gateway, because they all need to understand the content of the request. Token counting needs a tokenizer; semantic caching needs an embedding. The temptation is to implement it as APIM policy, and it ends as thousands of lines of untestable XML that still cannot do the parts that matter.

One trap worth mentioning at that boundary: mesh-level retries must be off for anything side-effecting. The mesh does not know a payment is non-idempotent, and a helpful platform default that retries on 503 will double-execute it.

Q7. "What admission policies would you enforce?"

Seven, and I would enforce them at two layers — Azure Policy for resources, Gatekeeper or Kyverno for Kubernetes objects, because neither substitutes for the other.

Restricted data needs a private endpoint and public access disabled. Images must be signed by a trusted signer and pinned to a digest — a tag is mutable, so the signature you verified was for a different artifact. Every workload declares CPU and memory limits. No public IPs on compute. Every resource has an owner tag. GPU pools carry taints.

Two properties matter more than the list. Deny by default, with every denial naming its rule, because "denied" is unactionable. And a policy that errors must deny — an engine that fails open when its own code breaks will fail open during an incident, which is exactly when it is needed.

18. References

Infrastructure as code

• Terraform documentation · the graph · state
• OpenTofu
• Terraform AzureRM provider

Kubernetes and GPUs

• AKS GPU node pools · AKS best practices
• NVIDIA MIG user guide · GPU Operator
• Volcano · Kueue
• Kubernetes QoS classes

Networking

• Azure Private Link · private endpoint DNS
• Azure Firewall FQDN rules
• Network Watcher connection troubleshoot
• AWS Reachability Analyzer · batfish

Mesh and gateways

• Istio · ambient mode · ext_authz
• Linkerd · Azure API Management

Identity and policy

• Workload identity federation · AKS workload identity
• Azure Policy · OPA Gatekeeper · Kyverno

Supply chain

• SLSA · Sigstore Cosign · Syft · Trivy
• GitHub Actions OIDC with Azure