« Phase 13 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: control against velocity
• 2. How to split state
• 3. Cluster topology
• 4. Sizing the warm pool
• 5. MIG pool or whole-GPU pool
• 6. Mesh or no mesh
• 7. Where policy is enforced
• 8. Multi-cloud, honestly
• 9. Proving residency to an examiner
• 10. Setting the numbers
• 11. Migration: bringing an estate under control
• 12. What I would not build

1. The central tension: control against velocity

Every control here costs a team's ability to move.

   FAST                                                            CONTROLLED
     │                                                                    │
   portal      Terraform,      + admission     + signing,     + change advisory
   clicks      no policy         policies        staged         board
     │              │                │              │                │
   minutes      minutes          minutes        ~an hour        ~a week
   nothing      reproducible     enforced       verifiable      approved

Choosing one point for everything fails in both directions. Too little control and the estate is not reproducible and not defensible; too much and teams route around it — and a bypassed control is worse than a loose one, because it is also invisible.

The per-class table I would defend:

And the property that makes the strict end survivable: make the paved road faster than the bypass. A module that creates a compliant namespace in one command competes with a portal click. A six-page form does not, and teams will find the portal.

2. How to split state

The decision that shapes every subsequent day of operating this.

By lifecycle is the answer, and the split is by how often it changes:

   network/     VNets, subnets, peerings, DNS zones     — changes quarterly
   platform/    AKS, ACR, Key Vault, model endpoints    — changes monthly
   workloads/   deployments, configs, scaling           — changes daily

Three properties this buys:

Plan time is bounded. A workload change plans in seconds against fifty resources, not minutes against a thousand. That difference determines whether people run plan before pushing.

Blast radius matches the change. Editing a workload cannot destroy the VNet, because the VNet is not in that state file.

Lock contention is separated. Ten teams applying workload changes do not block each other on the network state's lock.

The cost is cross-state references, and the discipline that makes them safe: reference by data source, never by remote state output. A data source looks up a resource by name and fails loudly if it is gone; a remote-state output couples you to another state file's internal structure and breaks silently when it is refactored.

And the ordering nothing enforces: applying workloads before network fails. Documenting the order is not enough — an orchestrator (Terragrunt, Atlantis, a pipeline) that encodes it is.

3. Cluster topology

For a bank the answer is usually the third, and the driver is residency rather than isolation: a cluster in a region is a hard boundary that a namespace is not, and "UAE workloads run in a UAE cluster" is a sentence an examiner can verify.

The honest position on namespace isolation: it is not a security boundary for hostile tenants. Network policies, RBAC and quotas separate teams; they do not contain a container escape. For an AI platform where all tenants are internal bank teams, that is acceptable and should be stated explicitly rather than implied. If one tenant were genuinely untrusted, the answer is a separate cluster or a sandboxed runtime (gVisor, Kata), not more network policies.

Node pools within a cluster are where the real separation lives:

The system pool is the one teams skip and then learn: CoreDNS evicted by a memory-hungry workload takes out name resolution for the cluster, and the symptom looks like a network problem.

4. Sizing the warm pool

The most consequential number in this phase, because it is a standing cost.

Do not size it from current load. Size it from the arrival process:

1. Measure the distribution of scale-out events, not utilization — how often does demand exceed capacity, and by how much?
2. Take the p95 of the increment. That is what the pool must absorb.
3. Add the time to replenish: after the pool is consumed, it takes ~9 minutes to refill, so the pool must cover demand for that window too.
4. Price it, and state the trade.

The sentence to be able to say: "we hold four idle A100s at roughly $X per month, which removes a nine-minute cold start on the p95 spike. Removing the pool saves $X and adds nine minutes of queueing to roughly N requests per week."

That framing turns an infrastructure cost into a product decision, which is where it belongs — and it is the two-in-a-box conversation.

Three refinements worth knowing:

Predictive scaling beats reactive when demand is diurnal, which for an internal bank platform it strongly is. Scale the pool on a schedule derived from last week, and use reactive scaling only for the residual.

Spot for the pool is tempting and wrong for the warm pool specifically — an evicted warm node is not warm. Spot is right for batch evaluation and indexing.

Reserved capacity or committed-use discounts change the arithmetic by 30–60%, and they change it in the direction of more standing capacity. Which means the finance conversation and the architecture conversation are the same conversation.

5. MIG pool or whole-GPU pool

The decision rule: MIG for many small models with predictable footprints; whole GPU for anything large; time-slicing for development only.

Time-slicing deserves the warning. It gives no isolation, so one workload's memory allocation can OOM another, and latency becomes unpredictable in a way that is very hard to attribute. For an internal dev cluster that is fine. For anything with an SLO it is a source of incidents that look like application bugs.

And the operational constraint that makes this a pool-level decision rather than a per-workload one: reconfiguring MIG requires draining the node. So you cannot adapt to demand; you decide in advance, and a real estate runs both pools with a routing decision between them (Phase 05).

Which raises the sizing question — what fraction MIG? Answered by the model portfolio, not by utilization: count the models that fit in a 10 or 20 GB slice, multiply by their replica counts, and that is the MIG demand. It changes when the portfolio changes, which is a quarterly review rather than an autoscaling problem.

6. Mesh or no mesh

The honest question, because a mesh is a large operational commitment.

What a mesh gives an AI platform, in order of value:

1. mTLS everywhere with no application change — this is the one that justifies it in a bank, because the alternative is every team implementing TLS correctly.
2. ext_authz as a uniform PEP hook (Phase 09).
3. Consistent-hash load balancing for session affinity (Phase 01).
4. Uniform telemetry.
5. Traffic shifting for canaries.

What it costs: memory and latency per pod, an upgrade that restarts everything, a new failure domain, and a debugging surface that is genuinely harder — "is this the app, the sidecar, or the control plane?" is a question that costs time on every incident for the first six months.

The decision rule I would defend: take a mesh if you need mTLS everywhere or a uniform PEP; do not take one for retries and telemetry. Retries you can do in a library — and for anything side-effecting you must do it in the gateway rather than the mesh (Phase 10), because the mesh does not know what is idempotent. Telemetry you get from OpenTelemetry SDKs with better semantics.

And if you take one, ambient over sidecars for a fleet of many small pods — the per-pod tax at two hundred pods is 16 GB of memory doing nothing but proxying, and the upgrade story is materially better for long-running agent tasks.

7. Where policy is enforced

Five enforcement points, and the design decision is which invariants live where:

The rule: shift left for speed, enforce right for guarantees. CI checks give a developer feedback in seconds and are trivially bypassable, so the same rule must also exist at an unbypassable point. Having only the CI check is the common mistake — it feels like enforcement and is not.

The corollary is that rules get written twice, and that is a real cost. It is worth paying for the handful of invariants that matter (restricted data, signing, public IPs) and not for stylistic preferences.

Azure Policy versus admission is not either/or:

• Azure Policy stops a storage account being created without a private endpoint — including from the portal, the CLI, or another team's pipeline.
• Admission stops a pod with an unsigned image — which Azure Policy cannot see.

Neither substitutes for the other, and a design with only one has a gap somebody will find.

And the deployment discipline for policy itself: audit mode first, always. Deploy a new policy in audit, look at what it would have blocked, fix the legitimate violations, then enforce. Enforcing a new policy directly is how you break a team's deploy at 4 p.m. on a Thursday and lose the argument for the next three policies.

8. Multi-cloud, honestly

The JD says Azure primary, AWS secondary, and the honest engineering position is worth stating clearly because it is usually stated badly.

What genuinely portable means: Kubernetes manifests, container images, Terraform structure, OPA policies, OpenTelemetry instrumentation.

What is not portable, whatever anyone claims: identity (Entra vs IAM), networking (VNet vs VPC — different primitives, not different names), managed services (AI Foundry vs Bedrock), and the operational model.

Which produces three positions:

For an AI platform, portable core is the defensible answer. The agent runtime, gateways and policy engine are container workloads that run anywhere. The model endpoints, identity and networking are cloud-specific and you accept that — with an abstraction only at the model layer, which Phase 04 already built for other reasons and which happens to be the portability that matters.

The trap to name explicitly: an abstraction layer over both clouds' primitives. It ends as the intersection of two feature sets, maintained by you, always behind both. If somebody proposes it, the question is which specific workload will move, when, and what it is worth.

9. Proving residency to an examiner

This is the phase's deliverable, and it is worth designing as an artifact rather than assembling under pressure.

An examiner asks: "how do you know no customer data leaves the UAE?" What actually answers it:

One — the topology proof. The reachability analysis, run on a schedule, with its output retained. Not "we configured it" but "here is the analysis, run daily for the last year, and here is the day it found something and here is the ticket that closed it."

Two — preventive policy. Azure Policy denying resource creation outside the region, and admission denying workloads without the right node selectors. Prevention plus detection, because either alone has a gap.

Three — the decision and action records. From Phase 09 and Phase 10: every model call recorded with its endpoint and region.

Four — the model-layer constraint. The gateway's routing policy refuses to route restricted classifications to a non-regional endpoint (Phase 04) — a control at the application layer that does not depend on the network being right.

Four independent controls at four layers. That is what "defence in depth" means concretely, and the argument to make is that no single one of them is sufficient: the network can be misconfigured, the policy can have an exemption, the log can have a gap, and the gateway can be bypassed. Together they are hard to defeat silently.

And the honest sentence that makes the rest credible: "the reachability tool proves the positive — it finds a path. Its negative result is bounded by these modelling assumptions, which are these." An examiner trusts a bounded claim far more than an unbounded one.

10. Setting the numbers

Terraform parallelism. Default 10. Lower it when the provider throttles; the symptom is intermittent apply failures that look like provider bugs.

Drift detection frequency. Weekly for the platform, daily for anything security-relevant. Quarterly for a full unmanaged-resource sweep of the subscription — and that first sweep always finds something.

GPU warm pool. §4.

Node pool max. From quota, not from ambition. A max_nodes above your quota produces a scale-out that fails silently and looks like a scheduling problem.

Admission webhook timeout. 3–5 s, not the 10 s default. A slow webhook adds latency to every object creation, and it shows up during a rollout as inexplicable slowness.

failurePolicy. Fail, with kube-system exempted and at least two replicas across zones. Ignore means the policy silently stops applying, which is the worse failure.

Certificate rotation. 24 h for mesh workload certs (the mesh does it automatically). This is Phase 08's short-lived-credential principle applied to infrastructure.

Image pull. Pre-pull anything over 5 GB into the node image. Below that, a registry cache in the region is enough.

Egress allow-list size. Under 20 FQDNs. If it is 200, it is not a control — and the review that gets it back under 20 is worth doing annually.

11. Migration: bringing an estate under control

Starting state: resources created in the portal, no IaC, no policy, secrets in pipeline variables.

Phase 1 — import, do not recreate. terraform import (or import blocks) brings existing resources under management without touching them. Tedious, and it is the only approach that does not require an outage.

Phase 2 — policy in audit mode. Deploy every policy as audit. The report is your gap list and your business case, and it costs nothing to produce.

Phase 3 — CI checks. tfsec/checkov on every PR. Fast feedback, no enforcement yet, so teams learn the rules before the rules bite.

Phase 4 — enforce the highest-value policies. Restricted data first — private endpoints and public access. One policy at a time, each with a stated deadline for existing violations.

Phase 5 — networking. Private endpoints, DNS zones, egress control. The largest piece of work and the one with the most breakage, so it goes after the process is established rather than before.

Phase 6 — remove stored credentials. Workload identity federation and OIDC in CI. Every removed secret is one that can no longer leak.

Phase 7 — drift detection and the reachability proof. The run-state controls, once there is something stable to detect drift from.

The mistake is starting at Phase 5, because networking is the visible problem. Locking down egress before there is IaC, policy or a paved road produces breakage that nobody can diagnose and a reversal that sets the programme back a year.

12. What I would not build

An abstraction over Azure and AWS primitives. It becomes the intersection of two feature sets, maintained by you, permanently behind both.

A custom scheduler. Volcano and Kueue exist, and scheduling is a research area where the naive implementation deadlocks in ways you will find in production.

A GPU driver installation on custom node images. The GPU Operator exists and driver/toolkit version matrices are a maintenance obligation with no upside.

My own service mesh. Not close.

A "cloud-agnostic" Terraform module. A module with a cloud variable that branches internally serves neither cloud well and is unreadable. Two modules with a shared interface is better and honest.

Secrets in Terraform variables, "temporarily". They land in state, in the plan output, and in CI logs. Every time.

A policy engine. Phase 09's answer applies here too: OPA and Kyverno exist. Your contribution is the policy set.

A network diagram as the residency control. A diagram is a claim. The proof is a tool that searches the topology and returns a path — which is the whole point of this phase, and the thing that survives contact with an examiner.