« Phase 13 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a new workload
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy every engine, build the paved road and the policy. A scheduler is a research area; a module that produces a compliant namespace in one command is fifteen files nobody else can write for you.

And the trap: building the "cloud-agnostic abstraction layer". It becomes the intersection of two feature sets, permanently behind both, maintained by you. The question that ends the proposal is "which specific workload will move, when, and what is that worth?"

2. A decision framework for a new workload

Ten questions. The last four are the ones usually unanswered:

1. CPU or GPU? If GPU: which type, how many, and is it a gang?
2. What is the data classification? It determines private endpoints, egress and region.
3. Is it stateful? Session affinity, PVCs, and what a rolling restart costs.
4. What does it need to reach? Every destination becomes a private endpoint or a firewall rule.
5. What must reach it? Ingress, or nothing.
6. What identity does it need? One service account per workload, federated.
7. What is the traffic shape? Steady, diurnal, spiky — it determines warm-pool sizing.
8. What happens when it is scaled to zero? For a GPU workload, nine minutes.
9. Which region, and can you prove it stays there? ← the one that becomes a finding
10. Who owns it, and what is the tag? ← the one that becomes an unowned cost

Question 4 is the one that generates the most work and is asked latest. Every destination is a private endpoint (six steps) or a firewall rule (a review), and discovering the list during deployment is how egress lockdown slips a quarter.

3. Review red flags

In a design document

• Resources created in the portal, "we'll import them later".
• One Terraform state for everything.
• No drift detection.
• A private endpoint with no mention of DNS zones or VNet links.
• "We have private endpoints" as the residency answer.
• No mention of public_network_access_enabled.
• Egress described as "we'll lock it down later".
• A wildcard egress rule, or a service tag like Storage used as an allow-list.
• GPU autoscaling described like CPU autoscaling.
• No gang-scheduling story for a tensor-parallel deployment.
• MIG chosen for a 70B model.
• No system node pool.
• Namespaces described as a security boundary for untrusted tenants.
• Mesh retries enabled globally, with no exemption for side-effecting calls.
• failure_mode_allow: true on ext_authz.
• A remote PDP on the request path.
• Stored service-principal secrets in CI.
• Images referenced by tag.
• No admission policy for signing.
• Policies enforced without an audit-mode period.
• A network diagram presented as the residency control.

In code

# Red flag: a refactor with no moved block
# (renaming a resource destroys and recreates it)

# Red flag: no lifecycle guard on something stateful
resource "azurerm_postgresql_flexible_server" "db" { }     # one ForceNew away

# Red flag: ignore_changes = all
lifecycle { ignore_changes = all }                          # drift, by design, forever

# Red flag: a secret in a variable
variable "db_password" {}                                   # lands in state and plan output

# Red flag: a module from a branch
source = "git::https://.../modules.git//ns?ref=main"        # changes under its consumers

# Red flag: a private endpoint with no zone link
resource "azurerm_private_endpoint" "pe" { }
# ...and no azurerm_private_dns_zone_virtual_network_link anywhere

# Red flag: the endpoint without the setting
private_endpoint = true
# public_network_access_enabled left at its default (true)

# Red flag: an NSG deny that never fires
# priority 100: ALLOW * -> *
# priority 200: DENY  * -> Internet

# Red flag: a service tag as an allow-list
destination_address_prefix = "Storage"       # EVERY storage account in the cloud

# Red flag: no limits
resources:
  requests: { cpu: "1" }                     # no limits: starves its neighbours

# Red flag: a mutable tag
image: registry.bank.ae/agent:latest         # verified something else

# Red flag: a GPU pool with no taint
# (every CPU pod may land on a $30/hour node)

# Red flag: mesh retries on everything
retries: { attempts: 3 }                     # including payments.release

# Red flag: fail-open authz
failure_mode_allow: true

# Red flag: a Fail webhook with one replica and no kube-system exemption

In an incident review

• "The cluster was recreated" → a ForceNew attribute, unread plan.
• "A cleanup PR destroyed production" → a refactor with no moved blocks.
• "It works but the traffic is public" → an unlinked DNS zone.
• "Nobody could create anything" → a Fail webhook whose pods were down.
• "The jobs were running but not progressing" → gang scheduling.
• "Scale-out took ten minutes" → reactive autoscaling on GPUs.
• "The payment went twice" → mesh retries.
• "We couldn't prove it to the auditor" → a diagram, not a proof.

4. Production war stories

The cleanup PR. A tidy-up moved resources into modules. No moved blocks. The plan showed 47 to destroy and 47 to create; it was approved because "the count matches". It destroyed the production VNet, and everything in it, on a Thursday afternoon.

Location. A variable default changed from uaenorth to uaecentral in a shared tfvars file. The plan said # forces replacement on the AKS cluster, in one line among sixty. The cluster was destroyed and recreated. Four hours, and the state file was the only record of what had been running.

The DNS zone nobody linked. Private endpoints were created for every service, reviewed, and signed off as the residency control. The private DNS zones existed. They were linked to the hub VNet and not to the spokes. Every call resolved publicly and went over the internet for fourteen months, and it was found when a routine public_network_access_enabled = false broke everything and nobody could work out why.

Both settings. A storage account holding customer documents had a private endpoint. It also had public network access enabled, because that is the default and the endpoint does not change it. It was internet-reachable with a SAS token for two years. The architecture diagram showed a private endpoint, correctly.

Storage. The egress allow-list used the Storage service tag, which reads like "our storage" and means every storage account in the cloud. It was an unmonitored exfiltration path for eighteen months, and it passed three reviews because the rule looked restrictive.

The GPU deadlock. Two tensor-parallel deployments, eight GPUs each, one sixteen-GPU cluster. The default scheduler placed six pods of job A and six of job B. Twelve GPUs held, neither job progressing, four GPUs idle. It looked like a model-loading problem for two days because the pods were Running.

The nine-minute scale-out. HPA on GPU utilization. A demo spike triggered scale-out; nodes were ready nine minutes later, by which time the demo was over. The nodes then scaled back down. The following week the same thing happened during a real incident, and the platform was effectively unavailable for the nine minutes that mattered.

No system pool. An embedding workload with a memory leak filled a node. CoreDNS was evicted. Name resolution failed cluster-wide, so every service looked broken and the actual cause was three layers away. Diagnosed in ninety minutes.

Untainted GPU nodes. A GPU pool without taints. Ordinary CPU workloads scheduled onto $30/hour nodes because they had capacity. Discovered in a cost review: 60% of GPU-node CPU capacity was running web services.

failure_mode_allow: true. Set during a proof of concept "to unblock testing", and shipped. The PDP was restarted during a routine deploy and every request was allowed for ninety seconds. Nothing bad happened, which is why it was not noticed for another four months.

Mesh retries. Global retry policy, three attempts on 5xx. Core banking returned a 503 after applying a payment. Three payments. The mesh was doing exactly what it was configured to do and had no way to know the call was not idempotent.

The webhook deadlock. A Gatekeeper deployment with failurePolicy: Fail, two replicas, both on the same node. The node was drained for maintenance. Nothing could be created — including the Gatekeeper pods, which needed admission to schedule. The cluster was unrecoverable without deleting the webhook configuration by hand.

The CI credential. A service-principal secret in a pipeline variable, marked secret, with contributor on the subscription. A pipeline printed its environment while debugging. The remediation was rotating the credential and reviewing four months of activity logs, and it is exactly what workload identity federation removes.

The unmanaged estate. A subscription sweep found 340 resources in no state file — proofs of concept, an abandoned migration, and a VM with a public IP running an unpatched OS since 2022. All invisible to every process the team had.

5. The interview signal

Signal 1 — reconciliation as the single idea. Terraform and Kubernetes as the same loop at different frequencies, and the consequence: "desired state" is meaningless without a stated reconciliation frequency.

Signal 2 — a plan diffs state, not reality. So an empty plan and a wrong estate are entirely compatible, and drift detection is a separate scheduled job.

Signal 3 — you read for forces replacement. With the specific consequence: for a stateful resource that is data loss, in one line among sixty.

Signal 4 — a failed resource must skip its dependents. Half-built infrastructure plus a state file that disagrees with reality is worse than stopping.

Signal 5 — "unmanaged" is the dangerous drift category. Not drifted, not missing. Destroyed by nothing, audited by nobody.

Signal 6 — a private endpoint does not disable public access. Two settings, two resources. This is the fastest way to tell whether somebody has actually shipped one.

Signal 7 — DNS is what breaks. The zone, the record, and the VNet link — and the failure signature is it works and it should not.

Signal 8 — the nine-minute number, derived. Image pull, drivers, engine warm-up. Followed by: therefore a warm pool, sized from the arrival distribution, at a stated cost.

Signal 9 — gang scheduling, with the deadlock. Six of eight GPUs held, Running, no progress. And the fix is a different scheduler, not a flag.

Signal 10 — MIG's memory ceiling. Seven 10 GB slices are seven places a 70B model does not fit.

Signal 11 — the LLM gateway is an application. Not APIM policy, because token counting needs a tokenizer and semantic caching needs an embedding.

Signal 12 — mesh retries must be off for side-effecting calls. Unprompted. It shows you have held Phase 10 and this phase in one head.

Signal 13 — failure_mode_allow is the fail-open/fail-shut dilemma, and the resolution is a local PDP so the question does not arise.

Signal 14 — a policy that errors must deny. An engine that fails open on its own bug fails open during an incident.

Signal 15 — a counter-example path, not a boolean. "Denied" is an opinion; "denied, and here is the path" is a finding somebody can fix.

Signal 16 — you bound your own claim. "The tool proves the positive. Its negative result is bounded by these modelling assumptions." An examiner trusts a bounded claim more than an unbounded one, and very few candidates offer one.

Anti-signals:

• "We have private endpoints" as the residency answer.
• A network diagram as evidence.
• GPU autoscaling described like CPU autoscaling.
• One Terraform state, unremarked.
• No drift detection.
• Stored CI credentials.
• Images by tag.
• Namespaces as a boundary for untrusted tenants.

The question to ask them: "An examiner asks you to prove no customer data leaves the UAE. What do you show them?" A weak answer is a diagram and a list of private endpoints. A strong one is four independent controls at four layers, a scheduled reachability analysis with retained output, and an explicit statement of what the analysis does not model.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Have them find forces replacement in a sixty-resource plan. Give them a real plan with one buried replacement of a database. Most people miss it. Nobody misses it twice, and it is the cheapest incident prevention available.
2. Break the DNS link and watch it work. Set up a private endpoint, do not link the zone, and show them the traffic going out the public path with everything green. The "it works and it should not" failure signature is not intuitive until you have seen it.
3. Deadlock a GPU cluster. Two gang jobs, a default scheduler, not quite enough GPUs. Watch pods sit in Running making no progress. Then install Volcano and watch it not happen. Twenty minutes, and it converts gang scheduling from a term into a thing.

And the framing for the platform team: this is the phase where the failures are invisible until they are findings. A wrong DNS link works perfectly. A public network access setting works perfectly. An unmanaged resource works perfectly. None of them page anyone, and all of them are found by an auditor or a penetration test rather than by monitoring.

Which is the argument for the reachability proof and the drift check specifically: they are the only controls in this phase that turn an invisible failure into a Tuesday-morning ticket.

The argument that gets it funded is not infrastructure rigour. It is: "today, if somebody forgets to link a DNS zone, our customer data goes over the public internet and everything looks fine. We would find out from an auditor. A scheduled reachability analysis finds it the next morning, and its retained output is the evidence pack we would otherwise assemble under pressure."