« Phase 10 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: safety against throughput
• 2. Where the gateway sits
• 3. Sagas versus durable execution versus neither
• 4. Who owns the idempotency key
• 5. Setting the numbers
• 6. The autonomy ladder
• 7. Where the audit log lives
• 8. Onboarding a tool
• 9. Degradation as a product decision
• 10. Testing what you cannot reproduce
• 11. Migration: from direct calls to a gateway
• 12. What I would not build

1. The central tension: safety against throughput

Every control in this phase costs latency, availability or engineering time.

   FAST                                                              SAFE
     │                                                                  │
  direct calls    gateway,        + idempotency   + dual control   + durable
  no gateway      no idempotency  + breaker       + blocking audit   sagas
     │                 │                │               │               │
   +0 ms            +5 ms           +15 ms          +30 ms       +100 ms & a quarter

You cannot sit at one point for everything, and the principal move is the same as in Phase 09: choose the position per class of action, and be able to state the table.

The failure mode of choosing one point for everything runs in both directions: maximum safety everywhere makes reads slow enough that teams route around the gateway, and minimum safety everywhere produces the double-payment incident. The first failure is the more common one, and it is worse, because a bypassed gateway is an unaudited one.

2. Where the gateway sits

The library is disqualified on the argument from the warmup: the gateway exists precisely because the kernel's input is untrusted, so sharing a process removes the property you built it for. It will be proposed, because it is much simpler and the latency is free. The answer is that the isolation is the feature.

The sidecar is a reasonable middle: process isolation, cheap latency. What it gives up is independent scaling and a separate identity — the sidecar generally shares the pod's service account, so an escape into the pod reaches the gateway's credentials.

The separate service is the default for a bank. It gets its own identity, its own deployment cadence, its own on-call, and a network boundary that shows up in a diagram an examiner can read. 20 ms against a 200 ms core-banking call is not the argument people think it is.

Where a product like Azure APIM helps: TLS, WAF, rate limiting, quota, subscription keys, developer portal. Where it does not: idempotency semantics, sagas, side-effect classes, dual control. Use the product for the transport concerns and write the mediation logic yourself — trying to express a saga in APIM policy is a well-documented way to lose a quarter.

3. Sagas versus durable execution versus neither

The honest ordering:

Most "sagas" should not be sagas. A single idempotent call is not a saga. Two steps where the second is a notification is not a saga. Reaching for the pattern early buys complexity for nothing.

A real multi-step flow across systems, with money involved, needs durability. And durability is a year of work — deterministic replay, timers, visibility, versioning of in-flight workflows. Buy it.

The middle ground is a trap. A hand-rolled saga with persistence "we'll add later" is the shape that leaves the bank half-updated after a deploy. If it is worth a saga, it is worth durability; if it is not worth durability, it probably is not a saga.

The real cost of Temporal is not the cluster; it is that workflow code must be deterministic, which is a constraint engineers repeatedly violate (datetime.now(), random, iterating a set) and whose violations only surface during a replay after an incident. Budget for the education.

4. Who owns the idempotency key

Three models, and the wrong one silently disables the entire mechanism.

The caller generates it per attempt. Useless: every retry has a new key, so nothing is ever recognized as a duplicate. This is the most common way idempotency is nominally present and actually absent, and it passes every test because a single call works fine.

The caller generates it per business intent. Correct. The key is derived from what the action is — release:PMT-771 — so any number of attempts to release that payment share a key. Deriving rather than generating is what makes it stable across a process restart, a different pod and a retried agent step.

The gateway generates it from the request hash. Tempting, and subtly wrong: two legitimately distinct actions with identical arguments become one. "Append the note 'called customer' to case C-1" is a thing you may genuinely want to do twice.

The rule I would put in the tool-onboarding checklist: the key must be derived from the business event, and it must survive a process restart. If an engineer cannot say what it is derived from, idempotency is not implemented, whatever the code says.

5. Setting the numbers

Retries. Reads: 3. Idempotent writes: 3. Everything else: 1. Not a preference — for a non-idempotent action, a retry is only safe because of the key, and the key's safety depends on the downstream, which you may not control.

Timeouts. From the downstream's measured p99, not its SLA. Add ~50% headroom, then cap by the caller's deadline. And propagate that deadline, or you spend real capacity on work whose callers are gone.

Breaker threshold. 50% is the default and it is fine. What actually matters is the minimum throughput: set it so the window contains enough calls for the rate to mean something. For an endpoint at 100 rps with a 60 s window, 20 is trivially met; at 0.1 rps, a time-based window is the wrong shape and you should be alerting on absolute errors instead.

Breaker open duration. Long enough for a pod to restart (~30 s), short enough that recovery is noticed quickly. Below 10 s you are effectively hammering; above 60 s you are extending an outage that has ended.

Bulkhead size. Little's law: concurrency = throughput × latency. At 100 rps and 200 ms, you need 20 concurrent. Size at 1.5–2× the steady state, so a latency excursion has room before it starts rejecting.

Dual-control threshold. A business decision, not an engineering one. Get it from whoever owns operational risk, write down who signed it, and test the exact boundary. The number will be questioned in an audit and "we picked 100,000 because it seemed round" is not an answer.

Audit retention. Seven years for UAE/CBUAE. That drives storage cost, schema stability (you will be reading seven-year-old records with today's code) and the encryption-key rotation strategy — which is a much bigger problem than it sounds.

6. The autonomy ladder

The single most useful framing I know for negotiating with risk, because it converts an argument about whether into a plan about when.

Three things make it work.

Promotion is earned with evidence, and the evidence comes from this phase's audit log: N actions at band 2, zero reversals, zero approval rejections, eval suite green throughout. That is a promotion case a risk officer can read.

Demotion is automatic. An incident drops the band immediately, mechanically, no meeting. Which is what makes promotion palatable — the downside is bounded and pre-agreed.

Per-tool, not per-agent. The same agent can be at band 4 for crm.append_note and band 2 for payments.release. An agent-level band forces the most dangerous tool to set the ceiling for everything.

7. Where the audit log lives

Four requirements that fight each other: append-only, queryable, 7-year retention, and regionally resident.

The practical answer is the last: write the authoritative chain to immutable storage and maintain an indexed copy for operational queries. The indexed copy may be rebuilt from the authoritative one, which is a property worth actually testing rather than assuming.

Two constraints that shape everything:

Residency. Under UAE rules, records about UAE customers stay in the UAE (Phase 15). That means regional stores, which means "show me every action by agent A last quarter" is a scatter-gather rather than a query.

Encryption over seven years. Keys rotate; records encrypted with a 2026 key must still be readable in 2033. That means key versioning in the record, an archival key store, and a rotation procedure somebody has actually tested. It is a bigger problem than the chain.

8. Onboarding a tool

The checklist is the control. Ten questions, and half of them are usually answered wrong the first time:

1. Side-effect class? No default. If the answer is "it depends on the arguments", it is two tools.
2. Idempotency key derived from what? If they cannot say, idempotency is not implemented.
3. Does the downstream honour a key? If not, the gateway is the only defence and no retry is safe.
4. Can we query by our key? The reconciliation question. Nearly free now, impossible during an incident.
5. What is the compensation? If none, it is irreversible and it goes last in every saga.
6. What business invariants can a schema not express? There are always some. "None" means nobody asked the business.
7. What is the value limit per action? And per day, and per agent.
8. What does the p99 look like, and what is the timeout? From measurement, not the SLA.
9. What is in the arguments that must be redacted? By key name and by shape.
10. Who owns this tool, and who approves changes to its contract? A named human.

Question 5 is the one that changes designs. "What is the compensation?" frequently gets the answer "there isn't one", which reclassifies the tool and reorders every saga that uses it.

9. Degradation as a product decision

When core banking is down, what does the platform do? This is not an engineering choice.

The choice belongs with the product owner, which is exactly the two-in-a-box conversation from Phase 16. What engineering owns is making the options available and stating their costs honestly.

The mechanism that makes degradation coherent rather than confusing is the link back to Phase 09: when the breaker for a tool is open, remove that tool from capability discovery. The agent then plans without it. That is a degraded platform behaving sensibly; the alternative — the tool visible, every call refused — is an agent looping and burning tokens against a wall.

And one rule: degraded output must be labelled, in the response and in the audit record. An agent that answers from a stale cache without saying so has produced an answer nobody can assess.

10. Testing what you cannot reproduce

The failures in this phase are exactly the ones that do not occur in a test environment.

Fault injection. A downstream stub that fails, is slow, or times out on demand — driven by a header, so it works in a shared environment. Cheap and it finds the missing timeout immediately.

Crash testing. Kill the gateway between reserving a key and storing a response, then retry. Do this deliberately, because it will happen accidentally and you would rather learn the behaviour on a Tuesday.

Chaos, scoped. Latency injection into one dependency in a non-production region, then watch whether the breaker, the bulkhead and the fallback do what the config says. Most teams discover the bulkhead was never wired.

Saga interruption. Restart the process mid-saga. If it is durable, it resumes; if it is not, you have just demonstrated the risk to whoever needs convincing.

Property tests on idempotency. For any sequence of calls with the same key, the effect count is exactly one. Hypothesis will find the ordering you did not think of.

Contract tests against the downstream. Because the invariants encode assumptions about the bank's behaviour, and those assumptions expire silently when a core-banking release changes a validation rule.

The one people skip is crash testing, and it is the one that finds the design gap rather than a bug.

11. Migration: from direct calls to a gateway

Starting state: agents call downstream APIs directly, with shared credentials and per-call-site retry logic.

Phase 1 — the gateway in shadow. Route calls through it; it validates, audits and forwards without enforcing. Free, and it tells you how many calls would have been refused. Expect a number that surprises people.

Phase 2 — enforce contracts for reads. Lowest blast radius. Fix the schema mismatches this surfaces — there will be many, because nobody was checking.

Phase 3 — idempotency for writes. Requires callers to supply keys, which is the step with real client-side work. Do it before enforcement so callers can adopt at their own pace.

Phase 4 — enforce writes, with a fast exception path. The exception path is essential. Teams will hit invariants nobody anticipated, and without a same-day route they will route around the gateway — which is much worse than a loose rule.

Phase 5 — remove the direct network paths. Firewall rules, not policy. Until an agent cannot reach core banking directly, the gateway is a convention.

Phase 6 — retire the shared credentials. The step that pays for the project.

The mistake is starting at Phase 4. A gateway that blocks legitimate work in week one gets a reputation it does not recover from, and its exception list becomes permanent.

12. What I would not build

A durable-execution engine. Temporal exists, it is very good, and the deterministic-replay machinery is a year of work you will get wrong in ways that only appear during incidents.

A rules engine for invariants. Invariants are code. They are tested, reviewed and deployed like code. A DSL for them means a DSL to maintain, and the "business writes the rules" promise does not survive contact with a bank's actual approval process.

A generic retry framework. Retry policy is derived from the side-effect class. That is a dict. A framework configurable per call site reintroduces exactly the per-call-site choice this phase exists to remove.

My own immutable store. Azure immutable blobs, S3 Object Lock and QLDB all exist with compliance attestations you would otherwise have to earn.

A saga engine for two-step flows. Two steps where the second is idempotent is a retry loop.

A universal PII detector. Presidio and the cloud DLP services exist; a regex is the floor, and building the middle is an ML project in disguise.

A second gateway for a "special" downstream. It always starts as "core banking is different". Two gateways means two audit logs, two idempotency stores and two answers to "what did the agent do?", and the second one is always the less careful one.