« Phase 10 · Lab 01 · Track Overview

Warmup — The Action Gateway, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: why a gateway at all
• 2. The side-effect class
• 3. Contract enforcement, both halves
• 4. Idempotency
• 5. Exactly-once, honestly
• 6. The crash window
• 7. Sagas
• 8. Compensation is not rollback
• 9. Circuit breakers
• 10. What "open" must do
• 11. Bulkheads, and why slow beats broken
• 12. Dual control
• 13. Human-in-the-loop and the parked task
• 14. Redaction
• 15. The audit record, field by field
• 16. Hash chaining
• 17. Numbers worth carrying
• 18. Interview questions, answered
• 19. References

0. Where this sits

Three phases form the spine of the enforcement path:

Everything before this is an agent deciding. This is where a decision becomes an effect on the bank, and it is the layer the whole track has been building toward.

1. From first principles: why a gateway at all

Start with the uncomfortable fact: the thing choosing the action is probabilistic and its input is attacker-influenced. An LLM decides which tool to call, with which arguments, based on a context window that contains retrieved documents, tool outputs and user text — any of which may have been written by someone who wants a payment released.

That is not an argument against agents. It is an argument about where the controls live.

Consider the alternative, which every platform builds first: the agent runtime holds credentials and calls core banking directly, with the rules expressed in its system prompt. The failure mode is immediate — a control expressed in a prompt is a control that can be argued with, and one prompt injection removes it.

So: the model proposes, the platform disposes. The agent emits a proposal — tool plus arguments. A separate component validates, authorizes, bounds and records it. The agent never holds a credential that works against the bank, and the rules are code.

Now the crucial structural point, which is often treated as tidiness and is not:

The gateway must be a separate process from the kernel.

The kernel executes model-proposed plans. The gateway exists precisely because the kernel's input is untrusted. Put them in one process and one bug — a deserialization flaw, a path traversal, an overly clever tool that can write to memory — removes both. The entire architecture rests on their independence, so that independence must be real: separate process, separate identity, separate deployment, and a network boundary between them.

The name "gateway" is slightly unfortunate, because it suggests a proxy. It is a mediator: it understands the semantics of what is being asked and refuses on grounds a proxy could never evaluate.

2. The side-effect class

Every tool declares one of four classes. This single field derives most of the gateway's behaviour.

Three things about this table are the lesson.

There is no default. A tool that has not declared its class cannot be registered. A default here would be a default retry policy, and both candidates are wrong: the safe default (never retry) makes reads fragile and pushes engineers to declare everything as read; the convenient default (always retry) double-executes payments. Refusing to guess is the only correct behaviour.

Retry policy is derived, not chosen. The integration author does not decide whether a payment is safe to retry — they declare a property of the tool, and the platform decides. This is the whole point. Every production double-payment story is, at root, a call site that chose.

irreversible is a distinct class, not "non-idempotent but worse". The difference is what happens when a call times out. For a non-idempotent write, the idempotency key makes the retry safe — the downstream recognizes it and returns the original result. For an irreversible action, the question "did it happen?" may have no answer available: the payment left the bank, the confirmation did not come back, and there is no query that tells you within the retry window. So the gateway does not retry. A human decides, and that is correct.

3. Contract enforcement, both halves

Half one: the schema. Types, required fields, patterns, enums, bounds. This is what everyone builds, and it is genuinely important — an agent producing amount: "250,000 AED" where an integer is expected should be refused at the boundary, not at core banking.

Two details that matter more than they look:

• Return every error, sorted. A validator that returns the first error makes a caller fix one field per round trip. With a model as the caller, each round trip is a full inference.
• A boolean is not an integer. isinstance(True, int) is True in Python, so the naive check accepts True as an amount. It is the most common validator bug in the language and it is worth a test.

Half two: the business invariants. This is what a schema cannot say:

Both halves are contract enforcement. The reason to insist on the second is negative: the rules the gateway does not check are the rules that live in the agent's prompt — where they are advice to a probabilistic system, and where one injected instruction removes them.

One implementation detail with a real consequence: do not run invariants on a structurally invalid payload. An invariant that reads request.arguments["debit_account"] will raise KeyError when the field is missing, and the caller gets a stack trace instead of "debit_account: required".

4. Idempotency

The single cheapest control in this track, preventing the most expensive incident.

The caller supplies a key. The store maps key → (state, request_hash, response). Four cases:

Each row is a decision worth defending.

Same hash returns the stored response, not a fresh execution and not a bare 200. The caller must receive the same payment reference it would have received the first time, because it may be storing that reference. A bare acknowledgement makes the caller believe the action did not produce a result.

A different hash is a conflict and executes never. This is counter-intuitive: surely a different request should just... run? No. The caller reused a key for a different request, which means the caller has a bug — most likely a key derived from something insufficiently unique. The one thing that must not happen is performing the second action quietly, because if the key generation is broken, so is the caller's model of what it has already done.

Check the conflict before the in-flight state. If an in-flight record with a different hash returns "retry shortly", the caller retries, the first completes, and now the second executes. The conflict must win.

The in-flight case is the one people forget. Two concurrent requests with the same key means the first has not finished. The second must be refused, not queued — a queue turns a double-click into a double payment one second later.

And what the key's hash covers: the tool, the arguments and the principal. Not the trace id and not the model version. A retry of the same business intent from a new trace is the same request, and including the trace would turn every retry into a 409.

5. Exactly-once, honestly

You will be asked for exactly-once semantics. The honest answer:

Exactly-once delivery is impossible. Exactly-once effects are routine, and they are what anybody actually wants.

The impossibility is the two-generals problem. The sender cannot distinguish "the message was lost" from "the response was lost", so it must either retry (risking a duplicate) or not (risking a loss). No protocol removes that, because the ambiguity is in the network, not the code.

What you can have is at-least-once delivery plus idempotent handling:

    at-least-once delivery  +  idempotent effect  =  exactly-once effect

The retry happens. It reaches the downstream. The downstream recognizes the key and returns the original result without re-applying. The effect occurred exactly once, and the delivery count is irrelevant.

This reframing is worth having ready, because it turns an impossible request into a design: stop trying to prevent duplicate delivery and make duplicate delivery harmless.

6. The crash window

The case that separates people who have run one of these from people who have designed one:

    t=0    gateway reserves the key (IN_FLIGHT)
    t=1    gateway calls core banking
    t=2    core banking APPLIES the payment
    t=3    gateway crashes before storing the response
    ---
    t=4    caller retries with the same key

What does the retry see? The record says IN_FLIGHT and always will, because nothing will ever complete it. Three options, and the choice is a real design decision:

The third is correct where possible, and it is why "does this downstream support a query-by-key?" is a question worth asking during integration design rather than during an incident. Where it is not available, the answer is option two plus a reconciliation process — and the honest thing is to say so rather than to claim the problem away.

Note the assumption the first option rests on: the retry is safe only because the downstream is itself keyed. If the gateway's key is the only key in the system, releasing it is a double payment.

7. Sagas

A payment investigation is five steps across four systems. Two-phase commit across a bank's estate is not available — core banking will not enlist in your distributed transaction, and if it would, you would not want a lock held across an agent's thinking time.

So: a saga. Forward steps, each committing locally, each with a compensation.

    place-hold ──► open-case ──► post-refund ──► notify-customer
         │             │              ✗
         │             │              │  post-refund fails
         ◄─────────────◄──────────────┘
      release-hold   close-case        compensations, in REVERSE

Reverse order because dependencies run forwards: step 3 may rely on step 2's effect, so undoing 2 before 3 leaves 3's compensation operating on state that no longer exists.

Compensations must be idempotent and retryable, because they run in exactly the conditions that just broke — a downstream that is flaky, timing out, or half-up. A compensation that can only run once, cleanly, is a compensation that will not run.

And the case that must never be swallowed: a compensation that itself fails. There is no third level of undo. The only correct behaviour is to record it loudly as an orphan and page a human, because the bank is now in a state no code will repair. An except: pass around a compensation is the worst line of code in this phase.

8. Compensation is not rollback

The distinction people skip, and the one that shows seniority.

A rollback restores the previous state as if nothing happened. The intermediate state was never visible; the database guarantees it.

A compensation performs a new business action that undoes the effect. The intermediate state was visible to everyone, the whole time:

• The hold was placed. The customer saw a reduced available balance. The fraud system scored it.
• The reversal appears on the statement as its own line. It has its own reference.
• Somebody may have made a decision based on the intermediate state, and no compensation unmakes that decision.

Practical consequences that follow directly:

1. Design for the intermediate state being visible. If a customer seeing a hold for eight seconds is unacceptable, a saga is the wrong shape and you need a different decomposition.
2. Compensations need their own contracts — they are actions, subject to the same validation, authorization and audit as forward steps.
3. Some steps have no compensation. A released payment. An email. A trade on an exchange. Those steps go last, after everything that could fail — which is the sequencing rule the whole pattern gives you.

9. Circuit breakers

Core banking gets slow. Every agent request queues. Threads fill. The platform becomes unavailable because a dependency is unavailable — and worse, the retries keep it unavailable.

A breaker is a state machine:

                failures exceed the threshold
        CLOSED ─────────────────────────────► OPEN
          ▲                                    │
          │ probes succeed          open_ticks │
          │                                    ▼
          └──────────────────────────── HALF-OPEN
                                       (one probe)
                    a probe fails ──► OPEN

Four parameters, and the two people omit are the two that matter:

Minimum throughput is the one that gets left out. Without it, one failure out of one call is a 100% failure rate, and every low-traffic blip opens the circuit — after which someone raises the threshold until the breaker never fires.

Half-open admits one probe, not all traffic. Full traffic at a recovering downstream re-kills it, and you oscillate.

And an implementation detail that causes a real bug: closing must clear the window. Otherwise the failures that opened it are still in the rolling window, and the first new failure re-opens it immediately.

10. What "open" must do

The question that separates a useful breaker from a decorative one:

When the breaker is open, what happens to the request?

If the answer is "it fails", you have converted a slow failure into a fast one. That is worth something — you stopped the thread exhaustion — but it is much less than it looks, and it is usually where teams stop.

The options, in increasing order of value:

The last one connects to Phase 09: if the breaker for payments.lookup is open, remove it from capability discovery. The agent then plans without it, instead of planning around it and failing. That is the difference between a degraded platform and a broken one.

11. Bulkheads, and why slow beats broken

A breaker handles a failing dependency. It does not handle a slow one — and slow is the more common outage.

A dependency responding in 30 seconds instead of 200 ms is not failing. The breaker sees successes. Meanwhile every worker is blocked on it, and requests to healthy dependencies cannot get a thread. One sick downstream has taken the whole platform.

The bulkhead is a concurrency limit per dependency: at most N in-flight calls to core banking, N to the CRM, N to the model gateway. Past the limit, requests are rejected immediately rather than queued. The name is from ship design — a hull compartment floods, the others do not.

Bulkhead plus timeout plus breaker is the complete set, and they handle three distinct failures:

Teams build the breaker first because it is the famous one. The bulkhead prevents more incidents.

12. Dual control

Four-eyes. Two distinct authenticated humans for actions above a threshold.

Three failure modes it exists to prevent, and all three have happened:

1. One approver clicking twice. Hence a set of approvers, not a list.
2. The agent counting itself. Hence the agent id is excluded.
3. The requesting user approving their own request. Four-eyes with one pair of eyes is not four-eyes. Exclude the initiating user and everyone in the delegation chain.

Two more properties worth stating:

The threshold boundary is inclusive. "Above 100,000" and "100,000 or more" differ by exactly one transaction, and that transaction is the one the auditor picks. Write it as >= and test the exact boundary.

The approver is authenticated at the moment of approval, not at the moment the workflow was created. An approval collected as a string in a payload is not dual control; it is a field. In production each approval is a signed assertion carrying who, when, and what exactly they approved.

13. Human-in-the-loop and the parked task

Where does the pause live? Not in the gateway.

The gateway is synchronous: request in, decision out. A four-hour wait for an approval does not belong in a request handler. The pause belongs in the kernel (Phase 01), as a state in the task's lifecycle, so that:

• the approval lands in the execution chain and is visible in the trace;
• the task can be persisted and resumed on a different pod;
• the credential is re-minted on resume rather than held for four hours (Phase 08);
• and policy is re-evaluated at resume — because the conditions that admitted the task at minute zero may not hold at minute two hundred and forty (Phase 09).

That last point is the one people miss, and it is the correct answer to "what happens to a task parked for four hours?"

14. Redaction

Account numbers, national IDs, credentials, full names — none of them belong in a log line.

The rule that makes it a control rather than a report: redact before serialization, never after. "We'll scrub the logs later" means the unredacted value already left the process, was buffered, shipped to the aggregator and indexed. A scrubbing job downstream is a cleanup, and cleanup is not containment.

Two design points:

Keep the last four digits. An audit record that cannot distinguish two accounts is not much of an audit record. ****3456 is both safe and useful.

Redact by key name and by value shape. Key-based catches password and api_key; shape-based catches an account number that arrived in a free-text note field, which is where it actually shows up.

And the honest limit, worth saying in a review: regex redaction over-matches and under-matches. A ten-digit phone number gets redacted as an account (harmless, slightly annoying); a name or an address sails straight through (not harmless). Production uses a real DLP engine, and the gateway's regex is the floor rather than the answer.

15. The audit record, field by field

Every field answers a specific person's question. If you cannot name the person, drop the field.

Two rules about when to write one:

Refusals are audited exactly as carefully as successes. They are the more interesting half: "the platform stopped it" is the sentence that demonstrates the control worked, and an examiner asking "has this ever been attempted?" needs the denials.

The audit write is not best-effort. If the record cannot be written, the action does not proceed. That is a real availability cost and it is the right trade in a bank — an action nobody can prove happened is worse than an action that did not happen.

16. Hash chaining

Each record's hash covers its own content and the previous record's hash:

    record 1:  hash₁ = H(content₁ ‖ GENESIS)
    record 2:  hash₂ = H(content₂ ‖ hash₁)
    record 3:  hash₃ = H(content₃ ‖ hash₂)

Edit record 2's content and hash₂ no longer matches — detected. Recompute hash₂ and record 3's prev_hash no longer matches — detected. To hide the edit you must rewrite every record after it, all the way to the head.

Which is why the verifier checks three things per record, not one:

1. the sequence number (catches a deletion);
2. prev_hash against the previous record's hash (catches a re-hashed edit);
3. digest() against this_hash (catches a raw content edit).

Check only the third and a careful attacker rewrites the chain. Check only the second and a truncation at the tail passes.

And the honest limit: this is tamper-evident, not tamper-proof. Whoever can write the log can rebuild it. What closes the gap is an external anchor: publish the head hash somewhere you do not control — a WORM store, another team's system, a public timestamp — hourly. Now rewriting history requires also rewriting something outside your blast radius. The code cannot do that for you; it is a deployment decision, and it is the difference between the claim and the property.

17. Numbers worth carrying

18. Interview questions, answered

Q1. "Why does the action gateway exist? Why not let the agent call core banking?"

Because the thing choosing the action is probabilistic and its input is attacker-influenced. The context window contains retrieved documents and tool outputs that someone else may have written, and a control expressed in a system prompt can be argued with.

So the model proposes and the platform disposes. The agent emits a proposal — tool plus arguments — and a separate component validates it against a contract, checks it against policy, bounds it with an idempotency key and a value limit, attaches a short-lived credential, and records it. The agent never holds a credential that works against the bank.

And the part that is structural rather than tidy: the gateway must be a separate process. The kernel executes model-proposed plans; the gateway exists precisely because the kernel's input is untrusted. One process means one bug removes both, and the whole architecture rests on their independence.

Q2. "Walk me through idempotency."

The caller supplies a key; the store maps it to a state, a hash of the request, and the response.

Four cases. No record: reserve, execute, store. Same key and same hash on a completed record: return the stored response, execute zero more times — and it must be the stored response, not a bare 200, because the caller may be storing the payment reference. Same key, different hash: conflict, and it executes never — the caller reused a key for a different request, which means their key generation is broken, and quietly doing the second action is the worst available outcome. Fourth: in flight, which means the first is still running, so refuse rather than queue — a queue turns a double-click into a double payment one second later.

Two details I would check in a review. The conflict check must come before the in-flight check, otherwise a conflicting request gets told "retry shortly" and eventually executes. And the hash must exclude the trace id and model version, or every legitimate retry becomes a 409.

Q3. "I want exactly-once semantics."

Exactly-once delivery is impossible — that is the two-generals problem, and the ambiguity lives in the network rather than the code. Exactly-once effects are routine, and they are what you actually want.

At-least-once delivery plus idempotent handling gives you it. The retry happens, reaches the downstream, the downstream recognizes the key and returns the original result without re-applying. The effect occurred once; the delivery count is irrelevant.

The interesting part is the crash window: we reserved the key, called the downstream, it applied, and we died before storing the response. The retry sees IN_FLIGHT forever. Three options — release on a lease timeout, which is only safe if the downstream is itself keyed; keep it and require a human; or reconcile by querying the downstream, which is correct where a query-by-key exists. Which is why "can I query this by my key?" is a question for integration design, not for the incident.

Q4. "Design a saga for a payment investigation."

Place a hold; open a case; post a refund; notify the customer. Each commits locally, because two- phase commit across a bank's estate is not available and you would not want a lock held across an agent's thinking time.

Each step has a compensation: release the hold, close the case, post a reversal. If step 3 fails, compensate 2 then 1 — reverse order, because step 3 may rely on step 2's effect.

Three properties. Compensations must be idempotent and retryable, because they run in exactly the conditions that just broke. notify-customer goes last and has no compensation, which is the sequencing rule the pattern gives you: uncompensable steps come after everything that could fail. And a compensation that itself fails is an orphan — recorded loudly and paged, never swallowed, because there is no third level of undo and the bank is now in a state no code will repair.

Q5. "Why is compensation not rollback?"

A rollback restores the previous state as if nothing happened; the intermediate state was never visible. A compensation is a new business action that undoes the effect, and the intermediate state was visible the whole time.

Concretely: the hold was placed, the customer saw a reduced balance, the fraud system scored it. The reversal appears on the statement as its own line with its own reference. Somebody may have made a decision based on the intermediate state, and no compensation unmakes that decision.

Which has design consequences. If a customer seeing a hold for eight seconds is unacceptable, a saga is the wrong shape and you need a different decomposition. Compensations are actions, so they need their own contracts, authorization and audit. And some steps have no compensation at all, so they go last.

Q6. "Configure a circuit breaker. What does 'open' do?"

Fifty percent over a rolling sixty-second window, with a minimum throughput of ten. Open for thirty seconds, then half-open admitting one probe, closing after two successes.

The minimum throughput is the parameter people leave out and it is the one that matters: one failure out of one call is a 100% failure rate, so without it every low-traffic blip at 3 a.m. opens the circuit — after which someone raises the threshold until the breaker never fires. Half-open must admit one probe rather than all traffic, or you re-kill a recovering dependency. And closing must clear the window, or the failures that opened it re-open it immediately.

Now the question I think is the actual question: what does open do? If it just fails, I have converted a slow failure into a fast one. Better is a useful error the agent can plan around; better still is a labelled stale answer for reads. Best is degrading the capability — when the breaker for payments.lookup is open, remove it from capability discovery, so the agent plans without it instead of planning around it and failing.

I would also add a bulkhead, because the breaker handles a failing dependency and not a slow one, and slow is the more common outage.

Q7. "What is in an audit record, and how do I know it hasn't been edited?"

Actor chain, tool and side-effect class, redacted arguments, outcome and error, policy version, model version, idempotency key, approvals, value, and the trace id. Each field answers a specific person's question — if I cannot name the person, I drop the field. And refusals are recorded as carefully as successes; they are the more interesting half, because "the platform stopped it" is what demonstrates the control.

For integrity: hash-chained. Each record's hash covers its own content and the previous hash, so editing record 7 breaks its hash, and re-hashing it breaks record 8's prev_hash, and so on to the head. The verifier checks three things per record — the sequence number, so a deletion is caught; prev_hash, so a re-hashed edit is caught; and the content hash, so a raw edit is caught.

And the honest part: that is tamper-evident, not tamper-proof. Whoever can write the log can rebuild it. What closes the gap is publishing the head hash hourly somewhere I do not control, so rewriting history means also rewriting something outside my blast radius.

19. References

Patterns

• Release It! (Nygard) — circuit breakers, bulkheads, timeouts; the origin text
• Microsoft — Saga pattern · Circuit Breaker · Bulkhead
• Sagas (Garcia-Molina & Salem, 1987) — the original paper
• Pat Helland — Life Beyond Distributed Transactions
• Idempotency-Key HTTP header (IETF draft)
• Stripe — idempotent requests

Implementations

• Temporal — durable execution; the production answer to sagas
• Azure Durable Functions
• resilience4j · Polly · Envoy outlier detection
• Netflix concurrency-limits — adaptive bulkheads

Audit and integrity

• RFC 6962 — Certificate Transparency — Merkle logs, the industrial version of hash chaining
• Sigstore Rekor — a transparency log you can read the implementation of
• Azure immutable blob storage (WORM)
• Microsoft Presidio — PII detection and redaction