« Phase 10 · Warmup · Track Overview

Lab 01 — The Action Gateway

The problem

An agent has decided to release a held payment for 250,000 AED. Identity says who is asking (Phase 08); policy says they may (Phase 09). This is the layer where the decision becomes money leaving the bank.

Between the decision and the effect, seven things have to be true, and every one of them has a production incident behind it:

1. The arguments are well-formed and the currency matches the debit account.
2. A network timeout on the way back does not turn one payment into two.
3. Two distinct humans approved — neither of them the agent, neither of them the requester.
4. Core banking being sick does not turn into a retry storm that keeps it sick.
5. If step 3 of a five-step process fails, steps 2 and 1 are undone — in that order.
6. No account number or credential appears in any log line.
7. Six months later, an examiner can be shown who authorized this and prove the record was not edited.

You build all seven.

What you build

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

• All 120 tests green against your lab.py.
• validate_schema rejects True for an integer, and returns every error sorted.
• Invariants never run on a structurally invalid payload.
• The same key with the same request returns the stored response and executes once.
• The same key with a different request is a conflict and executes never.
• A conflict is detected before the in-flight check.
• The request hash ignores the trace id and the model version, and is key-order insensitive.
• A write with no idempotency key is refused without touching the downstream.
• An irreversible action is attempted exactly once; a read is retried.
• The breaker opens exactly at the threshold, and not below the minimum throughput.
• Half-open admits one probe; one failed probe re-opens; closing clears the window.
• An open breaker does not consume an idempotency key.
• An agent cannot be its own second approver, nor can the requesting user.
• The dual-control threshold is inclusive.
• A saga failing at step 3 compensates 2 then 1, and never compensates 3.
• A failing compensation is reported as an orphan and does not stop the others.
• Editing a record's content, re-hashing it, or deleting it all fail verification.
• No secret or full account number appears in any audit record.

How this maps to the real stack

Honest limits. The saga is not durable — a process restart loses it, and durability is the entire reason Temporal and Durable Functions exist. The idempotency store is in-memory and single-threaded, so the in-flight case is demonstrated rather than raced; a real one needs a conditional write and must handle the crash-after-write-before-response window. Redaction is regex, so it catches account-shaped digits and misses everything else — and it over-matches too: a ten-digit phone number is redacted as an account, which is a false positive you would tune out with a real DLP engine. Nothing authenticates an approver; approvals=("ahmed",) is a string, and in production that must be a signed assertion from the moment of approval. And the audit chain is tamper-evident only: whoever can write the log can rebuild it, and closing that requires an external anchor this file cannot provide.

Extensions

1. Make the saga durable. Persist step state, then kill the process mid-saga and resume. You will discover that "did step 2 complete?" needs the idempotency store — which is the insight.
2. Race the idempotency store. Two threads, same key. Then fix it with a conditional write (INSERT ... ON CONFLICT DO NOTHING) and confirm exactly one wins.
3. The crash window. Downstream applied it; the gateway died before storing the response. What does the retry see, and how does the caller learn the truth?
4. Signed approvals. Replace the approver strings with signed assertions carrying a timestamp and an audience, verified at the moment of use (Phase 08).
5. External anchoring. Publish the head hash hourly to a store you cannot write to. Now tamper-evident becomes tamper-resistant, and you can say so precisely.
6. Bulkheads. Add a per-downstream concurrency limit so one slow dependency cannot consume every worker. The breaker stops a failing dependency; the bulkhead stops a slow one, and slow is the more common outage.
7. Adaptive concurrency. Replace the fixed breaker thresholds with a gradient-based limiter (Netflix's concurrency-limits) and compare behaviour under a partial brownout.
8. The outbox. Emit an event per audited action transactionally with the action itself (Phase 12), so the audit stream cannot diverge from what happened.

Interview / resume bullets

• "Built the bank's action gateway — the enforcement boundary between agent decisions and the core estate — where every tool declares a side-effect class from which the platform derives its retry, idempotency, approval and audit policy, so no integration author ever chooses whether a payment is safe to retry."
• "Implemented idempotency with all three replay cases plus in-flight detection, so a network timeout on a payment release returns the original reference instead of releasing a second payment."
• "Ran multi-step work as sagas with idempotent compensations executed in reverse, and made a failed compensation a loud, paged orphan rather than a swallowed exception — because there is no third level of undo."
• "Made the audit log hash-chained and verifiable, so 'tamper-evident' is a property the code demonstrates rather than a claim in a control document."
• "Gave the circuit breaker a defined open behaviour — a labelled stale answer — rather than a faster failure, which kept read paths available during a core-banking brownout."