« Phase 12 · Warmup · Track Overview

Lab 01 — The Integration Fabric

The problem

The agent has decided. Identity, policy and the action gateway have all said yes. Now the decision has to reach a mainframe.

Between here and there:

• The payment must be expressed in ISO 20022, where a missing element is not a validation error but a rejected file, and where a JPY amount parsed with two decimal places is a payment a hundred times too large.
• The rail you choose determines whether the payment can be recalled at all — and that determines whether a human had to approve it.
• The database write and the event publish are two systems, and a crash between them leaves the bank and the platform disagreeing about what happened.
• Downstream will receive the event more than once, because at-least-once is the only delivery guarantee anyone can actually offer.
• Someone will add a field to the event schema, and whether that breaks production depends on which side deploys first.
• And at the end of the day, two independent records have to agree — or somebody has to own the difference.

You build all of it.

What you build

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

• All 144 tests green against your lab.py.
• to_minor("1.15", "USD") == 115 — and to_minor("100", "JPY") == 100.
• Extra decimal precision is refused, not rounded.
• An IBAN with one digit changed fails; a BIC with a digit in the institution code fails.
• A pain.001 with six problems reports all six, sorted, each naming its element.
• The wrong namespace is a rejection, not a silent parse.
• NbOfTxs and CtrlSum disagreeing with the transactions are both caught.
• A duplicate EndToEndId within one file is rejected.
• An instant payment is FINAL immediately; a batch payment before cut-off is REVOCABLE.
• choose_rail always returns a non-empty reason, including when it returns no rail.
• A dual-write crash leaves the row and loses the event.
• An outbox crash before commit leaves neither, and consumes no sequence number.
• A relay crash after publishing republishes — at-least-once, honestly.
• The same key always routes to the same partition, in this process and the next.
• commit cannot rewind; seek can.
• Replaying the whole log five times produces one effect per event.
• A crash before commit re-delivers and does not re-effect.
• Each compatibility mode refuses exactly the right changes, and states its deploy order.
• A refused registration leaves the version count unchanged.
• A product fails on a missing field, a wrong type, a failed rule, or a stale clock.
• reconcile reports mismatches and both directions of absence, sorted by key.

How this maps to the real stack

Honest limits. The parser covers one message and a subset of its elements — real ISO 20022 validation is an XSD plus a market-practice guideline (CBPR+ for cross-border, HVPS+ for high value) that constrains what the schema permits, and the guideline is where most rejections come from. The consumer group has no rebalancing, which is the single richest source of real bugs: a partition moving between consumers mid-batch is how you get duplicate processing and lost commits, and it is exactly what makes the idempotent consumer non-optional. The dedup set is unbounded, so in production it needs a TTL and therefore a decision about how late a duplicate can arrive. The outbox has no polling interval, no batch size and no ordering guarantee across aggregates. And the rails are invented: real cut-offs vary by currency, by day, by correspondent bank, and by whether it is a Friday in the UAE.

Extensions

1. Add rebalancing. Two consumers, one group, partitions reassigned mid-batch. Watch the duplicate processing appear, then confirm the idempotent consumer absorbs it. This is the exercise that makes the design click.
2. Bound the dedup set. Add a TTL, then answer: how late can a duplicate arrive? The answer is the retention of your log, which makes it a capacity decision.
3. Validate against the real XSD. Download the pain.001.001.09 schema and run xmlschema against your fixtures. Then read a CBPR+ guideline and count how many additional rules it imposes.
4. Add pacs.002. The status report that comes back. Now model the full lifecycle: initiated → accepted → rejected → settled, and map each to a Finality.
5. Compacted topics. Add log compaction keyed on the message key, and work out what it means for a consumer that joins late — it sees the latest state, not the history, which changes what your handler can assume.
6. Transactional outbox with CDC. Replace the polling relay with a Debezium-shaped reader over a simulated transaction log, and handle the schema-drift problem it hands you.
7. A data-product lineage graph. Record which products a product derives from, then answer "what breaks if agent.traces is late?" (Phase 15).
8. Reconciliation with tolerance. Real recon allows a small tolerance and ages breaks. Add both, and discover that "how old is the oldest unresolved break" is the metric people actually watch.

Interview / resume bullets

• "Built the platform's payment-initiation path against ISO 20022, with a validator that reports every rejection at once and names the offending element — turning a six-problem file from six days of round trips into one."
• "Represented money as integer minor units with per-currency exponents, which removed an entire class of cross-border defect: JPY has no minor digits and KWD has three, and the two-decimal assumption is a hundred-fold error."
• "Mapped scheme finality onto the platform's side-effect classes, so an agent's autonomy is bounded by the rail it selects — an instant payment requires an approval that a pre-cut-off batch payment does not."
• "Eliminated the dual-write problem with a transactional outbox and idempotent consumers, giving exactly-once effects under at-least-once delivery without any exactly-once delivery claim."
• "Made schema compatibility mode an explicit deploy-order decision rather than a release-day discovery, and enforced it in the registry so an incompatible schema cannot be published."
• "Published the AI platform's traces, evaluations and outcomes as contract-bearing data products with freshness SLOs — applying to ourselves the discipline we asked of every upstream system."