« Phase 12 · Lab 01 · Track Overview

Warmup — The Integration Fabric, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: why you cannot just call core banking
• 2. The shapes of bank integration
• 3. ISO 20022: reading the map
• 4. What actually gets a payment rejected
• 5. Money is an integer
• 6. Payment rails
• 7. Finality, and what it means for an agent
• 8. The log
• 9. Ordering, and what it costs
• 10. Delivery semantics
• 11. The dual-write problem
• 12. The transactional outbox
• 13. CDC
• 14. Schema compatibility is deploy order
• 15. Data products
• 16. Reconciliation
• 17. Numbers worth carrying
• 18. Interview questions, answered
• 19. References

0. Where this sits

Phase 10 made the action safe: contract, idempotency, approval, audit. This phase is about what is on the other side of the gateway — the estate the action actually lands in, and the vocabulary it insists on.

The relationship is worth stating precisely, because it determines the boundary between the two phases:

The action gateway owns whether and how an action happens. The integration fabric owns what it looks like when it gets there, and what the bank tells you afterwards.

And one idea flows backwards: finality. The gateway's irreversible side-effect class is not an engineering judgment about difficulty; it is a statement about a payment scheme's rules, and this phase is where that statement comes from.

1. From first principles: why you cannot just call core banking

The naive design is one line: the agent calls the core banking API. Five reasons it does not survive a design review, and they are worth having in order because each one produces a different piece of architecture.

One — you cannot change core banking. It is a system of record with a decade-long change cycle and a blast radius covering the whole bank. There is no version of "we'll add an endpoint for the agent". So the mediation layer absorbs every impedance mismatch, forever.

Two — its availability is not yours. Core banking has scheduled downtime, batch windows, and a capacity envelope sized for known traffic. An agent fleet is unpredictable traffic. Direct coupling means their maintenance window is your outage (Phase 00).

Three — it does not speak your protocol. COBOL copybooks, fixed-width files, MQ, SOAP, an ESB with thirty years of accreted transformations. Every one of those is a translation somebody has to own.

Four — it has no concept of an agent. No per-agent credentials, no tenancy, no rate limiting, and its authorization model assumes a human at a terminal or a batch job with a service account. Everything in Phase 08 and Phase 09 exists because that gap has to be filled outside.

Five — and this is the one people underestimate — it is right. When core banking and your platform disagree, core banking is correct by definition. That single fact makes reconciliation (§16) a first-class part of the design rather than a nightly cron somebody wrote.

2. The shapes of bank integration

The pattern that matters most is the anti-corruption layer (Evans' term, and the right one): a translation boundary where the estate's model is converted into yours and back. Without it, the mainframe's field names, its date format and its account-number conventions leak into every service you build, and you can never change either side independently.

Three sub-patterns worth naming:

Read-through with a cache. The agent asks for a balance; the mediation layer caches it briefly. The design question is not the TTL — it is what the agent is allowed to do with a stale balance, and the answer is "read, not decide".

Write-behind. The agent's write is accepted, queued, applied later. Attractive and mostly wrong for money: the agent gets a success for something that has not happened, and if it fails there is no caller to tell. Acceptable for notes and annotations; not for payments.

Mediated synchronous write. The action gateway calls core banking and waits. This is what payments actually need, and it is why the gateway's idempotency and circuit breaker matter so much.

3. ISO 20022: reading the map

ISO 20022 is not a schema. It is a methodology plus a data dictionary plus a message catalogue, built on a shared business model, and it is replacing the old MT messages across payments worldwide.

Everything starts with reading a name:

    pain  .  001  .  001  .  09
     │        │       │       └── version
     │        │       └────────── variant (almost always 001)
     │        └────────────────── message number in the family
     └───────────────────────────  business area

The business areas you will meet:

And the handful of messages that carry most of the traffic:

Two of those rows are the whole finality lesson in miniature. pacs.004 is a return: money moving back as a fresh payment, with its own reference, visible on the statement. And camt.056 is a request to cancel, which the receiving bank may simply decline.

What "working knowledge of ISO 20022" means in an interview is exactly this: read the name, name the family, know which messages are requests and which are facts. The element-level detail is a lookup, and everyone looks it up.

4. What actually gets a payment rejected

Not the exotic things. The list is short and boring:

Two of these deserve emphasis.

The control sum is a checksum for the file. If a transfer truncated the file, every individual element is still valid and the count and sum are the only things that notice. That is why they exist, and why "we don't populate CtrlSum, it's optional" is a bad answer.

A duplicate EndToEndId within one file is a double payment waiting to happen, because downstream systems use it as the idempotency key (Phase 10). Detecting it at the boundary costs a set.

And the design rule that follows from all of it: collect every rejection, then decide. A parser that raises on the first problem forces one round trip per error. In a bank, a round trip with a corporate customer is a day, so a six-problem file takes six days.

5. Money is an integer

Never a float. The demonstration is one line:

>>> 1.15 * 100
114.99999999999999
>>> int(1.15 * 100)
114                    # one cent short, silently, on every payment
>>> round(2.675, 2)
2.67                   # not 2.68 either

Binary floating point cannot represent 0.1, 0.01 or 1.15 exactly, and money is base 10. So: an integer count of minor units, converted with Decimal, divided only at the last moment for display.

Then the part that catches people on their first cross-border payment: the number of minor units is not always two.

A JPY amount parsed with an assumed exponent of 2 is a payment one hundred times too large. The Gulf currencies go the other way. ISO 4217 carries the exponent; use it.

And one more rule: refuse extra precision, do not round it. 100.001 in AED is not a valid amount, and silently rounding it to 100.00 produces a payment that does not match the invoice — a reconciliation break that recurs daily and takes weeks to trace.

6. Payment rails

A "rail" is a scheme with its own rules, participants, cut-offs and finality. The categories:

Four properties matter for a platform:

Cut-off. After it, the payment goes in tomorrow's batch. An agent proposing a payment at 15:30 must know that "today" is no longer available, and saying so is part of a correct answer to the user.

Settlement time. How long until the money is actually somewhere else — and therefore how long the reversal window is.

Limits. Instant schemes cap per-transaction value. Above the cap you fall to RTGS, which has a cut-off the instant scheme did not.

Recall. Some schemes have a recall mechanism; most are a request the beneficiary bank may refuse. None are an undo.

Which produces a useful behaviour for an agent: the rail choice must explain itself. "Selected RTGS after ruling out instant (above its limit)" is an explanation a human can check, and the rail determines whether that human had to approve at all.

7. Finality, and what it means for an agent

Finality is the moment a payment becomes irrevocable. It is a scheme rule — often a legal one — not a database property, and no amount of engineering changes it.

Three states:

And here is the ordering trap worth knowing, because it is easy to code backwards: settlement dominates the cut-off. An instant payment has no cut-off and is final immediately — it settles in seconds. Checking the cut-off first would report the most irrevocable rail in the bank as revocable.

Now the link back to Phase 10, which is the point of this section:

    scheme finality  →  side-effect class  →  retry policy + approval requirement

• Revocable → write_non_idempotent; a compensation exists (cancel before cut-off).
• Conditionally revocable → irreversible in practice; the compensation may be refused, so you cannot rely on it.
• Final → irreversible; there is no compensation, only a new payment in the other direction.

So "may this agent send a payment autonomously?" has a rail-dependent answer. An instant payment is final on submission and needs a human. A pre-cut-off batch payment has hours of revocability and may not. Making that distinction is what a good design does, and collapsing it into "payments need approval" is what a merely-safe design does.

8. The log

Kafka and Azure Event Hubs are the same model with different words:

The model in four sentences: a topic is split into partitions; each partition is an ordered, append-only sequence; a producer's key determines the partition; a consumer group tracks an offset per partition.

Three consequences that follow directly:

The log is durable and replayable. Consuming does not delete. A new consumer can start from the beginning, which is what makes event streaming a substrate rather than a queue.

Parallelism is bounded by partitions. Ten partitions, at most ten consumers doing useful work in one group. Partition count is therefore a capacity decision made early and awkward to change later.

The key is the ordering unit. Which is the next section, and it is the important one.

9. Ordering, and what it costs

Order is guaranteed within a partition. Never across.

This is the single most important property, and the one most often assumed away.

Key by account and every event for that account lands in one partition, so per-account order holds: debit before credit, open before close. Global order across all accounts does not exist, and nothing in the system knows which of two events in different partitions happened first.

That is not a limitation; it is the trade that buys the parallelism. One partition gives you total order and one consumer. Ten partitions give you ten consumers and no total order. Choosing is choosing.

Two subtleties worth carrying:

Partitioning must be derived, not hash(). Python salts string hashing per process, so hash() routes the same key to different partitions after a restart — and silently breaks the one guarantee the model provides. Use a stable digest.

Collisions create a false guarantee. Two accounts that hash to the same partition get a total order between them that they never asked for. Fine — until somebody notices and relies on it, and then you repartition.

10. Delivery semantics

The reframing is the whole section:

    at-least-once delivery  +  idempotent effect  =  exactly-once effect

The duplicate arrives. The consumer recognizes it and does nothing. The effect happened once, and the delivery count is irrelevant.

Which makes idempotency non-negotiable rather than nice to have, and gives you a concrete rule: derive the dedup key from the message's content or a business key — never from the offset. A replay under a different partition assignment produces different offsets for the same event.

(Kafka does have "exactly-once semantics" via transactions and an idempotent producer. Read the scope carefully: it is exactly-once within Kafka — consume, transform, produce. The moment your effect is a payment in core banking, you are back to idempotent handling, and the transaction does not help you.)

11. The dual-write problem

You update the database and publish an event. Two systems, no shared transaction.

    write DB ──► ✗ crash ──► publish        the event is LOST
    publish  ──► ✗ crash ──► write DB       the event is a PHANTOM

Neither ordering works, and there is no third ordering. It is not a bug you can be careful about; it is a structural property of writing to two systems.

The failures are different and both are bad. A lost event means core banking made a payment and nothing downstream knows — the platform's own record is missing it, and nobody notices until a reconciliation. A phantom event is worse: downstream reacts to a payment that does not exist, notifies a customer, updates a ledger, and the correction is a manual mess.

12. The transactional outbox

The fix, and it is deliberately boring:

    BEGIN
      UPDATE payments SET status = 'RELEASED' WHERE id = ...
      INSERT INTO outbox (aggregate_id, event_type, payload) VALUES (...)
    COMMIT

    -- a separate relay:
    SELECT * FROM outbox WHERE published_at IS NULL ORDER BY seq
    -- publish, then mark

Both writes go to the same database, so its own atomicity covers them. Either both happened or neither did. The relay then publishes and marks — and here is the honest part: it can crash after publishing and before marking, so it will publish again.

The outbox promises no LOST events. It does not promise no duplicates.

Which is exactly why §10 comes first. The outbox and the idempotent consumer are two halves of one design, and either alone is insufficient.

Three implementation notes:

Order by sequence. The relay publishes in insertion order, which preserves per-aggregate ordering into the log.

A failed transaction must not consume a sequence number. A gap in the sequence is indistinguishable from a lost record, and someone will spend a day on it.

Prune the table. It grows forever otherwise. Delete published rows older than your replay window.

13. CDC

Change data capture: derive an event stream from the database's transaction log rather than from application code.

    core banking DB ──► transaction log ──► Debezium ──► Kafka

Why it matters here: it does not require modifying the source system. Which, per §1, is the binding constraint on integrating with core banking. CDC is often the only way a thirty-year-old system becomes an event source.

What you get: every change, in commit order, with before and after images, and no application code.

What you also get, and must plan for:

The mitigation is the outbox pattern applied to CDC: the source application writes business events to an outbox table, and CDC streams that table. Now you get CDC's no-modification property and outbox's business semantics. It is the standard combination and it is worth knowing by name.

14. Schema compatibility is deploy order

The section people skip and then learn during a release.

Reason it out rather than memorizing. Under BACKWARD, the new schema can read old data — so you upgrade the readers first, and they handle both the old data still in the log and the new data that arrives later. Under FORWARD, old readers cope with new data, so the producers can move first.

What each permits:

The practical advice that falls out: always give a new field a default. It makes the change compatible in both directions, and it costs one keyword.

BACKWARD is the usual default because consumers usually outnumber producers and are usually harder to coordinate. And the transitive variants (BACKWARD_TRANSITIVE) check against all previous versions rather than just the last — which matters when a consumer might be reading a year of retained history.

15. Data products

A data product is a dataset with a contract. Five parts, and dropping any one turns it back into a table:

The freshness SLO is what makes it a product: a promise with a number, which can be breached, which means somebody can be told. Without it there is no difference between "the pipeline is fine" and "the pipeline stopped on Tuesday".

The AI platform sits on both sides:

As a consumer — customer data, transactions, reference data, risk. If those have no contracts, your agents break silently when an upstream team changes a column, and the failure looks like the model getting worse.

As a producer — and this is the part that gets forgotten. The platform should publish:

with the same discipline you demand of your upstreams. It is also the cheapest way to make the platform's value legible to people who will never read a dashboard.

16. Reconciliation

The bank habit worth stealing. Two independent records of the same reality, compared on a schedule, with a process for the differences.

A difference is a break, and the word matters: it is a finding with an owner and a due date, not an exception somebody swallows.

    core banking (system of record)     the platform's own log
        PMT-1: 1,250,000                    PMT-1: 1,250,000     ✓
        PMT-2:   600,000                    PMT-2:   605,000     value mismatch
        PMT-4:   300,000                    —                    missing in B  (lost event)
        —                                   PMT-3:   100,000     missing in A  (phantom)

Read the last two rows against §11: a break in one direction is a lost event, the other is a phantom. Reconciliation is how you find out that your dual-write bug exists, and it is the only mechanism that does — monitoring shows both systems healthy, because they are.

Three properties of a real recon:

Independent sources. Comparing a system to its own cache proves nothing.

Scheduled and complete, not sampled. Daily is the norm.

A break process. Owner, due date, ageing. The metric people actually watch is the age of the oldest unresolved break, not the count — a hundred fresh breaks is a bad day, one six-month-old break is a finding.

For an AI platform this generalizes usefully: reconcile the action log against core banking, the cost records against the provider invoice, and the agent registry against what is actually running.

17. Numbers worth carrying

18. Interview questions, answered

Q1. "How do you integrate an AI platform with core banking?"

Never directly. Core banking is a system of record with a decade-long change cycle, its own availability envelope and no concept of an agent, so everything goes through a mediation layer — an anti-corruption layer that translates between the estate's model and ours.

Reads go through an API gateway with a short cache, and the design question there is not the TTL, it is what the agent may do with a stale balance: read, not decide. Writes go through the action gateway synchronously, which is why its idempotency and circuit breaker matter. Anything bulk stays a file, because that is what the estate is built for.

And the fifth reason, which I think is the interesting one: core banking is right. When it and the platform disagree, it wins by definition — which makes reconciliation a first-class part of the design rather than a nightly cron.

Q2. "What do you know about ISO 20022?"

It is a methodology and a message catalogue on a shared business model, replacing the MT messages across payments worldwide.

The practical knowledge is reading the name. pain.001.001.09 — pain is payments initiation, customer to bank; pacs is clearing and settlement, bank to bank; camt is cash management, statements and investigations. pain.001 is a credit transfer initiation, pacs.008 is the interbank leg, camt.053 is the statement.

Two messages I would call out because they carry the finality lesson: pacs.004 is a return, which is a new payment with its own reference, not an undo. And camt.056 is a cancellation request — the receiving bank may decline it.

What actually gets messages rejected is boring: a missing mandatory element, a missing Ccy attribute on an amount, an IBAN that fails mod-97, and NbOfTxs/CtrlSum disagreeing with the file — which is the truncated-file detector and the reason "CtrlSum is optional" is a bad answer.

Q3. "How do you represent money?"

An integer count of minor units, parsed with Decimal, divided only at the last moment for display. Never a float — int(1.15 * 100) is 114, silently, on every payment.

And the exponent is not always two. JPY and KRW have zero minor digits; KWD, BHD and OMR have three. A JPY amount parsed with an assumed exponent of two is a payment a hundred times too large, and it is the classic first cross-border bug. ISO 4217 carries the exponent.

One more rule: refuse extra precision rather than rounding it. 100.001 AED is not a valid amount, and rounding it silently produces a payment that does not match the invoice — a reconciliation break that recurs daily.

Q4. "Explain the dual-write problem."

You update the database and publish an event. Two systems, no shared transaction, so a crash between them leaves them disagreeing — and there is no ordering that fixes it. Database first loses the event; broker first invents a phantom one. The phantom is worse, because downstream acts on a payment that does not exist.

The fix is the transactional outbox: write the domain change and the event to the same database in one transaction, so its own atomicity covers both, and have a separate relay publish from the outbox table afterwards.

The relay can crash after publishing and before marking, so it will republish. That is the honest guarantee — the outbox promises no lost events, not no duplicates — and it is why the consumer must be idempotent. The two are halves of one design.

Q5. "Guarantee me exactly-once delivery."

I can't — that is two-generals, and the ambiguity lives in the network rather than the code. What I can give you is exactly-once effects, which is what you actually want.

At-least-once delivery plus idempotent handling. The duplicate arrives, the consumer recognizes it from a dedup key, and does nothing. The effect happened once.

The dedup key has to come from the message's content or a business key — never the offset, because a replay under a different partition assignment produces different offsets for the same event. And the key's TTL has to exceed the log retention, or a late duplicate slips through.

Kafka does have exactly-once semantics via transactions, and it is worth being precise about the scope: it is exactly-once within Kafka — consume, transform, produce. The moment the effect is a payment in core banking, you are back to idempotent handling.

Q6. "What ordering guarantees does Kafka give you?"

Order within a partition, never across. Nothing in the system knows which of two events in different partitions happened first.

Which makes the key the ordering unit: key by account and every event for that account lands in one partition, so per-account order holds even though global order does not. That is not a limitation — it is the trade that buys the parallelism. One partition gives total order and one consumer; ten partitions give ten consumers and no total order.

Two details I would check in a review. Partitioning must use a stable digest, not hash(), because Python salts string hashing per process and the same key would route differently after a restart — silently breaking the one guarantee the model provides. And hash collisions create a false guarantee: two accounts sharing a partition get a total order they never asked for, which is fine until someone relies on it and you repartition.

Q7. "You need to add a field to an event schema."

The question is which compatibility mode the subject is under, because the mode is the deploy order.

BACKWARD means a new reader can read old data, so consumers deploy first. FORWARD means an old reader can read new data, so producers deploy first. Getting it backwards produces a release where the consumers cannot read what the producers are writing, and you find out in production because staging deployed everything at once.

For the field itself: give it a default. A new required field with a default is compatible in both directions, and it costs one keyword. Without one it breaks BACKWARD, because old data does not have it and there is nothing to fall back on.

I would also make the registry enforce it rather than documenting it, and make sure a refused registration does not create a version — otherwise the version numbers lie about what was ever live.

Q8. "What is a data product, and does the AI platform produce any?"

A dataset with a contract: schema, semantics, quality rules, a freshness SLO, and a named human owner. The freshness SLO is what makes it a product rather than a table — it is a promise with a number, so it can be breached, so someone can be told. Without it there is no difference between "the pipeline is fine" and "the pipeline stopped on Tuesday".

And yes, and it is the part that gets forgotten. The platform should publish agent.traces, agent.evaluations, agent.outcomes, agent.decisions and agent.costs, with the same discipline we demand of our upstreams. SRE, model risk, audit and finance all need them, and publishing them as products is the cheapest way to make the platform's behaviour legible to people who will never open a dashboard.

19. References

ISO 20022

• ISO 20022 official site · message catalogue
• SWIFT — ISO 20022 programme · CBPR+
• ISO 4217 currency codes and minor units
• ISO 13616 (IBAN) · ISO 9362 (BIC)

Payments

• CBUAE — payment systems
• BIS CPMI — Principles for Financial Market Infrastructures — where "finality" is defined
• SWIFT gpi

Event streaming

• Apache Kafka documentation · design
• Azure Event Hubs · Kafka mapping
• Confluent Schema Registry — compatibility
• Kafka exactly-once semantics — read the scope carefully

Patterns

• microservices.io — Transactional Outbox · Saga
• Debezium — outbox event router
• Designing Data-Intensive Applications (Kleppmann) — chapters 11 and 12
• Domain-Driven Design (Evans) — the anti-corruption layer

Data products

• Data Mesh (Dehghani)
• Open Data Contract Standard
• Great Expectations · Soda