« Phase 12 · Warmup · Track Overview

Hitchhiker's Guide — The Integration Fabric

The fast orientation. What the pieces are, what they are called, and how they fit — before the deep dive takes them apart.

Table of Contents

• 1. Don't panic: the one-paragraph version
• 2. The map
• 3. The vocabulary
• 4. ISO 20022 in one table
• 5. The rails, memorized
• 6. Kafka and Event Hubs, side by side
• 7. The five things that will surprise you
• 8. Reading a Debezium config
• 9. Where the neighbouring phases connect
• 10. What to build first

1. Don't panic: the one-paragraph version

An agent's decision has to reach a mainframe, which you cannot change, whose availability is not yours, and which speaks a vocabulary you did not choose. So everything goes through a mediation layer: it translates into ISO 20022 (where a missing element is a rejected file and a JPY amount with two decimal places is a 100× error), picks a rail whose finality determines whether a human had to approve, writes the domain change and its outbound event in one transaction so a crash cannot lose the event, publishes to a partitioned log where ordering holds per key and nowhere else, and is consumed idempotently because at-least-once is the only delivery anyone can offer. At the end of the day, two independent records are reconciled, and the differences are findings with owners.

2. The map

   agent ──► action gateway (Phase 10) ──► INTEGRATION FABRIC
                                                  │
   ┌──────────────────────────────────────────────┴──────────────────────────┐
   │                                                                          │
   │  ISO 20022 mediation ──► rail selection ──► core banking / payments      │
   │        │                      │                     │                    │
   │        │                 finality ──────────────────┼──► back to the     │
   │        │                                            │    side-effect     │
   │        │                                            │    class           │
   │  ┌─────▼───────────────┐                            │                    │
   │  │  OUTBOX (one txn)   │◄───────────────────────────┘                    │
   │  └─────┬───────────────┘                                                 │
   │        │ relay (at-least-once)                                           │
   │  ┌─────▼───────────────────────────────────────┐                         │
   │  │  PARTITIONED LOG   Kafka / Event Hubs        │                        │
   │  │  order per key, never global                 │                        │
   │  └─────┬───────────────────────────────────────┘                         │
   │        │                                                                 │
   │  idempotent consumers ──► data products ──► reconciliation               │
   │                                                                          │
   └──────────────────────────────────────────────────────────────────────────┘

3. The vocabulary

4. ISO 20022 in one table

    pain  .  001  .  001  .  09
     │        │       │       └── version
     │        │       └────────── variant
     │        └────────────────── message number
     └───────────────────────────  business area

The last two rows are the finality lesson in miniature: a return is a fresh payment with its own reference, and a cancellation is something you ask for.

5. The rails, memorized

And the trap, because it is easy to code backwards: settlement dominates the cut-off. An instant payment has no cut-off and is final immediately. Checking the cut-off first reports the most irrevocable rail in the bank as revocable.

6. Kafka and Event Hubs, side by side

Event Hubs speaks the Kafka protocol, so a Kafka client can usually point at it unchanged. The differences that bite are compaction (Event Hubs does not have it in the same form) and Connect (you use Azure integrations instead).

7. The five things that will surprise you

1. NbOfTxs and CtrlSum are the truncated-file detector. They look like redundant metadata. They are the only thing that notices a file cut in half by a transfer, because every individual element is still valid.

2. JPY has zero minor digits. Assume two and the payment is 100× too large. KWD has three and goes the other way.

3. The outbox does not stop duplicates. It stops lost events. The relay can crash after publishing, so idempotent consumers are non-optional — the two are halves of one design.

4. Compatibility mode is deploy order. BACKWARD → consumers first. Getting it backwards is a release where nothing can read anything.

5. Reconciliation is how you find out your dual-write bug exists. Monitoring shows both systems healthy, because they are — each is internally consistent and they disagree with each other.

8. Reading a Debezium config

CDC, in the shape you will actually meet it:

{
  "connector.class": "io.debezium.connector.oracle.OracleConnector",
  "database.dbname": "COREBANK",
  "table.include.list": "COREBANK.OUTBOX",          // ← the outbox table, not the domain tables
  "snapshot.mode": "initial",                        // ← the first run reads everything
  "transforms": "outbox",
  "transforms.outbox.type":
      "io.debezium.transforms.outbox.EventRouter",   // ← outbox pattern applied to CDC
  "transforms.outbox.route.by.field": "aggregate_type",
  "transforms.outbox.table.field.event.payload": "payload"
}

Three things to notice:

• table.include.list names the outbox table, not the domain tables. Streaming domain tables directly leaks the physical schema into every consumer — the mistake this config avoids.
• snapshot.mode: initial means the first run reads the entire table. On a large one that is a capacity event, and it needs planning.
• EventRouter is the outbox pattern as a Kafka Connect transform: it unwraps the outbox row into a business event and routes it by aggregate type. This combination — CDC for no-modification, outbox for business semantics — is the standard answer, and it is worth knowing by name.

9. Where the neighbouring phases connect

10. What to build first

1. Money as integer minor units, with the exponent table. One afternoon, and retrofitting it means auditing every arithmetic site in the platform.
2. The anti-corruption layer boundary — one module that owns every translation. Without it the mainframe's field names leak into forty services.
3. The outbox, before the first event is published. Retrofitting it means finding every place that already dual-writes.
4. Idempotent consumers, at the same time. They are the other half.
5. The schema registry with an explicit mode, before the first schema change. Set the mode deliberately and write down the deploy order.
6. ISO 20022 validation at the boundary, before files go to a scheme. Catching a rejection yourself is minutes; catching it from the scheme is a day.
7. Reconciliation, as soon as two systems hold the same fact. It is the only thing that will tell you the previous six are working.
8. Data products, when the first person asks you for a CSV.