« Phase 12 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. Validation order, and why it matters
• 2. The checksums
• 3. Decimal arithmetic, precisely
• 4. Market practice: the layer above the schema
• 5. The payment lifecycle
• 6. Partitioning mechanics
• 7. Rebalancing
• 8. Offset commit strategies
• 9. The dedup table
• 10. Outbox mechanics
• 11. CDC and the schema-drift problem
• 12. Compatibility, transitively
• 13. Log compaction
• 14. Backpressure and lag
• 15. Performance
• 16. Failure modes

1. Validation order, and why it matters

The parser's structure is not arbitrary:

   1. XML well-formed?          → FF01, and stop
   2. namespace correct?        → FF02, and stop
   3. group header present?     → MS01, and stop
   4. per-element checks        → collect
   5. per-transaction checks    → collect
   6. CROSS-FIELD checks        → collect  (NbOfTxs, CtrlSum, duplicate E2E)
   7. sort, return everything

Steps 1–3 stop, because nothing downstream is meaningful without them — running per-element checks on a document in the wrong namespace produces a hundred spurious "element absent" rejections.

Steps 4–6 collect, because the sender needs every problem at once.

Step 2 deserves its own note. A namespace mismatch is a version mismatch, and treating it as a warning is a real trap: pain.001.001.03 and .09 share most element names and differ in semantics — structured addresses became mandatory, some elements moved. A parser that shrugs at the namespace will parse a .03 document into a .09 structure and produce a payment that validates and is wrong.

Step 6 is what a per-element schema cannot do. NbOfTxs and CtrlSum are redundant by design: they are a checksum over the file. A transfer that truncates the file leaves every remaining element valid, and only the count and sum notice.

And the duplicate EndToEndId check: downstream systems use that field as an idempotency key (Phase 10), so two transactions sharing one is a request to pay twice — or to pay once and silently drop the other, depending on whose deduplication wins.

2. The checksums

IBAN, mod-97 (ISO 13616):

   AE070331234567890123456
   → move the first 4 characters to the end:  0331234567890123456 AE07
   → letters to digits, A=10 … Z=35:          0331234567890123456 1014 07
   → int(...) % 97 must equal 1

A random string of the right shape passes with probability ~1%. Every single-character error is caught, and so is almost every transposition.

BIC (ISO 9362): 4 letters (institution) + 2 letters (country) + 2 alphanumerics (location), and optionally 3 more (branch). The institution code is letters only — accepting digits there is the common mistake, and it lets through a whole class of typo.

Luhn appears here too, on card numbers, and it is the same check as Phase 11. Banks reuse their checksums, which is a small mercy.

The general point: checksums are how a boundary rejects a typo without a round trip to a directory. They are cheap, they are local, and they catch the errors humans actually make.

3. Decimal arithmetic, precisely

Why floats fail:

>>> from decimal import Decimal
>>> 0.1 + 0.2
0.30000000000000004
>>> 1.15 * 100
114.99999999999999
>>> int(1.15 * 100)
114
>>> round(2.675, 2)
2.67
>>> Decimal("1.15").scaleb(2)
Decimal('115')

IEEE 754 binary cannot represent 0.1, 0.01 or 1.15 exactly. Every arithmetic operation compounds the error, and int() truncates the wrong way.

The rule: Decimal for parsing, int minor units for storage and arithmetic, divide last for display.

def to_minor(amount: str, currency: str) -> int:
    value = Decimal(amount)                     # exact, from the string
    scaled = value.scaleb(currency_exponent(currency))
    if scaled != scaled.to_integral_value():
        raise ValueError("more precision than the currency permits")
    return int(scaled)

Two design decisions in that function:

Decimal(amount) from the string. Decimal(1.15) inherits the float's error; only the string constructor is exact. This is the mistake that survives a code review because the type is right.

Refuse extra precision; do not round. 100.001 AED is invalid. Rounding it silently produces a payment that does not match the invoice, which is a daily reconciliation break and takes weeks to trace because the code is doing exactly what it says.

And the exponent table, which is the part people do not know exists:

Note that the 3-digit currencies are mostly Gulf, which makes them exactly the ones a UAE bank meets daily.

4. Market practice: the layer above the schema

The thing that surprises people who have only read the XSD: most rejections come from a rule the schema does not contain.

ISO 20022 defines a superset. Each scheme then publishes a market practice guideline that narrows it:

What they add:

• Elements the schema marks optional become mandatory (structured creditor address, purpose codes, LEI for certain party types).
• Character sets are restricted — the SWIFT x character set excludes much of Latin-1, so an accented name in a party field is a rejection.
• Field lengths are shortened.
• Codes are constrained to a subset.
• Sometimes the same element means something narrower.

So "we validate against the XSD" is a partial answer, and the correct one names the guideline. The practical consequence for a platform: your validator has two layers — schema, then profile — and the profile is versioned separately and changes on the scheme's calendar, not yours.

The current large example: the migration from unstructured to structured addresses, phased across schemes, where a free-text AdrLine that worked last year is now a rejection.

5. The payment lifecycle

   pain.001 ──► [your bank] ──► pacs.008 ──► [their bank] ──► credited
      │                             │
      └──► pain.002 (status)        └──► pacs.002 (status)

   went wrong?
      camt.056 (cancellation REQUEST) ──► may be refused
      pacs.004 (return) ──────────────► a NEW payment, own reference

The states, and what each means for an agent:

Two things to internalize:

A return is a new payment. It has its own EndToEndId, its own charges, its own settlement, and it appears on the customer's statement as a separate line. The original is not erased. This is exactly the Phase 10 compensation-is-not-rollback point, in its native habitat.

Status messages are asynchronous and can be late. pacs.002 may arrive minutes or hours later, so a platform that treats "no rejection yet" as "success" will report success for payments that are about to fail. The state machine needs a pending state and a timeout, and the timeout is a business decision.

6. Partitioning mechanics

partition = digest(key) % partition_count

Three consequences:

The digest must be stable across processes. Python's hash() is salted per process (PYTHONHASHSEED), so it routes the same key differently after a restart — breaking the one guarantee the model provides, silently, and only for keys that happen to move. Use blake2b, murmur, or Kafka's own CRC32-based default.

Changing the partition count re-routes everything. hash(k) % 4 and hash(k) % 8 disagree for most keys, so adding partitions breaks per-key ordering for the transition period: old events for a key are in the old partition, new ones in the new one, and a consumer reading both has no ordering between them. Which is why partition count is a decision made early and changed with a migration.

Collisions create a false guarantee. Two accounts sharing a partition get a total order between them that they never asked for. Harmless until someone relies on it, then you repartition and it disappears.

And the trade-off between skew and parallelism:

The tenant row is the one that bites in a bank: keying by tenant gives you clean isolation and one enormous partition for the wholesale business.

7. Rebalancing

The mechanism the lab deliberately omits, and the richest source of real bugs.

When a consumer joins or leaves a group, partitions are reassigned:

   before:  C1 → [p0, p1]   C2 → [p2, p3]
   C3 joins
   after:   C1 → [p0]       C2 → [p2]       C3 → [p1, p3]

What goes wrong:

Duplicate processing. C1 processed p1's message 47 and had not committed when the rebalance took p1 away. C3 starts from 47 and processes it again. This is the case the idempotent consumer exists for, and it happens on every deploy.

Stop-the-world. Classic (eager) rebalancing revokes all partitions from all consumers, then reassigns. Every consumer stops. On a large group this is seconds of complete pause, on every scale event.

Cooperative rebalancing (incremental, CooperativeStickyAssignor) moves only the partitions that need to move, so unaffected consumers keep working. It is strictly better and it is opt-in.

Rebalance storms. A consumer that takes longer than max.poll.interval.ms to process a batch is declared dead, which triggers a rebalance, which slows everyone, which triggers another. The fix is smaller batches or a longer interval — and the diagnosis is that the consumer lag graph looks like a sawtooth with no traffic change.

Static membership (group.instance.id) avoids a rebalance when a consumer restarts within a timeout, which turns a rolling deploy from N rebalances into zero.

The practical checklist: cooperative assignor, static membership, max.poll.records tuned to your handler's speed, and idempotent processing so none of it can corrupt anything.

8. Offset commit strategies

Auto-commit deserves a warning, because it is the default and it is not what people think: it commits on a timer, including offsets for messages the handler has not finished. A crash then skips them. It looks like at-least-once and behaves like at-most-once under exactly the conditions you care about.

Commit after processing. Then a crash re-delivers, and the idempotent consumer absorbs it.

The batching question: committing after every message is slow (a network round trip each); committing once per batch is fast and re-delivers the whole batch on a crash. Since the consumer is idempotent, batch commits are correct and the only cost is redundant work. Commit per batch.

And the off-by-one: the committed offset is the next one to read, not the last one read. Getting it wrong means either re-reading the final message forever or skipping it, and which you get depends on a convention nobody wrote down.

9. The dedup table

CREATE TABLE processed_events (
    event_id     TEXT PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON processed_events (processed_at);   -- for the pruner

Four design decisions:

Where the key comes from. The message's business key or a content digest. Never the offset — a replay under a different partition assignment yields different offsets for the same event, and the dedup silently stops working exactly when a rebalance makes it necessary.

The TTL. Must exceed the log retention, or a message replayed from the start of a 7-day log is not recognized. So: TTL >= retention, which makes it a capacity decision — a week of event ids at your volume.

Insert-or-skip, atomically.

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT DO NOTHING RETURNING event_id;

Rows returned means we won and should process. Check-then-act loses under concurrency, exactly as in Phase 10.

Same transaction as the effect, where possible. If the effect is a row in the same database, write the effect and the dedup record together and you have genuine exactly-once for that consumer. If the effect is in core banking, you cannot, and you are back to the idempotency key travelling with the request.

10. Outbox mechanics

CREATE TABLE outbox (
    seq          BIGSERIAL PRIMARY KEY,
    aggregate_id TEXT NOT NULL,
    event_type   TEXT NOT NULL,
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);
CREATE INDEX ON outbox (seq) WHERE published_at IS NULL;   -- partial: only the tail

The relay:

SELECT * FROM outbox WHERE published_at IS NULL ORDER BY seq LIMIT 100
FOR UPDATE SKIP LOCKED;              -- so two relay instances do not collide

Four details:

FOR UPDATE SKIP LOCKED lets you run more than one relay without duplicate publishing — each takes a disjoint batch. Without it, two relays publish the same rows.

Order by seq preserves per-aggregate ordering into the log. Note the guarantee is only per-aggregate once it lands, because the log partitions by key.

The partial index keeps the relay's query fast as the table grows. A full index on seq gets slower forever; a partial index covers only the unpublished tail.

Prune published rows. The table grows without bound otherwise. Delete beyond the replay window, and keep the window longer than your longest plausible incident.

The gap-in-sequence rule from the lab is worth restating: a failed transaction must not consume a sequence number, because a gap is indistinguishable from a lost record and someone will spend a day proving it was not one. (Note that BIGSERIAL does consume on rollback — which is why a real implementation either accepts gaps and documents it, or uses a different sequencing strategy.)

11. CDC and the schema-drift problem

   core banking DB ──► transaction log ──► Debezium ──► Kafka

CDC's virtue is that it requires no change to the source. Its cost is that you are now consuming the physical schema.

The outbox pattern applied to CDC is the standard mitigation and the one worth naming: the source application writes business events to an outbox table; CDC streams only that table. You keep no-modification-of-the-consumer-facing-schema and gain business semantics.

That requires the source application to be modifiable — which, for core banking, it may not be. When it is not, the honest answer is a translation service that consumes the raw CDC stream and publishes business events, owning the drift problem in one place instead of in every consumer. That service is your anti-corruption layer, and it will be modified every time the DBA does anything.

Two operational facts:

The initial snapshot reads the whole table. On a large one, that is a capacity event needing planning — and Debezium's incremental snapshot exists precisely because the blocking one was unusable.

CDC captures everything. Including the batch job that touches ten million rows at 2 a.m. Filter at the connector, not at the consumer.

12. Compatibility, transitively

Why transitive matters, in one example:

   v1: {id, amount}
   v2: {id, amount, currency?}      backward-compatible with v1
   v3: {id, currency?}              backward-compatible with v2  ← amount removed

Each step passes BACKWARD. But a v3 reader cannot read v1 data meaningfully — amount is gone. If your retention is 7 days and you shipped v2 and v3 in one week, a consumer replaying from the start of the log breaks.

So: if consumers replay history, use the transitive mode. If they only ever read the tail, non-transitive is fine and cheaper to live with. That is a genuine choice, and it should be recorded next to the retention setting because the two are coupled.

13. Log compaction

Retention by key instead of by time: keep the latest value for each key, forever.

   before:  (A,1) (B,1) (A,2) (C,1) (A,3) (B,2)
   after:   (C,1) (A,3) (B,2)

Useful for state that a late consumer needs in full — a reference-data table, a customer directory, the agent registry. A new consumer reads the compacted topic and has the current state without replaying a year.

What it changes for a consumer, and this is the part that surprises people:

You no longer see history. A consumer that joins late sees (A,3) and never knows about (A,1) or (A,2). If your handler computes a delta, it is now wrong.

A tombstone (a null value) means delete. A compacted topic's delete is a message, and a consumer that ignores nulls keeps the deleted key forever.

Compaction is asynchronous. Duplicates are visible until the compactor runs, so the handler must be idempotent anyway.

So the rule: compacted topics are for state, not for events. A payment-released event stream must not be compacted; a customer-reference topic should be.

14. Backpressure and lag

Lag = high-water mark − committed offset. The single most useful streaming metric, and the one to alert on.

Alert on lag in time, not lag in messages. "40,000 messages behind" means nothing without a rate; "12 minutes behind" is immediately actionable and comparable across topics.

And the property that makes the log a substrate: it absorbs backpressure. A slow consumer does not slow the producer — it just falls behind, and catches up later. Which is the whole architectural argument for streaming between an AI platform and a bank estate whose throughput you do not control. The failure mode is not overload; it is retention. If lag exceeds the retention window, messages are deleted before they are read, and that is silent, unrecoverable data loss. So the real alert is lag_time > retention × 0.5.

15. Performance

The core banking call dominates by two orders of magnitude, which is the number that should shape the design: everything else is noise, and the only optimizations that matter are the ones that avoid or batch that call.

Which is also why acks=all is not a real cost. It is 5 ms against a 200 ms downstream, and the alternative is losing events on a broker failure.

16. Failure modes