« Phase 12 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a new integration
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy the transport, build the semantics. A broker is a distributed-systems problem with a decade of engineering behind it. What a payment means, which rail it takes, what "fresh" means for your trace product — nobody will ship those.

And the row people get wrong is market-practice validation. Teams validate against the XSD, ship, and discover the scheme rejects for rules the schema does not contain. The vendor question is not "do you support ISO 20022" but "which profiles do you validate, and how do you keep them current?"

2. A decision framework for a new integration

Ten questions, in order. Four of them are usually unanswered the first time:

1. Is this a read or a write? Reads can be cached and degraded; writes cannot.
2. Is it reversible? This is the side-effect class, and it comes from the scheme, not from your opinion about difficulty.
3. Does the source accept an idempotency key? If not, no retry anywhere is safe.
4. Can we query it by our own key? The reconciliation-after-a-crash question. Nearly free at design time; impossible during an incident.
5. What is the availability, and what is our behaviour when it is down? Not "we retry".
6. What two records will we reconcile, and on what key? If there is no answer, a silent divergence is undetectable — add a correlation id now.
7. Sync, async or batch? And the honest answer is usually what the estate already does.
8. What is the throughput, and what is the peak-to-average? Batch windows create the peak.
9. Who owns the source, and what is their change budget? It determines what you may ask for.
10. What does the data mean? Field semantics, not field names. Ask what a null means.

Questions 3, 4 and 6 are the ones that get skipped and are the cheapest to answer early. Ask them in the first integration-design meeting, in writing.

3. Review red flags

In a design document

• Agents calling core banking directly.
• No anti-corruption layer; estate field names in the domain model.
• Money as a float, or as an int with no currency.
• No mention of currency exponents.
• ISO 20022 validation described as "we validate against the XSD".
• No market-practice profile named.
• "We'll reverse it if it goes wrong" about a settled payment.
• Retry policy identical across payment rails.
• A dual write, unremarked.
• "Exactly-once delivery."
• Consumers with no idempotency.
• A dedup key derived from the offset.
• No stated partition key, or one that is obviously skewed (tenant).
• Partition count chosen without a throughput calculation.
• No retention decision, or one made on storage cost.
• Lag alerting in messages rather than time.
• No compatibility mode stated for a topic.
• CDC directly on domain tables.
• A data product with no owner or no freshness SLO.
• No reconciliation, anywhere.

In code

# Red flag: float money
amount = float(row["amount"]) * 100                # 114.99999999999999

# Red flag: assumed exponent
minor = int(Decimal(amount) * 100)                 # JPY is now 100x too large

# Red flag: silent rounding
minor = round(Decimal(amount) * 100)               # the invoice no longer matches

# Red flag: bare int money
def transfer(amount: int, ...)                     # in what currency?

# Red flag: stop at the first error
if not msg_id: raise ValueError("MsgId missing")   # six errors, six days

# Red flag: namespace ignored
root = ET.fromstring(xml)                          # a .03 parsed as a .09

# Red flag: dual write
db.save(payment); broker.publish(event)            # crash between them

# Red flag: hash() for partitioning
partition = hash(key) % n                          # salted per process

# Red flag: auto-commit
enable.auto.commit = true                          # at-most-once, subtly

# Red flag: commit before processing
group.commit(offset); handle(msg)                  # a crash loses it

# Red flag: offset as the dedup key
if msg.offset in seen: skip                        # changes on rebalance

# Red flag: unbounded dedup set
self.seen.add(event_id)                            # grows forever

# Red flag: retry everything the same way
for _ in range(3): send_payment(...)               # including the settled rail

# Red flag: no CtrlSum check
# (the absence is the flag: a truncated file passes every other check)

# Red flag: acks=all with min.insync.replicas=1
# (reads as maximum durability; provides one replica)

In an incident review

• "The payment was a hundred times too large" → assumed minor units.
• "The file was rejected and we didn't know why" → validating only the XSD.
• "Half the payments in the file went through" → no control-sum check.
• "Downstream never got the event" → dual write, database first.
• "Downstream acted on a payment that never happened" → dual write, broker first.
• "Duplicates after every deploy" → rebalance, no idempotency.
• "Messages were deleted before we read them" → lag exceeded retention.
• "Ordering broke after we scaled" → partition count changed.
• "We found out from the customer" → no reconciliation.

4. Production war stories

The hundred-fold payment. A cross-border integration assumed two minor digits. The first JPY payment was for ¥1,000,000 instead of ¥10,000. It was caught by a limit check downstream — the limit check, not the payment code, and nobody had thought of it as a control. Every other currency in the first year had been AED and USD.

One cent, every day. Amounts were parsed as floats and multiplied by 100. 1.15 became 114.999... and int() made it 114. One cent short, on maybe 3% of payments, for eight months. It appeared as a persistent small reconciliation break that three people investigated and attributed to "timing".

The truncated file. A network hiccup during an SFTP transfer left a payment file cut in half. Every remaining element was valid; the schema passed. CtrlSum and NbOfTxs were not populated, because they were optional. Six hundred payments were made and four hundred were not, and nobody knew which until the beneficiaries called.

Rejected for a rule that was not in the schema. A corporate customer's payment file was rejected by the scheme for an unstructured creditor address. The XSD permitted it; CBPR+ did not. Three days of investigation, because the local validator said the file was fine.

The dual write, both directions. First version: database then publish. A pod restart lost an event, and core banking had a payment the platform's ledger did not. They reversed the order to "fix" it. Now a crash published an event for a payment that never happened, a customer was notified of a transfer that did not occur, and the correction was manual. Both bugs were live for two months. The outbox took an afternoon.

Duplicates on every deploy. A rolling restart triggered a rebalance; partitions moved mid-batch; uncommitted messages were reprocessed. The handler was not idempotent. Every deploy produced a handful of duplicate CRM notes, which looked like user error for a year — until one deploy moved a partition whose messages were payments.

hash(). Partitioning used Python's hash(). It worked perfectly, until a rolling deploy meant two consumers with different PYTHONHASHSEED values were producing to the same topic — and events for one account went to two partitions. Per-account ordering broke for exactly the accounts that were active during the deploy window, which is why it took months to reproduce.

Silent data loss. Consumer lag was alerted on message count with a generous threshold. A slow consumer fell behind by four days on a topic with seven-day retention over a long weekend. It never tripped the count alert because the topic was low-volume. Three days of events were deleted before they were read. Unrecoverable, and discovered by reconciliation a week later.

The release where nothing could read anything. A required field was added to an event schema. The subject was BACKWARD, so consumers should have deployed first. The release train deployed producers first, because that is alphabetical. Every consumer failed to deserialize for the forty minutes it took to work out why.

CDC on domain tables. The initial integration streamed core banking's tables directly. Eighteen months later a DBA renamed a column as part of an unrelated change, and every one of nine consuming services broke simultaneously. Nobody had told the DBA, because nobody knew the list.

The 2 a.m. flood. CDC captured every change including the nightly reconciliation batch, which touched eleven million rows. Consumer lag went to four hours every night, and the daily alert was tuned off as noise. Then a real incident happened at 2:30 a.m. and nobody looked.

The data product nobody could query. Agent traces were published as an "internal" data product. They contained prompts. Prompts contained retrieved documents. Some of those were MNPI. The product was suspended pending review and the review took four months, during which SRE had no trace access at all. Splitting metadata from content — which would have taken a day — would have avoided it.

No reconciliation. An integration ran for two years without one. When it was finally built, it found 1,400 breaks, the oldest eighteen months old. Nobody could reconstruct what had happened for most of them.

5. The interview signal

Signal 1 — you never integrate directly, and you can give five reasons. Change cycle, availability, protocol, no concept of an agent, and — the one that shows depth — core banking is right by definition, which is what makes reconciliation a design constraint.

Signal 2 — money is an integer count of minor units, and the exponent varies. JPY zero, KWD three. It is a small fact and it separates people who have shipped a cross-border payment from people who have not.

Signal 3 — you refuse extra precision rather than rounding it. And can say why: a silently rounded payment does not match the invoice, and it is a daily break that takes weeks to trace.

Signal 4 — NbOfTxs and CtrlSum are the truncated-file detector. Almost nobody says this, and it demonstrates you have thought about what a schema cannot check.

Signal 5 — you name the market-practice layer. "We validate against the XSD and the CBPR+ profile" is the answer of someone who has had a file rejected.

Signal 6 — a return is a new payment; a cancellation is a request. Finality, in two sentences, in the standard's own vocabulary.

Signal 7 — finality determines the side-effect class. So autonomy is rail-dependent: an instant payment is final on submission and needs a human; a pre-cut-off batch payment has hours of revocability and may not. Collapsing that into "payments need approval" is the merely-safe answer.

Signal 8 — settlement dominates the cut-off. The instant-payment case, which is the one that catches a careless implementation.

Signal 9 — the dual write has no ordering fix, and you name both failure modes. Lost event versus phantom event, and which is worse.

Signal 10 — the outbox promises no lost events, not no duplicates. Followed immediately by "which is why consumers are idempotent". The two halves, together.

Signal 11 — exactly-once effects, not delivery. With the dedup key derived from content, never the offset, and the TTL ≥ retention.

Signal 12 — ordering is per partition, and that is the trade that buys parallelism. Plus: a stable digest, not hash().

Signal 13 — compatibility mode is deploy order. Reasoned out rather than recited.

Signal 14 — you alert on lag in time against retention. Because the catastrophe is silent deletion, not being behind.

Signal 15 — reconciliation is how you find out your dual-write bug exists. Monitoring shows both systems healthy, because each is internally consistent.

Anti-signals:

• Agents calling core banking directly.
• Money as a float.
• "We validate against the XSD" as a complete answer.
• "We'll just reverse it."
• Exactly-once delivery claimed.
• No idempotency on consumers.
• No partition key stated.
• No reconciliation.

The question to ask them: "Your service writes a payment to the database and publishes an event. The process dies between them. What happens, and what do you do about it?" A weak answer reorders the writes. A strong one names both failure modes, reaches the outbox, and then volunteers that the outbox still produces duplicates so the consumer must be idempotent.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Show them int(1.15 * 100). Ten seconds. Then ask how many payments a day their system processes. Nobody uses a float for money again.
2. Kill the process between the write and the publish. Have them build the two-line dual write, then kill -9 it in the middle and look at the two systems. The outbox stops being a pattern from a blog post and becomes the obvious fix.
3. Run a rebalance with a non-idempotent consumer. Two consumers, kill one mid-batch, count the effects. Ten lines with Testcontainers, and it converts "idempotency is good practice" into "this happens on every deploy".

And the framing for the platform team: this is the phase where the failures are quiet and expensive. A wrong minor-unit assumption is a hundred-fold payment. A dual write is a divergence nobody notices for months. Lag past retention is unrecoverable data loss that no alert fired for. None of these page anyone; all of them are found by reconciliation or by a customer.

Which is the argument for building the boring things first. The argument that gets it funded is: "today, if our platform and core banking disagree about a payment, we find out when the customer calls. A daily reconciliation is a table and a query, and it turns that into a Tuesday-morning ticket."