« Phase 12 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: coupling against latency
• 2. Where the anti-corruption layer sits
• 3. Kafka or Event Hubs or the ESB
• 4. Choosing the partition key
• 5. Setting the numbers
• 6. Compatibility mode as an organizational decision
• 7. Buying an ISO 20022 stack
• 8. The data-product boundary
• 9. Negotiating with the core banking team
• 10. Reconciliation as a design input
• 11. Migration: adding an AI platform to an existing estate
• 12. What I would not build

1. The central tension: coupling against latency

   FAST / COUPLED                                          DECOUPLED / SLOW
      │                                                             │
   direct sync     mediated sync      async + outbox      batch file
      │                 │                    │                 │
   50ms, their      200ms, their        seconds, no      hours, none
   outage is        outage is           coupling
   your outage      degradation

Choosing one point for everything fails in both directions. The principal move — the same as in every phase of this track — is per interaction class:

And the property to protect: the agent's critical path should touch the estate at most once per step. Every additional synchronous hop multiplies its availability into yours (Phase 00) and adds its p99 to your p50.

2. Where the anti-corruption layer sits

The realistic answer in a bank is both the ESB and a mediation service, and being clear about which owns what is the actual decision:

• The ESB owns connectivity, protocol translation and the estate's own conventions. It exists, it is accredited, and there is a team whose job it is.
• Your mediation service owns the AI-platform-specific concerns: agent identity, per-agent rate limiting, the idempotency store, the response shape agents consume.

The failure mode to avoid is putting AI-specific logic in the ESB. It becomes a shared component with a change-advisory board, and your two-week iteration becomes a quarterly one. Conversely, rebuilding connectivity your ESB already has is a year of work to arrive where you started.

The line I would defend: the ESB owns the protocol; you own the semantics.

3. Kafka or Event Hubs or the ESB

The decision rarely turns on features. It turns on two things:

What is already accredited. In a bank, adopting a new piece of middleware means a security review, a DR plan, a capacity model and an operating procedure. That is quarters. If Event Hubs is already in the landing zone, the argument for Kafka needs to be worth two quarters.

Whether you need compaction and Connect. These are the real functional gaps. If the design needs compacted state topics or a large connector ecosystem, that is a genuine reason for Kafka. If it does not, Event Hubs' Kafka-protocol compatibility means the client code is the same.

And the ESB: it is a message bus, not a log. Usually no replay, no long retention, no consumer groups reading independently. Fine for point-to-point integration, wrong as the platform's event substrate — and worth saying explicitly, because "we already have MQ" is the first question.

4. Choosing the partition key

The decision that is hardest to change later, because changing it breaks ordering during the transition.

The tenant row is the trap in a bank. Keying by tenant gives clean isolation and one enormous partition for the wholesale business, which then becomes the throughput ceiling for the whole topic.

The questions, in order:

1. What must be ordered? If nothing, key by anything with good cardinality and stop worrying.
2. What is the cardinality? It must exceed the partition count by a wide margin, or partitions sit empty.
3. What is the skew? Look at the actual distribution, not the theoretical one. A SELECT key, count(*) ... ORDER BY 2 DESC LIMIT 20 answers it in a minute.
4. What happens when a key is hot? Sometimes the answer is a composite key — account:hash(n) — which splits a hot account across n partitions and gives up its ordering. That is a real trade and it must be deliberate.

And the number: partitions are set early and grown awkwardly. Over-provision. Fifty partitions on a topic doing 100 msg/s costs almost nothing and removes a migration you would otherwise do under pressure.

5. Setting the numbers

Partition count. From peak throughput ÷ per-consumer throughput, times a headroom factor of 2–3. Then round up generously, because adding partitions later breaks per-key ordering during the transition.

Retention. From the longest plausible incident, not from storage cost. If a consumer can be down for a long weekend, retention must exceed a long weekend. Seven days is the common default and it is a guess; make it a decision.

Consumer lag alert. In time, not messages, and at a fraction of retention — lag_time > retention × 0.5. Because the actual catastrophe is not "we are behind", it is "data was deleted before we read it", and that is silent and unrecoverable.

Dedup TTL. ≥ retention. Otherwise a message replayed from the start of the log is not recognized as a duplicate, and the mechanism fails exactly when it is needed.

Outbox relay interval. 100 ms – 1 s. Shorter means more database load for latency nobody notices; longer means an event lag that becomes visible in the UI.

Freshness SLOs. From what the consumer needs, negotiated with them, not from what the pipeline happens to deliver. An SLO derived from current behaviour is a description, not a promise.

Cache TTL for estate reads. Short — seconds to a minute — and the real decision is not the number but what the agent may do with a stale value. Read: yes. Decide: no. Write that down, because somebody will use a cached balance to authorize a payment.

6. Compatibility mode as an organizational decision

The technical content is small. The interesting question is: who is easier to coordinate?

For a platform publishing agent.traces to SRE, finance, model risk and audit — none of whom you control, all of whom have their own release cycles — the answer is FULL_TRANSITIVE, and the cost is that you can essentially never remove a field. That is the correct trade for a product with external consumers, and it should be a conscious one rather than a discovery.

Two organizational practices that matter more than the mode:

Publish the deploy order with the schema change. Not in a wiki — in the pull request, next to the diff. "This is BACKWARD; consumers deploy first" is one line and prevents the release-day outage.

Give every new field a default. It makes the change compatible in both directions and it costs a keyword. Make it a review checklist item; it removes most compatibility conversations entirely.

7. Buying an ISO 20022 stack

The one that surprises people is the second row. Most teams validate against the XSD, ship, and then discover that the scheme rejects for rules the XSD does not contain — character sets, mandatory structured addresses, purpose codes. The rejection arrives days later and costs a customer relationship.

So the question to ask a vendor is not "do you support ISO 20022" but "which market-practice profiles do you validate, and how do you keep them current?"

And the thing not to build: a full ISO 20022 message factory covering the catalogue. You need three or four messages. Build those.

8. The data-product boundary

The AI platform is both a consumer and a producer, and the producer half is where the interesting decision is.

What to publish:

agent.traces is the one to think about hardest. Traces contain prompts, and prompts contain whatever the user typed and whatever was retrieved — which means the trace product inherits the highest classification of anything the agent touched, including MNPI (Phase 11). Publishing it as an "internal" product is a leak with a schema.

Three positions, and I would defend the third:

1. Publish traces raw — simple, and it recreates the data-protection problem in a warehouse.
2. Do not publish traces — safe, and SRE and audit lose the thing they need.
3. Publish two products: a traces.metadata product (ids, timings, costs, decisions, outcomes — no content) at internal classification, and a traces.content product at the highest classification with restricted access. The first serves 90% of the demand.

That split is a small amount of work and it is the difference between a useful product and one nobody is allowed to query.

9. Negotiating with the core banking team

The relationship that determines whether this phase is six months or two years. What they care about, in their priority order:

1. Their availability. They are measured on it and your traffic is unpredictable.
2. Their change budget. Every change is a release with a CAB and a regression suite.
3. Their capacity model. Sized for known traffic; agents are not known traffic.
4. Blame. If your agents cause an incident, it will appear as a core banking incident.

So the asks that succeed are the ones that cost them nothing:

Rows two, three and four are the highest-value asks in this table, and they are all small. Ask for the idempotency key and the query-by-key endpoint during integration design, when they are one field and one query. Asking during an incident is impossible, and the incident is the crash window from Phase 10.

And the framing that works: "we will not add load to your critical path, and here is the rate limit we will hold ourselves to." Offering the constraint before being asked changes the conversation.

10. Reconciliation as a design input

Most teams treat reconciliation as an operational afterthought. Treating it as a design constraint changes the design, and for the better.

Ask, for every integration: what two independent records will we compare, and on what key?

That question, asked at design time, forces two properties into the design that are painful to add later: a shared correlation identifier, and both sides recording the same business key. Both are nearly free at design time.

For an AI platform, three reconciliations are worth having:

The second is the one people skip and then find a 30% discrepancy in, because token accounting and billing disagree about cached tokens.

And the metric: the age of the oldest unresolved break, not the count. A hundred fresh breaks is a bad day; one six-month-old break is a finding.

11. Migration: adding an AI platform to an existing estate

Phase 1 — reads only, through the existing ESB. No new middleware, no new accreditation. Slow and politically free. You learn the estate's actual behaviour, which is never what the documentation says.

Phase 2 — the mediation service. Your own layer, with the idempotency store, agent identity and rate limiting. Still reads only.

Phase 3 — CDC for the events you need. Near-zero cost to the source team, and it removes most of your polling.

Phase 4 — writes, through the action gateway, one tool at a time. Start with the most reversible thing you have — a CRM note — and get the whole path exercised before money is involved.

Phase 5 — the outbox and the event substrate. Once there is more than one consumer of what the platform does.

Phase 6 — data products. When the third person asks you for a CSV.

Phase 7 — payments. Last, with the full stack behind it: ISO 20022 validation, rail selection, finality-aware approval, reconciliation.

The mistake is starting at Phase 7 because it is the demo everybody wants. A payment path built before the mediation layer, the idempotency store and the reconciliation is a payment path that will have an incident, and the incident sets the programme back further than the sequencing would have.

12. What I would not build

A message broker. Kafka and Event Hubs exist. This is not close.

A schema registry. Confluent's and Azure's exist and are cheap. The interesting part is your compatibility policy, not the storage.

A CDC connector. Debezium handles the log formats of every major database, including the version-specific quirks you have not met yet.

A full ISO 20022 message factory. You need three or four messages. Generate or buy the validation; hand-build the construction for what you actually send.

My own reconciliation framework. It is a diff and a workflow. The bank already has both, and integrating with the existing break-management process is worth more than a better diff.

An exactly-once delivery mechanism. It does not exist. Every hour spent here is an hour not spent on idempotent consumers, which do.

A canonical enterprise data model. The idea that all systems will speak one schema. It is the integration project that never finishes, and the anti-corruption layer is the design that admits that.

Direct core-banking access "just for this one agent". It always starts as one agent. It ends as the reason nobody can change core banking, and the reason nobody can say what the agents did.