« Phase 12 · Warmup · Track Overview

Core Contributor — Working on the Engines Themselves

What it takes to contribute to Kafka, Debezium, a schema registry, or the integration layer your bank builds. Read this if you want to understand the systems rather than configure them.

Table of Contents

• 1. Why read the engines
• 2. Kafka: the log
• 3. Kafka: replication and the ISR
• 4. Kafka: the consumer group protocol
• 5. Kafka: transactions and EOS
• 6. Debezium: reading a transaction log
• 7. Schema Registry
• 8. Avro, Protobuf, JSON Schema
• 9. Building an in-house integration layer
• 10. Testing integration code
• 11. Contributing

1. Why read the engines

Because the configuration options only make sense from the inside. acks=all plus min.insync.replicas=2 is a durability guarantee; acks=all with min.insync.replicas=1 is theatre. Nothing in the option names tells you that; the replication protocol does.

Because the failure modes are protocol-level. "Why did my consumer group rebalance for ninety seconds?" is unanswerable without knowing that eager rebalancing revokes every partition from every member before reassigning.

2. Kafka: the log

A partition is a directory of segment files:

   /kafka-logs/payments-0/
       00000000000000000000.log      ← the records
       00000000000000000000.index    ← offset → byte position (sparse)
       00000000000000000000.timeindex← timestamp → offset (sparse)
       00000000000012345678.log      ← the next segment

Three design decisions worth stealing:

Append-only, sequential I/O. Writes go to the end of the active segment. Sequential disk writes are fast even on spinning disks, and this is why Kafka's throughput is what it is on unremarkable hardware.

Sparse indexes. The .index file maps some offsets to byte positions — one entry per index.interval.bytes (4 KB default). A lookup binary-searches the sparse index, then scans forward. Small index, fast enough seek, and the index stays in page cache.

Zero-copy. Kafka uses sendfile(2) to move bytes from page cache to socket without a trip through user space. Which is why any transformation on the broker — decrypt, filter, convert — is so expensive: it forfeits zero-copy, and the throughput drops by a large factor. That single fact explains most of Kafka's design philosophy of "the broker is dumb".

Worth reading in apache/kafka:

3. Kafka: replication and the ISR

Each partition has a leader and followers. The in-sync replica set (ISR) is the replicas caught up with the leader.

   producer ──► leader ──► followers fetch
                  │
                  └── acks=all: respond once every ISR member has it

The interaction that matters, and it is the classic interview question:

acks=all with min.insync.replicas=1 is the trap. It reads as maximum durability and provides none, because the ISR can shrink to just the leader and "all" is then satisfied by one machine.

unclean.leader.election.enable is the other one. Set true, a replica outside the ISR can become leader — availability over consistency, and it silently discards records the old leader had acknowledged. For a bank: false. Always. And know that it defaults to false in modern versions, because enough people got this wrong.

KRaft has replaced ZooKeeper for metadata: the controller quorum uses Raft, metadata is itself a log. Removes an entire operational dependency, and it is what any new deployment should use.

4. Kafka: the consumer group protocol

   consumer ──JoinGroup──► coordinator ──► picks a leader among the members
   leader   ──SyncGroup──► coordinator ──► distributes the assignment
   consumer ──Heartbeat──► coordinator ──► "still alive"

The assignment is computed by one of the consumers, not by the broker — a pluggable ConsumerPartitionAssignor on the client. Which is why you can change assignment strategy without touching the cluster, and why a mixed-strategy rolling deploy misbehaves.

The assignors:

Two timeouts that cause most operational pain:

session.timeout.ms — no heartbeat within this and the member is dead. Heartbeats are on a background thread.

max.poll.interval.ms — the gap between poll() calls. Exceed it and the member is removed even though it is heartbeating, because it is clearly stuck. A slow handler with a large max.poll.records trips this, triggering a rebalance, which slows everyone, which trips it again: the rebalance storm.

Static membership (group.instance.id) is the one to know: a consumer that restarts within session.timeout.ms keeps its assignment, so a rolling deploy of N consumers causes zero rebalances instead of N.

5. Kafka: transactions and EOS

Kafka's exactly-once semantics, and the scope is what matters:

   producer.initTransactions()
   producer.beginTransaction()
     producer.send(...)                       # to output topics
     producer.sendOffsetsToTransaction(...)   # the INPUT offsets, atomically
   producer.commitTransaction()

The mechanism:

• An idempotent producer — a producer id plus a per-partition sequence number, so the broker discards duplicates from a retry.
• A transaction coordinator with its own log, doing two-phase commit across partitions.
• Control markers written into the partitions; a read_committed consumer skips aborted data.

What it guarantees: consume → transform → produce, atomically, within Kafka.

What it does not: any effect outside Kafka. The moment your handler calls core banking, the transaction cannot include it, and you are back to idempotent handling with a dedup key. Which is why this section is short and §9's dedup table is long.

Cost: 3–10× lower throughput and higher latency for the coordinator round trips. Worth it for a stream-processing topology, rarely worth it for a consumer whose effect is external.

6. Debezium: reading a transaction log

Debezium turns a database's write-ahead log into events, per database:

Three problems every connector solves, and reading the solutions is the education:

The initial snapshot. The log only has recent changes, so the first run must read the whole table. The naive version locks the table. Debezium's incremental snapshot (the DDD-3 algorithm) chunks the table and interleaves chunks with live streaming — no lock, no long pause. It is genuinely clever and it is in AbstractIncrementalSnapshotChangeEventSource.

Exactly-once from an at-least-once log. Offsets are committed periodically, so a restart replays some events. Every event carries its log position, so consumers can dedup — the same content-derived-key idea as everywhere else.

Schema evolution. A DDL change mid-stream must be reflected in the event schema. Debezium tracks the schema history in its own topic and replays it on restart, which is why that topic's loss is a connector that cannot start.

The transform to know by name:

"transforms.outbox.type":
    "io.debezium.transforms.outbox.EventRouter"

The outbox pattern as a Kafka Connect transform: it reads an outbox table, unwraps the payload into a business event, and routes by aggregate type. This combination — CDC so the source needs no change, outbox so the events have business semantics — is the standard answer to §11 of the deep dive.

7. Schema Registry

Confluent's registry is a Kafka application: schemas live in a compacted topic (_schemas), the service is a cache with an HTTP API.

The wire format is worth knowing because you will debug it:

   byte 0        magic byte (0)
   bytes 1-4     schema id, big-endian int32
   bytes 5+      the Avro/Protobuf payload

Five bytes of framing, and the consumer fetches the writer's schema by id and reads with it. Which is why a message written with schema 7 stays readable after schema 8 is registered — the writer's schema is recorded, not just the reader's.

Subject naming strategies, which is a design decision people make by accident:

The default forbids multiple event types on one topic — which matters, because putting PaymentInitiated and PaymentSettled on one topic is exactly what preserves their ordering. If you want that, you need TopicRecordNameStrategy, and you need to have decided it before the first message.

8. Avro, Protobuf, JSON Schema

Avro's model is the one to understand. It records the writer's schema and resolves against the reader's, field by field, using defaults for what is missing. Which is why "always give a new field a default" is not style advice — it is the mechanism Avro uses to read old data with a new schema.

Protobuf identifies fields by number, so names can change freely and numbers can never be reused. That is a different discipline: reserved 5; after removing field 5 is what stops a future field from silently inheriting old data.

JSON Schema is readable and verbose, and it is usually the right choice for events crossing an organizational boundary where the consumer is not in your ecosystem — which for a bank's data products is common.

9. Building an in-house integration layer

One module owns every translation. The anti-corruption layer is a place, not a principle. If mainframe field names appear in two modules, they will appear in twenty.

Money is a type. Not an int with a comment.

@dataclass(frozen=True)
class Money:
    minor: int
    currency: str
    def __add__(self, other):
        if other.currency != self.currency:
            raise ValueError("cannot add different currencies")
        return Money(self.minor + other.minor, self.currency)

The type is what stops somebody adding AED to USD. A bare int does not.

Validation returns a list, never raises on the first problem. And every entry names its element.

The outbox is a table, not a queue. Same database as the domain change, or it is not an outbox.

Consumers are idempotent by construction. Make the dedup check part of the consumer base class, so writing a non-idempotent consumer requires effort.

Property tests on the invariants:

# to_minor / from_minor round-trip for every currency and precision
# to_minor never rounds — it raises
# reconcile(a, a) == []
# reconcile(a, b) and reconcile(b, a) report mirrored break types
# the same key always maps to the same partition
# processing any permutation with duplicates yields one effect per event id

The last is the important one, and Hypothesis finds the ordering you did not consider within seconds.

Deterministic tests. Injected clock, injected broker, derived partitioning. The lab's 144 tests run in 0.12 s with no flakiness, which is not an accident — it is the direct consequence of never calling time.time(), hash() or a network.

10. Testing integration code

Two that are usually missing.

Rebalance testing. Start two consumers, kill one mid-batch, assert exactly-once effects. It is ten lines with Testcontainers and it exercises the single most common source of production duplicates. Every team that writes this test finds something.

Real scheme test files. Schemes publish test message sets. Running them through your validator finds the market-practice rules you did not know existed — before a customer's payment file does.

11. Contributing

Apache Kafka (apache/kafka) — Java/Scala, huge, high bar. Anything protocol-level needs a KIP (Kafka Improvement Proposal) and a vote. Approachable entry points: client-side fixes, documentation, the AdminClient, connectors. Read the KIP archive first — it is the best record of why the system is shaped the way it is.

Debezium (debezium/debezium) — Java, Red Hat, active and friendly. New connectors and transforms are the natural contribution. The incremental-snapshot implementation is the most interesting code in the project.

Confluent Schema Registry (confluentinc/schema-registry) — Java, moderate size. Compatibility checkers and subject strategies are self-contained entry points.

Testcontainers (testcontainers/testcontainers-java) — the fastest way to make integration testing tractable, and the module ecosystem welcomes additions.

ISO 20022 tooling — this is a gap. Open-source market-practice validation (CBPR+, HVPS+) barely exists, and a well-tested library for one profile would be genuinely useful to a lot of people.

For all of them the useful preparation is the same: implement the mechanism yourself first — the lab is a small version of exactly that — then read theirs and find every place they differ. The differences are where the real engineering is.