d10 — Event Streaming Platform

A fully worked design. Building the thing every other design in this set depends on. The tension is ordering versus parallelism, and every hard decision here is a point on that axis.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Ordering vs Parallelism
• 7. Deep Dive B: Consumer Group Rebalancing
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"Design a durable event streaming platform. Producers write events, many independent consumer groups read them at their own pace, and we need to be able to replay history. It has to handle a service being down for a day and catching up without losing anything."

"Many independent consumer groups reading at their own pace" is the requirement that rules out a queue. A queue deletes on consume; a log retains and lets each consumer track its own position. That distinction — log, not queue — is the first thing to say, and everything follows from it.

"Catching up without losing anything" is the second constraint, and it is sneakier: a consumer that has been down for a day comes back and reads at maximum speed, which is a self-inflicted thundering herd on the brokers at exactly the moment other consumers are healthy.

1. Requirements and Scope

Clarifying questions asked

"Do you need ordering, and at what granularity?" The fulcrum. Assumed per-key ordering, not global. Global ordering means a single writer and caps throughput at one machine. Per-key is what almost everyone actually needs, and it is what makes partitioning possible at all.

"At-least-once or exactly-once?" Assumed at-least-once delivery with idempotent consumers, and I will be precise about what "exactly-once" means when vendors claim it.

"How long is retention?" Assumed 7 days by default, unlimited by opt-in with compaction — because "replay history" and "storage is finite" are in tension and the resolution is per-topic.

"How many consumer groups per topic?" Assumed up to 50. That number matters: read amplification is 50×, so the design is read-heavy, not write-heavy, which is the opposite of most people's mental model.

Functional

1. Produce events to a topic with a partition key.
2. Consume from any offset; commit progress; resume after restart.
3. Multiple independent consumer groups per topic.
4. Replay from an offset or a timestamp.
5. Log compaction — retain the latest value per key indefinitely.

Non-functional

Explicitly out of scope

• Stream processing (joins, windows, aggregation) — that is a layer above; we provide the log.
• Schema management, though I will note where it must plug in.
• Cross-region replication — noted in §9 as a different design.

2. Scale Numbers

Write. 5 GB/s × 3 replicas = 15 GB/s of replication traffic. At 25 Gbps per NIC (~3 GB/s usable) that is ~5 NICs saturated purely by replication, before any consumer reads. That is the number that makes people take replication factor seriously as a cost rather than a checkbox.

Read. 50 consumer groups × 5 GB/s = 250 GB/s if every group reads everything live. That does not fit on any realistic fleet — so the page cache is the design. A consumer reading the tail hits RAM; only lagging consumers hit disk. Sizing: 5 GB/s × 60 s of buffer = 300 GB of page cache across the cluster keeps every real-time consumer off disk entirely.

Storage. 5 GB/s × 86,400 s = 432 TB/day raw, × 3 replicas = 1.3 PB/day, × 7 days = 9 PB. That is a serious storage bill and it is the argument for tiered storage (§9).

Partitions. Throughput per partition is bounded by single-writer ordering — call it 10 MB/s sustained. 5 GB/s ÷ 10 MB/s = 500 partitions minimum. In practice you want more for consumer parallelism, so 1,000–2,000.

The write path's actual cost. A sequential append to a page-cached file is ~100 µs; the replication round trip to 2 followers same-AZ is ~1 ms. So the p99 of 10 ms is dominated by replication and batching, not by disk. That is why sequential-append log storage works: you are never disk-bound on writes, which is the insight the whole category rests on.

Catch-up arithmetic — the number people miss. A consumer 24 hours behind on a 5 GB/s topic must read 432 TB. Even at 1 GB/s it takes 5 days to catch up — it never will. So a day of lag is not recoverable by reading faster; it requires either accepting data loss or having far more consumer capacity. Saying that out loud reframes the requirement, and §8 has the resolution.

3. API Surface

# Produce
produce(topic, key, value, headers)      -> {partition, offset}
   acks = 0 | 1 | all          durability vs latency, explicit

# Consume
subscribe(topic, group_id)                # the broker assigns partitions
poll(max_bytes, max_wait_ms)             -> [records]
commit(offsets)                           # explicit; never auto-commit by default
seek(partition, offset | timestamp)       # replay

# Admin
create_topic(name, partitions, replication, retention, compaction)
describe_group(group_id)                 -> {members, assignment, lag_per_partition}

Four choices worth defending:

• acks is a per-produce parameter. acks=all is ~1 ms slower and is the difference between "durable" and "probably". Making it per-call lets a metrics topic choose speed and an orders topic choose safety. A system-wide setting forces the wrong answer on one of them.
• Explicit commit, never auto-commit by default. Auto-commit acknowledges received, not processed — so a crash between the two silently loses events while appearing to work. This is the single most common data-loss bug in streaming consumers, and defaulting to auto-commit builds it in.
• seek(timestamp) as well as offset. During an incident, "replay from 14:00" is what an operator actually knows; "replay from offset 8,472,193" is not.
• lag_per_partition on describe_group, not just an aggregate. Aggregate lag hides the case where one partition is stuck and 999 are healthy — which is the common failure.

4. Data Model

TOPIC
  └── PARTITION (the unit of ordering AND of parallelism — this is the key idea)
        └── SEGMENT files, immutable, ~1 GB each
              00000000000000000000.log       records
              00000000000000000000.index     offset -> byte position (sparse)
              00000000000000000000.timeindex timestamp -> offset (sparse)

RECORD: offset (int64) · timestamp · key · value · headers · CRC

Per partition:
  leader + N-1 followers (Raft, or ISR-style)
  high_watermark = the highest offset replicated to enough replicas;
                   consumers may only read below it

Per consumer group, per partition:
  committed_offset — itself stored in a compacted internal topic

Two decisions worth defending:

Partition = the unit of both ordering and parallelism. That is not two facts, it is one, and it is the source of every tension in §6. Ordering within a partition comes free from sequential append; parallelism across partitions comes free from independence. You cannot increase one without decreasing the other.

Sparse indexes, not dense. An entry every ~4 KB rather than per record means the index fits in memory for a huge log; a lookup binary-searches to the nearest entry and scans forward a few KB. Dense indexing would be larger than useful and would buy microseconds on an operation measured in milliseconds.

The high_watermark is the consistency boundary. Consumers cannot read a record that is not yet replicated, so a leader failure can never "un-read" data a consumer already saw. Reading uncommitted data would make a consumer's view non-monotonic across a failover, which is far worse than a small latency cost.

5. High-Level Architecture

producers
    │  batched, compressed, keyed
    ▼
┌──────────────────────────────────────────────────────────┐
│  BROKERS                                                  │
│                                                           │
│   partition 0   leader on B1, followers B2 B3             │
│   partition 1   leader on B2, followers B3 B1             │
│   ...           leadership SPREAD, not concentrated       │
│                                                           │
│   append → page cache → sequential flush                  │
│   consumers read from page cache (sendfile, zero copy)    │
└──────────┬───────────────────────────────┬───────────────┘
           │                               │
           ▼                               ▼
 ┌──────────────────┐            ┌──────────────────┐
 │ consumer group A │            │ consumer group B │   independent offsets
 │ (3 members)      │            │ (10 members)     │
 └──────────────────┘            └──────────────────┘
           ▲
           │ assignment, heartbeats, rebalance
 ┌─────────┴──────────┐
 │  Group coordinator │  one broker per group
 └────────────────────┘

Zero-copy is the reason this is affordable. A consumer read is sendfile() from page cache straight to the socket — the data never enters userspace. That is what makes 50× read amplification feasible on commodity hardware, and it is why the format on disk must be exactly the format on the wire. Any server-side transformation (decompression, filtering, schema conversion) destroys it — which is a strong argument against "smart brokers" and is worth saying.

The two hard parts — say these at minute 10:

1. Ordering vs parallelism — one partition means order and no scale; many means scale and no global order.
2. Consumer group rebalancing — the operation that stops the world, and the one that actually hurts in production.

6. Deep Dive A: Ordering vs Parallelism

The fundamental tension

1 partition    → total order, throughput capped at one writer (~10 MB/s)
N partitions   → N× throughput, order only WITHIN a partition

There is no way around it: total order requires a single serialization point, and a single serialization point is a single machine's throughput. Anyone offering both is offering one of them under a different name.

Per-key ordering is the resolution

Partition by hash(key) % N. All events for one key land on one partition, so they are totally ordered relative to each other. Events for different keys have no order — and almost always do not need one.

key = user_id     → all events for a user are ordered
key = order_id    → all events for an order are ordered
key = null        → round-robin, maximum parallelism, no order

Choosing the key is the single most consequential decision a producer makes, and it is irreversible without a full topic migration. The failure modes:

The consequence people miss: the ordering contract propagates

If your consumer processes a partition with 10 threads for speed, you have destroyed the ordering you paid for. The ordering guarantee is only as strong as its weakest link, and the consumer is usually it.

Broker: partition P is ordered  ✓
Consumer: 10 threads on P       ✗   order lost
Consumer: 1 thread per P        ✓   order preserved, parallelism = partition count

So partition count is the consumer's maximum parallelism, permanently. That is why partition count matters more than it looks and why it is chosen for future scale, not current load — increasing it later rehashes keys to different partitions, so a key's history is split across old and new partitions and per-key ordering breaks for the transition.

The mitigation for consumer parallelism without losing order: a per-key work queue inside the consumer — many threads, but all events for one key handled by one thread, in order. This preserves the guarantee while using more cores, and it is what a well-written consumer does.

Hot partitions, and the honest answer

A key that is 40% of traffic caps that partition at ~10 MB/s regardless of cluster size.

There is no option that keeps both ordering and scale for a single key. Say that plainly. The right response is usually to discover why one key is 40% of traffic, because it is often a modelling error upstream.

7. Deep Dive B: Consumer Group Rebalancing

This is what actually hurts in production, and it is the part most candidates skip.

The problem

A consumer group has N members and M partitions. Each partition is assigned to exactly one member (so ordering holds). When membership changes — a deploy, a crash, a scale-up — partitions must be reassigned.

Naive rebalancing is stop-the-world:

1. A member leaves (or a heartbeat times out — 45 s default).
2. The coordinator revokes ALL assignments from ALL members.
3. Every member stops consuming.
4. New assignment is computed and distributed.
5. Every member re-fetches state and resumes.

Total: seconds to minutes of ZERO consumption for the entire group.

The pathological case, and it is common: a rolling deploy of 10 consumers triggers 10 rebalances — one per instance restart — each stopping the whole group. A deploy becomes minutes of accumulated lag, every time.

Fix 1 — Incremental cooperative rebalancing

Only revoke the partitions that actually move.

Old assignment:  A=[0,1,2]  B=[3,4,5]  C=[6,7,8]
C leaves.
Naive:           revoke all 9, reassign all 9        → everyone stops
Cooperative:     A keeps [0,1,2], B keeps [3,4,5],
                 only [6,7,8] are reassigned         → 2/3 never stop

Two rounds: first revoke only what must move, then assign it. Members keeping their partitions never stop consuming. This is Kafka's CooperativeStickyAssignor and it is the default choice for any group above trivial size.

Fix 2 — Static membership

The insight: a consumer restarting during a deploy is not really leaving the group. Give each member a stable group.instance.id; on restart it reclaims its previous assignment without triggering a rebalance at all, as long as it returns within a session timeout.

Deploy with static membership:
  instance-3 restarts, returns in 20 s, reclaims [6,7,8].
  NO rebalance. The rest of the group never noticed.

This turns a rolling deploy from 10 rebalances into 0, and it is the single highest-leverage setting in a production consumer group.

The tradeoff to state: a genuinely dead member is not detected until the session timeout (minutes rather than seconds), so its partitions are unconsumed for that period. That is the right trade when deploys are frequent and hard crashes are rare — which is the normal case.

Fix 3 — Sticky assignment

When a rebalance is necessary, minimize movement: keep each member's existing partitions where possible. This matters enormously for stateful consumers, which hold per-partition local state (an aggregation, a cache, a RocksDB store). Moving a partition means rebuilding that state — potentially minutes of replay.

Sticky assignment turns state rebuild from routine into exceptional, which for a stateful stream processor is the difference between a usable system and an unusable one.

The interaction that bites

A slow consumer triggers a rebalance, which makes it slower. If processing a batch takes longer than max.poll.interval, the coordinator assumes the member is dead and rebalances it out. The member then rejoins, gets partitions back, is still slow, and is evicted again — a rebalance loop where the group makes no progress at all.

The fix is not a bigger timeout, which just delays detection of real failures. It is:

• Smaller batches so a poll cycle is bounded.
• Decouple poll from process — poll on one thread, hand work to a bounded queue, pause partitions when the queue fills. This is backpressure applied to consumption, and it is what a robust consumer looks like.
• Alarm on rebalance rate, which is a leading indicator of this loop and of several other problems.

8. Failure and Recovery

The catch-up problem deserves its own treatment

A consumer 24 hours behind reads from disk, and its reads evict the page cache that healthy consumers depend on. One lagging consumer degrades everyone. That is the containment failure in this design, and it needs an explicit answer:

1. Throttle lagging consumers. A consumer beyond a lag threshold is rate-limited, so it cannot consume all the disk I/O. It catches up more slowly; everyone else stays fast.
2. Separate read paths. Tail reads (page cache) and historical reads (disk) go through different quotas and, ideally, different broker threads — a bulkhead.
3. Tiered storage. Old segments live in object storage. A lagging consumer reads from S3, not from the broker's disk — which removes the contention entirely and is the structurally correct fix.
4. Accept the arithmetic. Per §2, a consumer 24 h behind on a full-rate topic cannot catch up by reading faster. The honest options are to add consumer parallelism (bounded by partition count — which is why partition count matters), or to seek(latest) and accept the gap, explicitly and with an alert. Pretending it will drain is how you end up 3 days behind instead of 1.

On "exactly-once"

Kafka's exactly-once is at-least-once delivery plus deduplication within a transactional boundary Kafka controls — the idempotent producer prevents duplicate appends on retry, and transactions let a consume-process-produce cycle commit offsets and output atomically.

It does not extend past that boundary. A consumer that writes to an external database gets exactly-once only if that write is idempotent or participates in the transaction. Being precise about this is a strong signal, because "we use exactly-once" is a very common overclaim.

Deliberately accepted: consumers see duplicates after a crash between processing and commit. I accept it because the alternative — committing before processing — silently loses events, and for an event log that is the worse failure. The mitigation is idempotent consumers, and the platform makes that possible by providing a stable (partition, offset) as a natural idempotency key.

9. Bottlenecks and Evolution

1. Replication network, immediately. 15 GB/s of replication saturates ~5 NICs. Fixes: rack-aware replica placement so cross-rack traffic is minimized; compression at the producer (the broker stores and serves the compressed batch, preserving zero-copy — compressing at the broker would destroy it); and honestly evaluating whether every topic needs RF=3.

2. Page cache pressure from lagging consumers. Covered in §8; the structural fix is tiered storage.

3. Partition count ceiling. Each partition costs file handles, memory for the index, and per-partition metadata in the controller. Tens of thousands per cluster is where the controller struggles. Fix: more clusters, or fewer-and-larger partitions with in-consumer key-level parallelism.

4. Storage cost — the big one. 9 PB for 7 days. Tiered storage moves segments older than a few hours to object storage at ~10× lower cost, keeping only the hot tail on broker disks. This changes the economics of long retention completely and is why it is now standard.

5. Cross-region. A separate design: async mirroring (offsets do not match across clusters, which breaks naive failover), or a stretched cluster (writes pay cross-region quorum latency). The offset mismatch is the subtle part and it is the thing that makes disaster recovery harder than it looks.

At 10× (50 GB/s): the design holds but the economics do not — replication alone is 150 GB/s. That forces RF=2 with tiered storage as the durability backstop, or erasure coding. Worth naming as the direction rather than pretending RF=3 scales indefinitely.

10. Tradeoffs Explicitly Rejected

Rejected: a queue (delete on consume). Simpler, less storage. Rejected because multiple independent consumer groups and replay both require retention, and both are stated requirements. Flip condition: a single consumer with no replay need genuinely wants a queue, and a log is over-engineering there.

Rejected: global ordering. Rejected on arithmetic: a single serialization point caps throughput at one machine (~10 MB/s here, against a 5 GB/s requirement). Per-key ordering gives what consumers actually need. Flip condition: a system where total order is genuinely required — a replicated state machine, a ledger — should use consensus and accept the throughput ceiling.

Rejected: broker-side filtering / transformation ("smart brokers"). Attractive: consumers receive less data. Rejected because it destroys zero-copy — the broker must decompress, parse and re-serialize, so a sendfile becomes a full userspace round trip. At 50 GB/s of reads that is the difference between feasible and not. Filtering belongs in the consumer or in a downstream processing layer.

Rejected: auto-commit by default. Convenient. Rejected because it acknowledges received, not processed, so a crash between them silently loses events while appearing correct. It is the most common data-loss bug in this category and the default should not build it in.

Rejected: unclean leader election by default. Allows a topic to stay available when all in-sync replicas are lost, by electing an out-of-sync replica. Rejected as a default because it silently loses acknowledged writes. Offered per-topic: a metrics topic can enable it; an orders topic must not.

Rejected: a database as the storage engine. Rejected because the access pattern is append-and-sequential-scan, which is exactly what a log-structured file plus page cache does optimally and what a B-tree does badly. The absence of updates and random reads is what makes this category fast.

The Hostile Critique

C1. "Static membership means a genuinely dead consumer isn't detected for the session timeout — you said minutes. Its partitions are unconsumed that whole time. For an orders topic, walk me through what that means to the business, and how you'd know."

C2. "You throttle lagging consumers so they don't evict the page cache. The lagging consumer is the payments reconciliation job and it's lagging because of an incident. You've just slowed down the recovery of the most important consumer to protect the least important ones."

C3. "Partition count is the consumer's permanent parallelism ceiling, and you can't increase it without breaking key ordering. So you pick 2,000 up front. What does 2,000 partitions cost a consumer group with 3 members?"

C4. "acks=all waits for the in-sync replica set. A follower is slow but not slow enough to be evicted from the ISR. What's your produce latency, and what's your p99?"

C5. "Tiered storage puts old segments in S3, so lagging consumers read from S3 instead of broker disk. What's the latency of a sequential scan over 400 TB in S3, and does that actually help the consumer that's 24 hours behind?"

C6. "Idempotent producer dedupes on (producer_id, sequence). The producer restarts. New producer ID. Walk me through what happens to the batch that was in flight."

The Revision

R1 — Static membership needs a liveness signal that isn't the session timeout (answers C1)

The critique is correct and it exposes a real gap: I traded rebalance frequency for detection latency without bounding the cost.

Change: separate crash detection from membership churn.

1. A short heartbeat (5 s) for liveness, a long session timeout (5 min) for membership. A member that stops heartbeating is marked suspect immediately, without triggering a rebalance.
2. Suspect members are probed. If the process is genuinely gone — TCP RST, or the orchestrator reports the pod as terminated — that is positive evidence of death, and a rebalance fires immediately rather than waiting out the timeout.
3. Lag-based escalation, which is the real answer. If a suspect member's partitions accumulate lag beyond a threshold, force a rebalance regardless of the timeout. The business impact is lag, not membership, so trigger on the thing that matters.

And the observability that should have been there: alarm on per-partition lag, not group aggregate. A single unconsumed partition among 2,000 is invisible in an aggregate and is exactly the failure the critique describes.

Cost: more coordinator state and orchestrator integration. Worth it: static membership's value is real, and this recovers the failure detection it costs.

R2 — Throttle by priority, not by lag (answers C2)

The critique lands hard, and it inverts the policy: lag is not a proxy for unimportance. The most important consumer is often the one lagging, precisely because it is doing the most work during an incident.

Change: consumers have a priority class, and throttling respects it.

critical    payments, reconciliation, fraud   never throttled
standard    the normal case                   throttled beyond a lag threshold
bulk        analytics, ML training            throttled aggressively; may be paused

Plus:

1. Reserve I/O capacity per class, so bulk consumers cannot starve critical ones even when bulk is the one lagging — the same reserved-floor pattern as d05.
2. A critical consumer that lags triggers a page, because it is a signal about the system rather than about the consumer.
3. Bulk consumers can be paused entirely during an incident, freeing the whole read path for critical catch-up. That is an explicit lever an operator can pull, and it should exist.

Cost: a class must be assigned per group, and the assignment can be wrong. Mitigated by defaulting to standard and requiring justification for critical — otherwise everything becomes critical, which is the same as nothing being critical.

R3 — Decouple partition count from consumer parallelism (answers C3)

The critique identifies a real cost I understated. 2,000 partitions across 3 members means ~667 partitions per consumer: 667 fetch sessions, 667 offset commits per cycle, 667 sets of buffers. Memory and commit overhead can dominate the actual work.

Change, three parts:

1. Fetch coalescing. One fetch request per broker, not per partition — the protocol already supports multi-partition fetches, and using it turns 667 requests into ~10 (one per broker holding partitions for this member).
2. Batched offset commits. All partitions' offsets in one commit, not 667. This is by far the biggest saving and it is a common misconfiguration.
3. Right-size at creation, and be honest that it is a bet. Partition count should be chosen for plausible peak consumer parallelism, not for maximum imaginable. 2,000 is right if you might run 2,000 consumers; if realistic peak is 50, choose 200 and accept a future migration if you are wrong.

And the escape hatch that should be stated: increasing partitions breaks key ordering for keys that move, but a topic can be migrated cleanly by producing to a new topic with more partitions and having consumers read both during a transition, keyed consistently. It is real work — days, not minutes — but it is not impossible, and saying so is better than presenting partition count as permanently unchangeable.

R4 — ISR membership needs a latency bound, not just a lag bound (answers C4)

The critique finds a genuine and well-known failure: a follower that is just fast enough to stay in the ISR sets the produce latency for every write, and it is invisible in lag metrics because it is not falling behind — it is just slow.

Change:

1. Evict from the ISR on latency, not only on lag. A follower whose replication-ack p99 is more than, say, 3× the median of its peers is removed from the ISR even if its lag is small. It keeps replicating and rejoins when healthy — this is outlier detection, the same fail-slow pattern as everywhere else.
2. acks=all means "min.insync.replicas", not "all replicas". With RF=3 and min.insync.replicas=2, a write is acked when the leader plus the fastest follower have it. One slow follower is then irrelevant to latency while durability still survives one failure. This is the important configuration detail and it is frequently misconfigured to require all three.
3. Alarm on the ISR-membership rate, because a follower flapping in and out is a symptom (bad disk, noisy neighbour) that shows up here first.

Cost: evicting on latency risks a smaller ISR and therefore less durability headroom. Bounded by never shrinking the ISR below min.insync.replicas — at that point you keep the slow follower and take the latency, because durability wins.

R5 — Tiered storage helps throughput, not latency, and that is the point (answers C5)

The critique is right to be skeptical, and the honest answer is that it does not make the lagging consumer faster — it stops that consumer from hurting everyone else. Those are different benefits and I conflated them.

The arithmetic: S3 sequential read at ~100 MB/s per connection, but highly parallelizable. 400 TB at 100 MB/s is 46 days on one connection; at 100 parallel connections it is 11 hours. So it is feasible only with heavy parallelism, and that parallelism is bounded by partition count — which brings §6 back around.

Change, being precise about what tiering buys:

1. The real benefit is isolation. Historical reads leave broker disk and page cache entirely, so a lagging consumer costs S3 bandwidth (elastic, someone else's problem) rather than broker I/O (fixed, shared). That is the win, and it is a bulkhead, not a speedup.
2. Prefetch aggressively for historical reads — a lagging consumer's access pattern is perfectly sequential and therefore perfectly predictable, so read-ahead of many segments converts latency into throughput.
3. And restate the honest conclusion from §2: a consumer 24 h behind on a full-rate topic probably cannot catch up. The options are more consumer parallelism (capped by partitions), accepting the gap with seek(latest), or a parallel catch-up job that processes history out of order into a side store while the live consumer stays current. That third option is usually the right one and it is a design decision the consumer's owner must make deliberately.

R6 — Producer restart needs a durable identity (answers C6)

The critique finds the real limit of idempotent producers. A new producer ID means the broker's dedupe state — keyed on (producer_id, sequence) — does not recognize the retried batch, so a batch that was written but not acked before the restart is written again. Idempotence covers retries within a producer session, not across one.

Change:

1. Transactional producers with a stable transactional.id. On restart, the producer re-registers the same ID; the broker fences the old epoch (rejecting any in-flight writes from it) and exposes the last committed sequence. Zombie writes from the previous instance are rejected — which is the fencing-token pattern applied here, and it is worth naming as such.
2. The transactional.id must be stable and unique per logical producer — derived from a pod's stable identity, not randomly generated at startup. A random ID at startup makes the whole mechanism inert, and it is a common misconfiguration.
3. And be honest about what remains. With a stable transactional ID and fencing, a producer restart is safe. Without one, duplicates on restart are unavoidable and the consumer must dedupe on a business key. The platform cannot solve it alone, and saying so is better than implying idempotent producers make duplicates impossible.

Cost: transactional producers are slower (an extra coordinator round trip per transaction) and require operational discipline about IDs. Right for topics where duplicates matter; unnecessary overhead for a metrics topic — which is another per-topic decision rather than a global one.

References

• ../WARMUP.md#49-delivery-semantics — at-least-once, the outbox, DLQs
• ../WARMUP.md#43-leases-and-fencing — the fencing argument reused in R6
• d04-webhook-delivery.md — a consumer of exactly this kind of log
• ../../coding/harness/problems/event_dedupe/ — idempotency, windowed dedupe and reordering as a timed problem
• Kreps, Narkhede, Rao. Kafka: a Distributed Messaging System for Log Processing. NetDB 2011
• Kreps, J. The Log: What every software engineer should know about real-time data's unifying abstraction. — the clearest statement of log-not-queue
• Wang et al. Building a Replicated Logging System with Apache Kafka. VLDB 2015
• Confluent. Transactions in Apache Kafka and Incremental Cooperative Rebalancing — the mechanisms in §7 and R6
• Kleppmann, M. Designing Data-Intensive Applications, 2nd ed. — Ch. 11 (stream processing, exactly-once semantics)
• Amazon Builders' Library. Avoiding insurmountable queue backlogs.