Warmup Guide — Event Ingestion & Messaging

Zero-to-principal primer for Phase 02: Kafka and Kinesis from the inside, the durability and ordering knobs that decide correctness, consumer groups and rebalancing, retention vs compaction, transactions and idempotent producers, dead-letter safety, replay, and the migration off legacy Flume.

Table of Contents

• Chapter 1: Kafka's Anatomy
• Chapter 2: The Log on Disk — Segments, Page Cache, Zero-Copy
• Chapter 3: Durability — acks, ISR, and Leader Election
• Chapter 4: Producers — Partitioning, Idempotence, Transactions
• Chapter 5: Consumer Groups and Rebalancing
• Chapter 6: Retention vs Compaction
• Chapter 7: Kinesis — The Managed Cousin
• Chapter 8: Choosing the Substrate — Kafka vs Kinesis vs Pulsar
• Chapter 9: Ingestion Safety — DLQs, Poison Messages, Backpressure
• Chapter 10: Replay, Reprocessing, and Legacy Flume Migration
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Kafka's Anatomy

Kafka is a cluster of brokers storing topics, each split into partitions — and a partition is exactly the log from P01: append-only, offset-addressed, ordered. The pieces:

• Broker: a server holding some partitions (as leader for some, follower for others).
• Partition: the unit of parallelism, ordering, and replication. More partitions = more consumer parallelism and more producer throughput, but also more open file handles, more rebalance cost, more end-to-end latency, and (until recently) more controller load. Sizing partitions is a real design decision, not a default.
• Replica / leader / follower: each partition has a replication factor (RF, usually 3) copies; one is the leader (handles all reads/writes) and the rest are followers that replicate. Clients only talk to the leader.
• Controller: one broker coordinates leadership and cluster metadata. Historically this lived in ZooKeeper; modern Kafka uses KRaft (a self-managed Raft quorum) — fewer moving parts, faster failover, the thing to mention as "current."
• __consumer_offsets: an internal compacted topic where consumer groups store their committed offsets — Kafka eating its own dog food (offsets are just a compacted log).

The mental model to keep: a topic is a sharded log; a partition is a single log; ordering is per-partition; everything else is operations around that.

Chapter 2: The Log on Disk — Segments, Page Cache, Zero-Copy

Why is Kafka fast despite writing everything to disk? Three mechanical reasons worth knowing because they explain its limits:

• Sequential I/O: appending to a log is sequential disk write, which is ~hundreds of MB/s even on spinning disks — orders of magnitude faster than random I/O. Kafka's whole performance story is "never do random I/O."
• Segments: each partition is split into segment files (+ an offset index and time index). Retention deletes whole old segments (cheap), which is why retention is coarse (segment-granular), not per-record — and why log_start_offset advances in chunks.
• Page cache + zero-copy: Kafka doesn't maintain its own cache; it relies on the OS page cache, and serves consumers via sendfile (zero-copy) — bytes go disk→NIC without passing through user space. This is why a consumer reading the tail (hot in page cache) is nearly free, and why a consumer reading old data (cold, e.g. a big replay) causes disk I/O and can hurt the brokers — a real operational gotcha during large backfills.

The lab models offsets and start_offset faithfully so this maps directly: retention advancing start_offset is segments being deleted.

Chapter 3: Durability — acks, ISR, and Leader Election

This is the chapter interviewers probe hardest because it's where correctness lives.

• ISR (in-sync replicas): the set of replicas currently caught up to the leader. A follower that falls too far behind is removed from the ISR (and re-added when it catches up).
• acks (producer): acks=0 (fire-and-forget, may lose), acks=1 (leader only — lost if the leader dies before a follower replicates), acks=all (leader waits for all in-sync replicas).
• min.insync.replicas (topic): with acks=all, a write only succeeds if at least this many replicas are in-sync. The classic safe config is RF=3, min.insync.replicas=2, acks=all: you can lose one broker and still accept writes and never lose an acknowledged write. This is P01's quorum math (R + W > N → 2 + 2 > 3) wearing Kafka's clothes.
• unclean.leader.election: if all in-sync replicas die, do you (a) promote an out-of-sync replica (available but loses data) or (b) refuse writes until an in-sync one returns (consistent but unavailable)? This toggle is the CAP choice, exposed as a config. Default it false for any data you care about.

The durability sentence to own: "RF=3, acks=all, min.insync.replicas=2 survives one broker loss with no acknowledged-write loss; the moment unclean leader election is on, you've traded that for availability."

Chapter 4: Producers — Partitioning, Idempotence, Transactions

• Partitioning: a record with a key hashes to a partition (per-key ordering); a record without a key is distributed (sticky-partitioner batching in modern clients). Key choice = ordering + skew. A low-cardinality key (e.g. country) creates hot partitions (P01's skew problem); fix with a higher-cardinality key or a key + bucket salt.
• Idempotent producer (enable.idempotence=true, now default): the producer gets a producer id (PID) and stamps each record with a sequence number per partition, so the broker can drop a retried duplicate — exactly-once append under producer retries. Note the scope: this dedupes producer retries, not application-level duplicates.
• Transactions: a producer can write to multiple partitions and commit its consumer offsets atomically (initTransactions / sendOffsetsToTransaction / commit). Consumers with isolation.level=read_committed only see committed records. This is how Kafka offers exactly-once within Kafka (consume→process→produce). The instant you write to an external sink (S3, Cassandra), you leave Kafka's transaction and re-earn the guarantee yourself (P04 two-phase-commit sinks, P09 Iceberg commits, P11 upserts).

Chapter 5: Consumer Groups and Rebalancing

A consumer group is a set of consumers cooperatively reading a topic; each partition is assigned to exactly one consumer in the group, so group parallelism is capped at the partition count (extra consumers sit idle — the lab tests this). Mechanics:

• Assignment strategies: range (contiguous blocks per topic — can imbalance across multiple topics), roundrobin (even but reshuffles more), sticky / cooperative-sticky (minimise partition movement on rebalance — the modern default).
• Rebalancing: when a member joins/leaves/dies (or partitions change), the group reassigns partitions. Eager rebalancing stops the world (everyone revokes, then re-joins) — a latency hit; cooperative rebalancing moves only the partitions that actually need to move. Frequent rebalances (from max.poll.interval.ms timeouts because a consumer is slow) are a classic production pain — the symptom is "lag sawtooths and throughput drops."
• Offset commits: a consumer commits the next offset to read (the lab's commit_through(last) = last+1). Commit after processing for at-least-once; commit before for at-most-once. Auto-commit is convenient and a duplicate/loss footgun under failure — principals usually commit manually around the processing boundary.

Chapter 6: Retention vs Compaction

Two completely different ways a topic forgets:

• Delete retention (cleanup.policy=delete): drop data older than retention.ms or beyond retention.bytes. The log is a time window of events. log_start_offset advances as old segments are deleted; offsets never renumber (the lab models this exactly).
• Log compaction (cleanup.policy=compact): keep at least the latest value per key forever; older values for a key are garbage-collected; a record with a null value (tombstone) marks a key for deletion. A compacted topic is a changelog that doubles as a table — the foundation of Kafka Streams' KTables, CDC snapshots, and Kafka's own __consumer_offsets. You can also combine (compact,delete).

When to use which: events → delete retention (you want the history window); current-state / config / CDC → compaction (you want the latest per entity, compactly). The lab's compacted_view is this fold.

Chapter 7: Kinesis — The Managed Cousin

Kinesis Data Streams is "Kafka as an AWS service," with different vocabulary and limits:

• Shard ≈ partition, but with hard quotas: 1 MB/s and 1000 records/s ingress, 2 MB/s egress per shard. Throughput is shards × those limits — so you size in shards (the lab's kinesis_shards_needed).
• Partition key → shard: an MD5 hash of the partition key maps records to a shard's hash range; ordering is per-shard. Same hot-key problem, same fixes.
• Resharding: split a hot shard or merge cold ones — but resharding is a manual, rate-limited operation (vs Kafka's fixed partitions), and consumers must handle parent→ child shard lineage. On-Demand mode auto-scales shards at a cost premium.
• Consumption: classic GetRecords is polled and shared (5 tx/s, 2 MB/s per shard across all consumers); Enhanced Fan-Out gives each consumer a dedicated 2 MB/s push pipe (lower latency, more cost). The KCL (Kinesis Client Library) manages leases & checkpoints in a DynamoDB lease table — the moral equivalent of consumer groups + __consumer_offsets.
• Retention: 24h default, extendable to 365 days; replay = re-read by sequence number / timestamp.
• Firehose is a different product: managed delivery to S3/Redshift/OpenSearch with buffering — no replay, no custom processing; great for "just land it," wrong for stream processing.

Chapter 8: Choosing the Substrate — Kafka vs Kinesis vs Pulsar

The decision matrix (and the dial each row turns — P00):

The principal answer is never "use X." It's: "all-AWS team optimising for low ops and we're already in the ecosystem → Kinesis/MSK; need maximum throughput control, the Connect/Streams ecosystem, or cloud portability → Kafka; need elastic storage decoupled from compute and first-class multi-tenancy/geo → Pulsar." Then write the ADR.

Chapter 9: Ingestion Safety — DLQs, Poison Messages, Backpressure

The front door must stay open under bad input and overload:

• Poison message: a record that can't be processed (malformed, fails schema, triggers a bug). Without handling, it blocks its partition forever — the consumer retries, fails, retries, and every record behind it starves. One bad message takes down a partition.
• Dead-letter queue / quarantine (the lab's ingest): route the failing record to a DLQ topic with the reason attached, and move on. The pipeline keeps flowing; the bad data is preserved for diagnosis; the DLQ rate becomes an alert (a step change = upstream broke something — you find out in minutes, not at model-metric time). After fixing the cause, redrive the DLQ back into the main flow.
• Schema validation at ingress (P03): the most common DLQ trigger; validate against the contract at the door so bad data never enters the lake.
• Backpressure & quotas (P01 Ch. 8): when consumers can't keep up, lag grows — that's the signal. Kafka offers client quotas (bytes/s per principal) to stop one noisy producer from starving others; on overload you slow producers or shed load on purpose, never grow an unbounded buffer.

Chapter 10: Replay, Reprocessing, and Legacy Flume Migration

• Replay is just seek to an older offset and re-consume — possible because the log is retained (or compacted). It's how you backfill a new consumer, recover from a processing bug, or re-derive a corrupted dataset. Replay is only correct if processing is event-time + idempotent (P01): otherwise the replay double-counts or lands in the wrong windows. The test of a well-built pipeline is literally "can you replay it and get the same answer."
• Flume migration (a JD requirement): Flume is the legacy Hadoop-era agent (source → channel → sink, e.g. spooldir/tail → memory/file channel → HDFS sink). It's push-based, has weak ordering/replay, and couples producers to HDFS. The migration pattern: (1) stand up Kafka/Kinesis alongside; (2) dual-write — keep Flume running while a new producer also writes to Kafka; (3) build the new consumer (Flink/Spark) reading Kafka and validate parity against the Flume-fed HDFS output; (4) cut over consumers to the Kafka path; (5) decommission Flume once parity holds for a soak period. The principles are the same as any migration (P15): dual-write, validate, cut over, keep rollback.

Lab Walkthrough Guidance

Lab 01 — Mini Broker, suggested order:

1. fnv1a_32 + partition_for_key, then Partition.append/end_offset/read with absolute offsets — verify offsets are monotonic and stable.
2. retain_count / retain_time — confirm start_offset advances but end_offset doesn't (Ch. 2/6).
3. compacted_view — latest-per-key, tombstone deletes (Ch. 6).
4. assign_partitions (range, then roundrobin) and re-run with a new member = rebalance (Ch. 5); confirm extra consumers go idle.
5. ConsumerGroup position/poll/commit_through/seek/lag — then the replay test: seek(0,0) makes lag jump back (Ch. 10).
6. ingest with a DLQ — poison messages quarantined with a reason; good records after a poison one still land (Ch. 9).

Success Criteria

You are ready for Phase 03 when you can, from memory:

1. Draw Kafka's anatomy (broker/topic/partition/replica/leader/ISR/controller).
2. Explain acks=all + min.insync.replicas durability as quorum math, and what unclean leader election trades.
3. Explain idempotent producers and Kafka transactions, and exactly where the guarantee stops.
4. Describe consumer-group rebalancing (eager vs cooperative) and why frequent rebalances hurt.
5. Contrast delete-retention and compaction and pick the right one for events vs state.
6. Size Kinesis shards (and Kafka partitions) for a stated throughput.
7. Choose Kafka vs Kinesis vs Pulsar for a stated context, with the deciding dial.
8. Explain the DLQ pattern and the Flume→Kafka migration steps.

Interview Q&A

Q: A single bad message is stalling a partition and lag is climbing. Walk me through it. That's a poison message: the consumer fails on it, retries, and starves everything behind it in that partition. Immediate mitigation: route the failing record to a DLQ with the error attached and advance past it, so the pipeline flows and the bad data is preserved. Permanent fix: validate at ingress against the schema contract (so it never enters), alert on DLQ rate (a step change is an upstream regression), and add a redrive to reprocess the DLQ once fixed. The anti-pattern is infinite in-place retry, which converts one bad record into a partition outage.

Q: Producers are creating hot partitions. Diagnose and fix. The partition key is too low-cardinality or skewed (Zipfian "whale" keys), so the hash piles many keys onto few partitions — P01's skew factor, now in Kafka. Confirm by per-partition byte/record rate. Fixes, in order of preference: choose a higher-cardinality key that still preserves the ordering you need; if a single real key is hot (one giant account), add a key + bucket salt and reconcile downstream; only then consider more partitions (doesn't help a single hot key). The thing not to do is raise hardware — skew is a layout problem.

Q: When would you choose a compacted topic? When consumers need the current state per entity, not the event history: a config topic, a CDC snapshot of a table, a feature/profile keyed by user, or offsets themselves. Compaction keeps the latest value per key (tombstones delete), so a new consumer can bootstrap the full current state by reading the topic once — a changelog that doubles as a table. For genuine event streams (clicks, payments) you want delete-retention because the history is the data.

Q: "We're on Kinesis, we get exactly-once." True? No more than Kafka does. Kinesis gives ordered, at-least-once delivery per shard with replay; exactly-once is still an effect you engineer — KCL checkpoints can replay a batch after a worker failure, so your processing must be idempotent (dedup by sequence number, or idempotent upserts at the sink). The guarantee, as always, is re-earned at the external boundary, not granted by the transport.

References

• Kafka: The Definitive Guide (Shapira, Palino, Sivaram, Petty) — the operational bible
• Kafka docs — Design (delivery, ISR, transactions) and KRaft
• Kreps, The Log (2013); Kleppmann, DDIA Ch. 11 (stream processing)
• Amazon Kinesis Data Streams developer guide (shards, resharding, KCL, enhanced fan-out)
• Apache Pulsar concepts (segments, tiered storage, multi-tenancy, geo-replication)
• Confluent: Exactly-Once Semantics in Apache Kafka (the transactions design)