🛸 Hitchhiker's Guide — Phase 02: Event Ingestion & Messaging

Read this if: you want Kafka and Kinesis in your head fast — the anatomy, the durability knobs, the numbers, the gotchas — before the WARMUP derives them. Skim, then read the WARMUP, then build the broker.

0. The 30-second mental model

Kafka/Kinesis = a partitioned, replicated, retained log (P01's log, made real). You produce keyed records (key → partition → per-key order), they're replicated for durability (ISR + acks), consumed by groups that track offsets (so they can replay), and they age out by retention or fold to a table by compaction. The front door must survive bad input (DLQ) and overload (backpressure). One sentence: get ordering, durability, and schema right here — nothing downstream can fix ingestion.

1. Kafka anatomy in one diagram

Producer ──key→hash→ Partition (leader) ──replicate→ Followers (ISR)
                         │                               ▲
                  append (offset N)               acks=all waits for
                         │                          min.insync.replicas
Consumer GROUP ──assigned partitions──> poll(from committed offset) ──commit(next)
                         │
        retention(delete)  OR  compaction(latest-per-key + tombstones)
Controller (KRaft quorum) coordinates leaders & metadata

2. The durability knobs (memorise this combo)

RF=3  +  acks=all  +  min.insync.replicas=2
   → survive 1 broker loss, never lose an ACKed write   (P01: 2 + 2 > 3)
unclean.leader.election = false   → consistency over availability (turn ON only if you
                                    can tolerate data loss)
acks=1  → fast, loses data if leader dies pre-replication
acks=0  → fire & forget, may lose

3. The numbers

4. Producer truths

• Key = ordering + skew. Low-cardinality key → hot partition (P01 skew, again). Fix: higher-cardinality key or key+bucket salt.
• Idempotent producer (default now): PID + per-partition sequence → broker drops retried dups. Exactly-once append, not app-level dedup.
• Transactions: atomic multi-partition write + offset commit → exactly-once inside Kafka (read_committed). Stops at the external sink.

5. Consumer-group truths

• Each partition → exactly one consumer in the group.
• Rebalance = reassign on join/leave/death. Eager = stop-the-world; cooperative-sticky = move only what must move (use it).
• Frequent rebalances = a slow consumer blowing max.poll.interval.ms. Symptom: lag sawtooth + throughput drop.
• Commit after processing = at-least-once (dedup downstream). Auto-commit = footgun.

6. Retention vs compaction (pick by intent)

events (clicks, payments)   → cleanup.policy=delete   (history IS the data)
state (config, CDC, profiles)→ cleanup.policy=compact  (latest-per-key + tombstone delete)

A compacted topic = a changelog that doubles as a table (KTable, __consumer_offsets).

7. Kafka vs Kinesis vs Pulsar (the one-liners)

• Kinesis: lowest ops, AWS-native, shard-quota'd, AWS lock-in. "We're all-in on AWS, optimize ops."
• Kafka/MSK: max control, richest ecosystem (Connect/Streams), portable. "We need throughput control / portability / the ecosystem."
• Pulsar: broker/storage split → elastic storage, built-in multi-tenancy + geo. "We need elastic decoupled storage and tenants."

8. Ingestion safety in three moves

1. Validate at ingress (P03 schema) → bad data never enters the lake.
2. DLQ with reason → poison messages quarantined, pipeline flows, DLQ rate = alert.
3. Backpressure / quotas → slow producers or shed load on purpose; never unbounded buffer.

9. Beginner mistakes that mark you

1. acks=1 (or default-without-checking) for data you can't lose.
2. Infinite in-place retry on a poison message → partition outage.
3. Over-partitioning "for scale" → rebalance + latency + handle costs.
4. Low-cardinality partition key → hot partition; then blaming Kafka.
5. "Kinesis/Kafka gives exactly-once" → only inside; sink re-earns it.
6. Replaying a processing-time pipeline → double counts (need event-time + idempotency).
7. Auto-commit + crash → silent duplicates or loss.

10. War stories

• "Lag is sawtoothing." → rebalance storm; a slow consumer keeps getting kicked. Tune max.poll.records/max.poll.interval.ms or speed the handler; switch to cooperative-sticky.
• "Replay melted the brokers." → big backfill read cold (off page cache) → disk I/O storm. Throttle the replay / use a read replica / tiered storage.
• "One country's events vanished into one partition." → country as the key. Re-key.
• "We can't reprocess last month." → retention was 7 days, no lake copy. The log wasn't kept; this is why you tee raw events to S3 (P08) on day one.

11. How this phase pays off later

• DLQ + validate → schema registry & contracts (P03).
• Replay + compaction → Flink reprocessing (P04), CDC/KTable (P05), serving snapshots (P11).
• Partition/shard sizing → the capstone's global ingestion platform (P16).
• acks/ISR quorum → Cassandra consistency (P11), multi-region (P14/P15).

Read the WARMUP, build the broker, then P03 puts a schema contract on the front door.