Phase 02 — Event Ingestion & Messaging
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (30–40 hours) Prerequisites: Phase 01 (the log, partitioning, replication, exactly-once)
Why This Phase Exists
Ingestion is the front door of the platform: if events are lost, reordered, duplicated, or unschematized here, nothing downstream can fix it. The JD's first hard problem is exactly this — "a multi-region event ingestion platform that accepts billions of events per day … Kafka or Kinesis … schema validation at ingress … dead-letter routing … reprocessing support … exactly-once or effectively-once persistence." This phase turns Phase 01's "the log" from theory into the two systems you'll actually run — Kafka and Kinesis — plus the operational machinery around them: consumer groups, retention, compaction, replay, DLQs, idempotent/transactional producers, and the migration off legacy Flume.
Concepts
- Kafka internals: brokers, topics, partitions, the segment-based log, the page cache +
zero-copy path, replication & ISR,
acks&min.insync.replicas, leader election, KRaft (vs ZooKeeper), controller, consumer groups & rebalancing protocols (eager vs cooperative-sticky),__consumer_offsets, retention vs compaction. - Kinesis internals: shards (1 MB/s & 1000 rec/s in, 2 MB/s out), the partition key → shard hash, resharding (split/merge), enhanced fan-out, KCL & lease tables, vs Kafka.
- Choosing the substrate: Kafka (MSK) vs Kinesis vs Pulsar (broker/storage split, tiered storage, multi-tenancy) vs cloud queues — the decision matrix and its dials.
- Delivery & correctness on the wire: idempotent producers (PID + sequence),
transactions (atomic offset+output within Kafka),
read_committed, exactly-once vs effectively-once, and where the guarantee stops (the external sink). - Ingestion safety: schema validation at ingress (handoff to P03), dead-letter queues / quarantine, poison messages, redrive, backpressure & overload, quotas.
- Replay & reprocessing: offset seek, retention windows, compacted topics as state, replay correctness (event-time + idempotency from P01).
- Hot partitions/shards: key design, salting, resharding — the P01 skew problem in its ingestion costume.
- Legacy migration: Flume (source→channel→sink) → Kafka/Kinesis; dual-write & cutover.
Labs
Lab 01 — Mini Broker (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a faithful partitioned log: keyed partitioning, absolute offsets surviving retention, consumer-group offsets + replay (seek), range/round-robin assignment + rebalance, log compaction (changelog→table), consumer lag, and a DLQ ingestion path |
| Concepts | Per-key ordering, log_start_offset, consumer lag, replay, compaction, poison-message quarantine |
| How to Test | pytest test_lab.py -v — 15 tests |
| Talking Points | Why do offsets never renumber after retention? When is a compacted topic the right design? Why does a DLQ keep the pipeline flowing? |
| Resume bullet | Implemented a partitioned commit log with consumer-group offset management, replay, log compaction, and DLQ-based ingestion safety |
→ Lab folder: lab-01-mini-broker/
Lab 02 — Partitioned-Log Ingestion in Scala (implemented, sbt test)
| Field | Value |
|---|---|
| Goal | The Scala twin of Lab 01: stable FNV-1a partitioning (per-key ordering), monotonic offsets, consumer-group poll/commit/replay, retention, log compaction, and a DLQ ingest gateway |
| Concepts | Same ingestion substrate, in idiomatic JVM Scala |
| How to Test | sbt test — 7 ScalaTest specs (runs here) |
| Resume bullet | Built a partitioned-log ingestion library in Scala (stable partitioning, offsets, replay, compaction, DLQ) with ScalaTest coverage |
→ Lab folder: lab-02-scala-ingestion/
Extension Project A — Run real Kafka (spec)
docker compose single-broker Kafka; create a 6-partition topic; produce keyed events;
demonstrate per-key ordering, consumer-group rebalancing (start/stop consumers),
replay-from-offset, and a compacted topic. Wire a validating consumer with a quarantine
topic.
Extension Project B — Flume → Kafka migration memo (spec)
One page + diagram: migrate a Flume spooldir → memory channel → HDFS sink ingest into
Kafka/Flink, dual-writing during cutover, validating parity, and decommissioning Flume
safely (the JD's explicit "migrate legacy Flume" requirement).
Integrated-Scenario Hooks
- This phase's DLQ + validation front-ends the schema registry (P03).
- Its replay feeds Flink reprocessing (P04) and Spark backfills (P06).
- Its compacted topics become CDC/KTable state (P05) and serving snapshots (P11).
- Its partitioning/shard sizing is the ingress of the capstone's global platform (P16).
Guides in This Phase
Key Takeaways
- Ingestion is unfixable downstream — get ordering, durability, and schema right here.
- Kafka and Kinesis are the same log idea with different operational models; choose by the dials, not by fashion.
- Replay + compaction + DLQ are the three operational superpowers; build them in from day 1.
- Exactly-once on the wire stops at Kafka's edge; the external sink re-earns it (P04/P09/P11).
Deliverables Checklist
- Lab 01 implemented; all 15 tests pass
- You can size partitions/shards for a stated throughput and consumer parallelism
-
You can explain ISR +
acks=all+min.insync.replicasdurability precisely - Extension A demo or Extension B memo