Lab 01 — Mini Broker (a faithful partitioned-log)

Phase: 02 — Event Ingestion & Messaging | Difficulty: ⭐⭐⭐⭐☆ | Time: 6–8 hours

You don't understand Kafka until you've built one. This lab implements the Kafka mental model in ~200 lines of pure Python: a partitioned, offset-addressed log with keyed partitioning, absolute monotonic offsets that survive retention, consumer-group offset tracking + replay, partition assignment + rebalancing, log compaction (the changelog→table duality), consumer lag, and a dead-letter-queue path that quarantines poison messages without blocking the pipeline.

What you build

  • partition_for_key — stable hashing → per-key ordering (P01 carried forward)
  • Partition — append-only log with absolute offsets; retain_count / retain_time advance start_offset (records age out) but offsets never renumber
  • compacted_view — fold a keyed log into the latest-value-per-key table; tombstones (value None) delete — the basis of KTables and CDC snapshots
  • assign_partitions — range and roundrobin assignors; re-running with a different membership is a rebalance
  • ConsumerGroup — committed offsets, poll, commit_through, seek (replay), lag
  • ingest — validate-at-ingress; valid → main log, invalid → DLQ with the reason attached so the pipeline keeps flowing

Key concepts

ConceptWhat to understand
Per-key orderingthe only ordering that exists at scale; chosen by the partition key
Absolute offsetsretention deletes data but offsets stay stable (log_start_offset)
Consumer lagΣ(end − position) — the #1 streaming SLI
Replay = seekrewind offsets; the platform's most valuable capability
Compactiona keyed log is a table; tombstones delete
DLQpoison messages quarantined with a reason; the pipeline never blocks

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 15 tests pass.
  • You can explain why test_retention_by_count_advances_start_offset asserts end_offset == 10 even after dropping 6 records (offsets never renumber).
  • You can explain why seek(0, 0) makes lag jump back up — and why that's a feature.
  • You can state what the consumer of a compacted topic gets vs a regular topic.

Extensions

  • Add replication: each partition has RF replicas + an ISR; produce(acks="all") only returns once min.insync.replicas have it; kill a replica and watch the ISR shrink (P01 quorum math made concrete).
  • Add idempotent producer semantics (producer id + sequence number) so a retried produce doesn't duplicate (P01 exactly-once carried forward).
  • Add a DLQ replay/redrive: after fixing the validator, re-ingest the DLQ into main and assert the poison records now land.
  • Add sticky assignment and measure how many partitions move on rebalance vs range (the Kafka CooperativeStickyAssignor motivation).