Lab 02 — Partitioned-Log Ingestion in Scala

Phase: 02 — Event Ingestion & Messaging | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

The Scala twin of Lab 01's Python mini-broker — the ingestion substrate as a JVM library. Stable partitioning (per-key ordering), monotonic offsets, consumer-group offset tracking

  • replay, retention, log compaction, and a DLQ ingest gateway. Verified with ScalaTest.

What you build

  • Partitioner — 32-bit FNV-1a stable hash → partition (a key always maps to the same partition → per-key ordering survives restarts, unlike a salted runtime hash)
  • PartitionedLog — append (monotonic offsets), read-from-offset, retain (retention), compact (keep latest per key)
  • ConsumerGroup — poll / commit / seek (replay)
  • IngestGateway — validate → append, or route to the DLQ (poison-message safety)

Run

sbt test

Expected: 7 tests, all green in IngestionSpec — including per-key ordering, replay via seek, retention, compaction, and DLQ routing.

Key concepts (Scala expression)

ConceptScala
Per-key orderingpartitionFor(key) is stable → same partition, increasing offsets
Offsetsmonotonic per partition; survive retention
ReplayConsumerGroup.seek(p, offset) re-reads
CompactionLinkedHashMap keeps latest-per-key
DLQinvalid payloads diverted, main log never blocked

Success criteria

  • sbt test → all 7 specs pass.
  • You can explain why a stable hash (not the JVM's hashCode across processes) is required for per-key ordering.
  • You can explain compaction vs retention and when each applies.

Extensions

  • Add consumer rebalancing: N consumers share P partitions; reassign on join/leave.
  • Add idempotent producer semantics (dedup by a producer-supplied sequence number).
  • Add exactly-once commit: commit the consumer offset and the downstream write atomically (the 2-phase-commit idea — P04).