Capstone 04 — Unified Platform (multi-module sbt build reusing the real lab code)

Phase: 16 — Capstone Systems | Difficulty: ⭐⭐⭐⭐⭐ | Time: 1 week

Proof that the labs aren't toys: a multi-module sbt build whose ingestion and lakehouse modules compile the actual Scala sources from the Phase 02 and Phase 09 labs (no copy/paste — via unmanagedSourceDirectories), and a platform module that dependsOn both and wires them into one pipeline. Verified with sbt test and runnable with sbt "platform/runMain unified.UnifiedMain".

The module graph

root (aggregate)
├── ingestion   ← compiles ../../phase-02-.../lab-02-scala-ingestion/src/main/scala  (REAL PartitionedLog, ConsumerGroup, IngestGateway)
├── lakehouse   ← compiles ../../phase-09-.../lab-02-scala-iceberg/src/main/scala     (REAL IcebergTable, snapshots, OCC)
└── platform    ← dependsOn(ingestion, lakehouse); the glue + tests

The platform code imports ingestion.PartitionedLog and lakehouse.IcebergTable directly — if those didn't resolve to the lab modules, it wouldn't compile (the first test is that smoke check).

The pipeline

ingest "key"->"amount"  --validate--> DLQ (non-integers)          [REAL P02 IngestGateway]
                        \--ok--> PartitionedLog (per-key ordering) [REAL P02 PartitionedLog]
ConsumerGroup --read all partitions--> sum amount per key          [REAL P02 ConsumerGroup]
              --publish--> IcebergTable.append (atomic snapshot)   [REAL P09 IcebergTable]

Run

sbt test                                  # 7 specs across the composed modules
sbt "platform/runMain unified.UnifiedMain"  # the demo

runMain output: 1 event rejected to the DLQ, an Iceberg snapshot published, per-key sums (alice 20, bob 8).

What the tests prove

  • The real Phase 02 + Phase 09 types compile and compose (the smoke test).
  • End to end: ingest → aggregate → publish a snapshot; invalid payloads hit the DLQ only.
  • Order independence: shuffling the input yields identical published rows.
  • Per-key ordering via the real PartitionedLog; replay via the real ConsumerGroup.seek.
  • Publishing twice produces two Iceberg snapshots (real history).

Why this matters

The other capstones are self-contained miniatures; this one demonstrates the principal-level move of composing independently-built, independently-tested components into a platform without duplicating code — and that the seams (validation→DLQ, log→consumer, aggregate→commit) actually line up. The build itself is the artifact: unmanagedSourceDirectories is the pragmatic way to reuse sibling modules; in a real monorepo these would be published library modules the platform depends on by version.

Extend

  • Add a windowing module reusing Phase 04 lab-02-scala-windows for event-time aggregation instead of a plain sum.
  • Add a serving module reusing Capstone 03's FeatureStore as a second sink (lakehouse for analytics, Cassandra for low-latency serving — the dual-sink platform).
  • Publish the lab modules as real versioned artifacts and depend on them by coordinate.