Capstone 01 — Global Real-Time Event Ingestion Platform (end-to-end, runnable)

Phase: 16 — Capstone Systems | Difficulty: ⭐⭐⭐⭐⭐ | Time: 1–2 weeks

The JD's Problem 1, built as ONE runnable Scala pipeline that composes four phases — not a diagram, a working system you can sbt run. Raw events → contract validation + DLQ (P02/P03) → event-time windowed aggregation (P04/P05) → exactly-once lakehouse sink (P09) → query (P10). The end-to-end correctness proofs run with sbt test.

The pipeline

RawEvent ─validate─►  Valid?  ──no──►  DLQ (reasons accumulated)        [Stage 1: P02/P03]
                          │yes
                          ▼
              event-time windowed aggregate   (dedup by eventId)        [Stage 2: P04/P05]
                          │
                          ▼
              publish: atomic overwrite snapshot                        [Stage 3: P09]
                          │
                          ▼
              query: totalByUser / bigSpenders                          [Stage 4: P10]

The whole Pipeline.run is deterministic in the set of input events: shuffling or duplicating the input produces the identical published table — the exactly-once effect, end to end (P01). That's the property PipelineSpec proves.

Run it

sbt run     # drive the demo: prints the DLQ, the published table, and two queries
sbt test    # the correctness proofs (5 specs)

sbt run output (abridged): 2 events rejected to the DLQ (one with 4 accumulated reasons), a duplicate event deduped, 3 aggregated rows published in 1 snapshot, queries answered.

What each file maps to

The acceptance properties (what sbt test proves)

• Exactly-once effect: shuffled + duplicated input → byte-identical table.
• Idempotent sink: re-running republishes the same data (new snapshot, same rows).
• DLQ safety: invalid events are rejected with all reasons and never reach the table.
• Query correctness: totalByUser / bigSpenders over the published table.

How to extend toward production (the spec items)

• Swap the in-memory log for the Phase 02 PartitionedLog (per-key ordering, replay) and the in-memory table for the Phase 09 IcebergTable (OCC commits, time travel, compaction) — both are in this repo as sibling Scala labs.
• Add watermarks + allowed lateness from Phase 04 lab-02-scala-windows for true streaming emission instead of batch aggregation.
• Add multi-region publish + RPO/RTO (P14/P15), cost attribution per producer (P07), and observability (lag/DLQ-rate/freshness SLOs — P13).
• Replace the demo driver with an FS2 stream (P12) reading a real Kafka topic.

Success criteria

• sbt test → 5 specs pass; sbt run → prints the four stages.
• You can point at each line of the pipeline and name the phase + the dial it turns (P00).
• You can explain why the pipeline is exactly-once in effect without exactly-once delivery.