Phase 16 — Capstone Systems

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 4–8 weeks (pick one capstone to build deeply; spec the rest) Prerequisites: Phases 00–15 (the capstones compose their labs)

Why This Phase Exists

The previous sixteen phases each built one component in isolation and proved it with tests. A principal's real test is the seams — composing components into a system that satisfies all five forces (P00) at once, under production constraints. This phase is the JD's five "Example Hard Problems" (and its "build a full production-grade data platform" deliverable), each laid out as an integration blueprint that wires together the labs you already built. Build one end to end (Docker-composed real engines for the extension); spec the others as architecture + ADRs (P15).

How to use this phase

For each capstone: (1) draw the architecture and the lifecycle (P00); (2) state the five-force numbers and the dials; (3) map each component to the phase/lab that implements it; (4) define the acceptance criteria (the correctness properties + SLOs); (5) write the key ADRs (P15). The "build" is gluing the verified miniatures into a flowing system; the "principal" is defending every seam.

Capstone 1 — Global Real-Time Event Ingestion Platform (JD Problem 1)

Billions of events/day from product/mobile/backend/partners, multi-region, Protobuf, schema-validated at ingress, DLQ, low-latency Flink, durable raw S3 + partitioned Iceberg, Athena/Trino query, reprocessing, regional failover, exactly-once-effect, end-to-end observability, cost attribution by producer team.

✅ BUILT & RUNNABLE: capstone-01-ingestion-platform/ implements this end to end in real Scala — ingest → Cats-Validated contract + DLQ → event-time windowed aggregation → exactly-once lakehouse publish → query. sbt run drives the demo; sbt test proves the exactly-once effect (shuffled+duplicated input → identical table), DLQ safety, and query correctness (5 specs). The table below maps each concern to the phase/lab that deepens it.

Acceptance: exactly-once-effect from ingress to Iceberg (replay/dup test, P01/P04); raw events recoverable for N days; 99.9% queryable within the freshness SLO; a poison message is quarantined without blocking; a region failover meets RPO/RTO; per-team cost is reported. Key dials: latency↔cost (Flink vs hourly batch), exactly-once↔simplicity, freshness↔replay.

Capstone 2 — Stateful Fraud / Risk Stream Processing (JD Problem 2)

Near-real-time suspicious-behavior detection: event-time, out-of-order, sliding + session windows, pattern detection, Cassandra enrichment, broadcast rules, RocksDB state, S3 checkpoints, savepoint-based deploys, rule versioning, low-latency alerts, replay for validation.

✅ BUILT & RUNNABLE: capstone-02-fraud-detection/ — a Flink-style keyed, event-time stateful detector in Scala: amount/velocity/impossible-travel rules, hot-swappable broadcast ruleset, idempotent (deduped) alerts, and savepoint checkpoint/restore. sbt test proves savepoint-redeploy and replay correctness (8 specs).

Acceptance: identical alerts under shuffled + duplicated delivery (P01/P04); a rule change deploys via savepoint with no state loss and no missed/duplicated alerts; late events handled per policy (corrected or side-output); replay reproduces historical alerts exactly. Key dials: detection latency↔completeness (watermark delay), state size↔cost, freshness↔ replay correctness.

Capstone 3 — Lakehouse Migration: Hive/Tez/Flume → Kafka/Flink/Iceberg/Athena (JD Problem 3, Round 5)

Migrate a legacy warehouse: preserve history, convert ORC/Parquet, keep schemas backward- compatible, introduce Iceberg, support Athena/Trino/Spark, eliminate small files, compact, fix broken partitions, replace batch-only with incremental + streaming writes, maintain lineage, reduce query cost, decommission Flume/Hive safely.

Acceptance: parity (data diff = 0) between legacy and lakehouse for a soak period before cutover; rollback available at every step; small-file count and query $ both drop materially; Flume/Hive decommissioned only after the soak; lineage preserved end to end. Key dials: migration speed↔risk, cloud-native↔portable, cost↔freshness.

Capstone 4 — Cassandra Serving Layer for Real-Time Features (JD Problem 4)

Low-latency derived-feature serving: Cassandra schema + partition-key design, hot-partition avoidance, TTL, compaction strategy, consistency selection, write idempotency, streaming upserts from Flink, Spark backfills, read-path SLOs, tombstone monitoring, repair, multi-region replication.

✅ BUILT & RUNNABLE: capstone-03-feature-serving/ — a Cassandra-style feature store in Scala: LWW upserts, TTL, tombstones + the read-fail threshold, tunable consistency (R+W>RF), median hot-partition detection, and a token-bucket rate-limited backfill that protects the read SLO. sbt test (8 specs).

Acceptance: read-path p99 within SLO under live load + concurrent backfill (backfill is rate-limited); upserts are idempotent (replay-safe); no partition exceeds size/hot thresholds; tombstones-per-read stays under the fail threshold; multi-region reads use LOCAL_QUORUM. Key dials: serving speed↔analytical flexibility, consistency↔latency, backfill throughput↔ read SLO.

Capstone 5 — Enterprise Data Governance Platform (JD Problem 5)

Governance for thousands of datasets/streams: dataset + schema registry, field-level ownership, PII classification, access-control integration, lineage graph, data-quality checks, contract testing, audit logging, retention, dataset certification, breaking-change prevention, integration with Lake Formation/Glue/Athena/IAM.

Acceptance: a breaking schema change is rejected in CI (never reaches prod); PII is masked for uncleared principals and access is audited; every dataset has owner/SLA/lineage/quality (certified); a GDPR erasure removes data including from time-travel snapshots; lineage answers "where did this PII flow?". Key dials: self-service↔governance, flexible schema↔safe contract.

Capstone 6 — The Full Production-Grade Platform (JD final deliverable)

Compose 1–5 into one platform: ingestion → contracts → streaming + batch → lakehouse → query + serving → governance → reliability → cost. This is the JD's mission statement realized — a unified (Kappa-style, P01) platform that ends the streaming/batch split (P09), serves both analytics (P10) and low-latency features (P11), is governed (P14) and observable (P13), and is defensible decision-by-decision in ADRs (P15). The deliverable is the architecture document + the working slice you built + the ADRs + the runbooks (P13) + the cost model (P00/P07).

✅ BUILT (composition slice): capstone-04-unified-platform/ is a multi-module sbt build that reuses the real Phase 02 PartitionedLog and Phase 09 IcebergTable lab sources (no copy/paste) and wires them — ingest → DLQ → consume → aggregate → atomic Iceberg publish — proving the components compose at the seams. sbt test (7 specs); sbt "platform/runMain unified.UnifiedMain" runs it.

Runnable capstones in this folder

(Problems 3 and 5 — the Hive→lakehouse migration and the governance platform — remain architecture+ADR specs, as they are design/leadership exercises rather than single runnable systems.)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the capstone playbook (how to build & present one)
• BROTHER-TALK.md — the graduation

Deliverables Checklist

• Four capstones built and sbt test-verified (01 ingestion, 02 fraud, 03 serving, 04 unified multi-module) — 28 ScalaTest specs
• Problems 3 & 5 specified as architecture + acceptance criteria + key ADRs
• Each capstone's five-force numbers and dials stated
• A runbook (P13) and a cost model (P00/P07) for a built one
• Capstone 6 architecture document (the unified platform) — partial slice built in capstone-04