Capstone 03 — Cassandra Serving Layer for Real-Time Features (runnable Scala)
Phase: 16 — Capstone Systems | Difficulty: ⭐⭐⭐⭐⭐ | Time: 1–2 weeks
The JD's Problem 4, built as a Cassandra-style feature store in Scala (P11) — verified with
sbt test. The parts that actually bite in production: LWW upserts, TTL, tombstones + the read-fail threshold, tunable consistency (R+W>RF), hot-partition detection, and a token-bucket rate-limited backfill that protects the read SLO.
What it implements
| Capability | In the code | Why it matters |
|---|---|---|
| LWW upsert | upsert ignores stale writeTs | idempotent → replay-safe streaming writes (P04) |
| TTL | ttlMillis; expired reads return None | feature freshness |
| Tombstones | expiry/delete create them; scanPartition fails past threshold | the #1 Cassandra outage (P11) |
| Consistency | Consistency.isStrong(r,w,rf) = R+W>RF | read-your-writes vs latency (P01) |
| Hot partition | hotPartitions (median-based, P06) | skew/celebrity keys (P01) |
| Rate-limited backfill | TokenBucket + Backfill.run | backfill must not blow the read SLO |
Run
sbt test
Expected: 8 specs, all green — LWW idempotence, TTL→tombstone, tombstone-overflow read
failure, the R+W>RF boundary (R+W==RF is not strong), median hot-partition detection,
and a backfill that takes ceil(rows/rate) ticks while landing every row.
Success criteria
sbt test→ all 8 specs pass.- You can explain why
R + W = RFis not strong (no quorum overlap) and whyLOCAL_QUORUMis the multi-region default. - You can explain the tombstone trap (delete/TTL → marker → read scans them → fails past a threshold) and why TWCS suits TTL-heavy time series.
- You can explain why a backfill must be rate-limited (P06 throughput vs P11 read-path p99).
How to extend toward production
- Wire streaming upserts from Capstone 02's detector or the Phase 04 windower (idempotent, LWW).
- Add compaction strategies (STCS/LCS/TWCS) and model tombstone GC after
gc_grace. - Add multi-region replication (NetworkTopologyStrategy + per-DC
LOCAL_QUORUM) and an RPO/RTO story (P14/P15). - Replace the in-memory map with a real Cassandra driver session; keep these as the modeling/SLO layer that decides partition keys, TTLs, and consistency.