Phase 11 — Serving: Cassandra & DynamoDB
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 01 (quorums, partitioning), Phase 04/05 (the streams that feed it)
Why This Phase Exists
The lakehouse (P09/P10) answers any query in seconds; a serving store answers one query shape in single-digit milliseconds — and you need both. The JD's Problem 4 is exactly this: "a low-latency serving store for derived real-time features … Cassandra schema design, partition key design, hot partition avoidance, TTL policy, compaction strategy, consistency-level selection, write idempotency, streaming upserts from Flink, backfill from Spark, read-path latency SLOs, tombstone monitoring, repair strategy, multi-region replication." This phase teaches the masterless, query-first serving model — and the ways it bites you (large partitions, tombstones, hot partitions) that have no analog in the analytics world.
Concepts
- Cassandra architecture: masterless (Dynamo-style), consistent hashing token ring,
replication factor +
NetworkTopologyStrategy, tunable consistency (ONE/QUORUM/ALL, LOCAL_QUORUM), hinted handoff, read repair, anti-entropy repair, gossip. - Query-first data modeling: model around the queries, not the entities; denormalize; one table per query pattern; partition key (distribution) + clustering keys (sort within partition).
- The partition pitfalls: large partitions (cells & bytes limits — GC, slow reads, repair pain), hot partitions (skew — P01's enemy again), and the fixes (better key, bucketing, salting).
- Writes & LSM: writes are appends to a commitlog + memtable → flushed to immutable SSTables → compaction merges them. Last-write-wins by timestamp; idempotent upserts (P01) are natural.
- Tombstones & TTL: deletes and TTL expiry create tombstones; reads must scan past them until compaction removes them; too many per read → slow → failed queries. TTL-heavy / queue-like tables are the classic disaster.
- Compaction strategies: STCS (size-tiered, write-heavy), LCS (leveled, read-heavy), TWCS (time-window, time-series/TTL) — and their write/read/space amplification tradeoffs.
- Consistency-level selection: R + W > RF for strong; LOCAL_QUORUM for multi-region; the latency/durability/availability dial (P01) per query.
- DynamoDB: partition key + sort key, RCU/WCU (or on-demand) capacity, adaptive capacity + hot-partition limits, GSIs/LSIs, DynamoDB Streams, single-table design.
- Connecting streams → serving: idempotent streaming upserts from Flink (P04 sink), Spark backfills (P06) rate-limited so they don't overwhelm the cluster, read-path SLOs, and multi-region replication.
Labs
Lab 01 — Cassandra Data Modeler & Serving Calculator (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build consistency/durability math, partition health, hot-partition detection, tombstone thresholds, compaction-strategy selection, and time-series bucketing |
| Concepts | R+W>RF, large/hot partitions, tombstones, compaction strategies, bucketing |
| How to Test | pytest test_lab.py -v — 14 tests |
| Talking Points | Why model around queries? Why are large partitions dangerous? When TWCS vs LCS vs STCS? |
| Resume bullet | Built a Cassandra data-modeling calculator (consistency, partition health, tombstone & compaction strategy, time-series bucketing) for a low-latency serving layer |
→ Lab folder: lab-01-cassandra-modeler/
Extension Project A — Real Cassandra/Scylla hands-on (spec)
docker Cassandra; model a "features by user" and a "events by sensor by day" table; insert
with TTL; force flush + compaction; observe tombstones (nodetool); test QUORUM reads with a
node down.
Extension Project B — Stream → serving pipeline (spec; → capstone)
Flink (P04) idempotent upserts into Cassandra + a rate-limited Spark (P06) backfill; define read-path p99 SLOs and tombstone alerts; sketch multi-region with LOCAL_QUORUM (JD Problem 4).
Integrated-Scenario Hooks
- This serving layer is fed by P04 (Flink upserts) and P06 (Spark backfill), enriched into in P05 (as-of joins read it), and is the serving tier of capstone Problems 2 and 4 (P16).
- Its consistency math is P01's quorums; its hot partition is P01's skew, sixth costume.
Guides in This Phase
Key Takeaways
- Serving stores answer one query shape in ms; model around the query, denormalize.
- Large partitions and hot partitions are the killers — model cells/bytes and skew up front.
- Tombstones from deletes/TTL can fail reads; pick the compaction strategy for the workload.
- Tunable consistency (R+W>RF, LOCAL_QUORUM) is P01's quorum dial, per query.
- Streams feed serving via idempotent upserts; backfills must be rate-limited.
Deliverables Checklist
- Lab 01 implemented; all 14 tests pass
- You can design a partition key avoiding large and hot partitions
- You can choose a consistency level and compaction strategy for a workload
- You can explain the tombstone problem and its fixes
- Extension A hands-on or Extension B stream→serving pipeline