🛸 Hitchhiker's Guide — Phase 11: Serving (Cassandra & DynamoDB)

Read this if: you want the low-latency serving model fast — and to avoid the large/hot partition and tombstone traps that have no analog in analytics. Skim, read WARMUP, build the modeler.

0. The 30-second mental model

A serving store (Cassandra/DynamoDB) answers one query shape in single-digit ms — the opposite of the lakehouse's "any query in seconds." You model around the query, denormalize, and the partition key is everything: it decides distribution (avoid hot), size (avoid large), and ordering. One sentence: design the table from the read you must serve, and protect the partition from getting hot, large, or tombstone-ridden.

1. Cassandra in one breath

masterless (Dynamo-style) token ring; RF replicas; tunable consistency (ONE/QUORUM/ALL)
writes: commitlog + memtable → immutable SSTables → COMPACTION merges them (LSM)
last-write-wins by timestamp; upserts are natural & idempotent (P01)
reads: merge SSTables + memtable, skip tombstones, read-repair

2. Consistency = P01's quorum, tunable

strong (read-your-writes): R + W > RF   e.g. RF=3, QUORUM(2)+QUORUM(2) > 3 ✓
ONE → fast, tolerates RF-1 down, may read stale
QUORUM → balanced, tolerates RF-quorum down
ALL → strongest, tolerates 0 down (one node down = unavailable)
multi-region → LOCAL_QUORUM (don't pay cross-region latency per read)

3. Model around the query

NOT: normalize entities, join later (there are no joins!)
YES: one table per query pattern; denormalize; partition key = how you look it up
partition key → distribution + ordering; clustering keys → sort within partition

4. The two partition killers

LARGE partition: cells = rows × cols; keep < ~100k cells & < ~100MB (hard limit ~2B cells)
  → GC pressure, slow reads, repair pain. Bucket time series; cap unbounded growth.
HOT partition: a celebrity key takes all the traffic (P01 skew, again)
  → melts one replica set. Fix: higher-cardinality key, bucket, or salt.

5. Tombstones (the TTL trap)

delete / TTL expiry → TOMBSTONE (a "this is gone" marker), lingers until compaction
reads scan past tombstones; > ~1000/read = warn, ≥ 100,000/read = FAILED query
queue-like / high-churn / TTL-heavy tables are the classic disaster
fix: TWCS for time-series TTL, model to avoid range scans over deleted data

6. Compaction strategy picker

STCS (size-tiered) → write-heavy / general default; low write amplification
LCS  (leveled)     → read-heavy; fewer SSTables/read, higher write amplification
TWCS (time-window) → time-series + TTL; drops whole expired SSTables cheaply

7. DynamoDB quick contrast

partition key (+ optional sort key); RCU/WCU or on-demand; adaptive capacity for mild skew
hot partition still real (per-partition throughput cap); GSIs cost extra capacity
single-table design; DynamoDB Streams for CDC-out
managed: no compaction/repair to run, but you pay per capacity + less tuning control

8. Streams → serving

Flink (P04) idempotent UPSERTS into Cassandra (last-write-wins; dedup by key)
Spark (P06) backfill RATE-LIMITED so it doesn't overwhelm the cluster
read-path p99 SLOs + tombstone alerts + repair schedule

9. Beginner mistakes that mark you

  1. Modeling like a relational DB (normalize + join) — there are no joins.
  2. A partition key that grows unbounded (every event for a user in one partition).
  3. A low-cardinality/celebrity key → hot partition.
  4. TTL-heavy table on STCS → tombstone storm → failed reads (use TWCS).
  5. ALL consistency "to be safe" → one node down = outage.
  6. Unthrottled Spark backfill → overwhelms the serving cluster.

10. War stories

  • "p99 spiked then reads started failing." → tombstone threshold on a TTL/queue table.
  • "One node is hot, others idle." → celebrity partition key; re-key/bucket/salt.
  • "Repair takes forever / OOMs." → a giant partition. Model the size first.

11. How this phase pays off later

  • Idempotent upserts ← P04 exactly-once sink; rate-limited backfill ← P06.
  • Consistency math = P01 quorums; hot partition = P01 skew.
  • Serving tier of capstone Problems 2 & 4 (P16); multi-region → P14/P15.

Read WARMUP, build the modeler, then P12: Scala & functional programming — the language and SDK layer that ties the whole platform together with type-safe, resource-safe code.