Warmup Guide — Serving: Cassandra & DynamoDB

Zero-to-principal primer for Phase 11: the masterless, query-first serving model, partition design and its killers (large + hot partitions), the LSM write path, tombstones & TTL, compaction strategies, tunable consistency, DynamoDB, and wiring streams to serving safely.

Table of Contents

Chapter 1: Serving vs Analytics

The lakehouse (P09/P10) answers any query over all the data in seconds; a serving store answers one query shape over one key in single-digit milliseconds. They are different jobs with opposite designs, and a real platform needs both (the JD's Problem 4 wires a streaming pipeline into Cassandra serving):

Lakehouse (P09/P10)Serving (Cassandra/DynamoDB)
Queryany, ad-hoc, scansone known key/shape
Latencysecondssingle-digit ms
Modelnormalized-ish, columnardenormalized, query-shaped
Scale axisdata volumeread/write QPS

The mistake is using one for the other: scanning Cassandra like a warehouse (it can't), or point-looking-up the lakehouse per request (too slow). This phase is the serving side, and its rules will feel wrong if you bring relational/analytics instincts.

Chapter 2: Cassandra Architecture

Cassandra is a masterless (Dynamo-style) distributed database — every node is equal, there's no single leader:

  • Token ring + consistent hashing: the partition key hashes to a token; the ring assigns token ranges to nodes (with vnodes for smoother distribution). This is P01's consistent hashing — adding a node moves only ~1/N of the data, which is why Cassandra scales horizontally without resharding pain.
  • Replication: each partition is replicated to RF nodes; NetworkTopologyStrategy places replicas across racks/datacenters (the basis of multi-region, P14).
  • Coordinator + gossip: any node can coordinate a request (route to replicas); nodes share state via gossip; failures are handled by hinted handoff (store writes for a down node) and read repair (fix stale replicas on read) + scheduled anti-entropy repair.
  • No joins, no foreign keys, no ad-hoc WHERE — you query by the partition key (and clustering keys). That constraint is the design (Ch. 3).

Chapter 3: Query-First Data Modeling

The cardinal rule, and the one that breaks relational brains: model around your queries, not your entities. You start from "what exact lookups must I serve?" and build a table per query shape, denormalizing freely (writing the same data several ways is normal and correct):

  • Partition key: decides distribution (which node) and is required in every query — it's how you look the row up. All rows for a partition key live together on one replica set.
  • Clustering keys: sort rows within a partition (e.g. events by time within a user), enabling efficient range scans inside a partition.
  • Denormalization: no joins means you pre-join at write time — store the data in the shape each query needs. A "messages by user" table and a "messages by channel" table may hold the same messages twice. That's the trade: write amplification + storage for read speed.

If you find yourself wanting a join or an ad-hoc filter on a non-key column, you've modeled it like a relational DB — step back and design the table from the read.

Chapter 4: The Partition Killers — Large and Hot

Two partition pathologies cause most Cassandra incidents (the lab models both):

  • Large partitions: a partition's storage is cells = rows × columns; keep it under ~100k cells and ~100 MB (the hard limit is ~2 billion cells, but you die long before that). Oversized partitions cause GC pressure, slow reads, painful repair, and node instability. The trap is an unbounded partition — "all events for a user" grows forever. Fix: bucket the partition key (e.g. user + day), so each partition is bounded (the lab's bucket_key). You must reason about partition growth over time at design.
  • Hot partitions: a celebrity key (a viral product, a whale account) takes disproportionate traffic and melts its replica set — P01's skew, sixth costume (the lab's hot_partition_factor). Fix: a higher-cardinality key, bucketing, or salting (and reconcile downstream).

Model both before shipping — Cassandra won't warn you; it just degrades until it falls over.

Chapter 5: The Write Path, LSM, and Compaction

Cassandra (like RocksDB, P04/P08) is an LSM-tree store, which is why writes are fast and why tombstones (Ch. 6) exist:

  • A write appends to the commit log (durability) and a memtable (in-memory); when the memtable fills it flushes to an immutable SSTable on disk. Writes never do random I/O (P08's sequential-I/O lesson) — that's the speed.
  • Because SSTables are immutable, an "update" is just a new write with a later timestamp; reads merge across SSTables + memtable, newest-timestamp-wins (last-write-wins). This makes upserts naturally idempotent (a replayed write with the same data is a no-op effect) — perfect for streaming sinks (Ch. 8, P04).
  • Compaction merges SSTables (and drops superseded/expired data). The strategy matters (the lab's choose_compaction_strategy):
    • STCS (Size-Tiered): merge similar-sized SSTables — low write amplification; the write-heavy / general default.
    • LCS (Leveled): keep SSTables in non-overlapping levels — fewer SSTables per read (better read latency) at higher write amplification; read-heavy workloads.
    • TWCS (Time-Window): group by time window and drop whole expired windows — **time-series
      • TTL** (avoids per-row tombstones, Ch. 6).

Chapter 6: Tombstones and TTL

The LSM's dark side, and a top cause of mysterious Cassandra outages:

  • A delete doesn't remove data — it writes a tombstone (a "this is gone" marker with a timestamp). TTL expiry does the same. The real data + tombstone linger until compaction removes them (after gc_grace_seconds, to ensure deletes propagate).
  • On read, Cassandra must scan past tombstones to find live data. Too many tombstones per read = slow; past a threshold (warn ~1,000, fail ~100,000 — the lab's tombstone_scan_ok) Cassandra aborts the query. The symptom is baffling: "reads were fine, now they error," with no obvious cause unless you check tombstone counts.
  • The classic disasters: queue-like tables (constant write+delete) and TTL-heavy tables on STCS. Fixes: avoid the queue anti-pattern; use TWCS for time-series-with-TTL (drops whole expired SSTables without generating per-row tombstones); model to avoid range scans over deleted data; monitor tombstones-per-read (P13).

Chapter 7: Tunable Consistency

Cassandra exposes P01's quorum dial per query:

  • Consistency levels: ONE (fast, tolerates RF−1 down, may read stale), QUORUM (majority = RF//2 + 1), ALL (strongest, tolerates 0 down — one node down = unavailable), LOCAL_QUORUM (quorum within the local DC — the multi-region default, avoids cross-DC latency).
  • Strong (read-your-writes) iff R + W > RF (the lab's consistency_is_strong) — the read and write replica sets must overlap. RF=3 with QUORUM reads + QUORUM writes (2+2>3) is the canonical strong-but-available setting, tolerating one node down (tolerated_failures).
  • The dial is per data product: a balance lookup wants R+W>RF; a "recently viewed" list is fine at ONE. Reaching for ALL "to be safe" is the rookie move — it's the least available setting.

Chapter 8: DynamoDB and Connecting Streams to Serving

  • DynamoDB is AWS's managed serving store: a partition key (+ optional sort key), RCU/WCU provisioned capacity or on-demand, adaptive capacity (smooths mild skew, but a single hot partition still has a throughput ceiling), GSIs/LSIs (secondary query shapes — GSIs cost extra capacity), DynamoDB Streams (CDC-out), and single-table design (model multiple entities in one table by key design). It trades Cassandra's open-source/tuning control for zero-ops (no compaction/repair to run) at a capacity price.
  • Connecting streams → serving (the JD's Problem 4): a Flink job (P04) computes a real-time feature and upserts it into Cassandra/DynamoDB — and because the store is last-write-wins and upserts are idempotent, the exactly-once sink (P04) and the serving store compose perfectly (a replayed upsert rewrites the same value). A Spark job (P06) backfills history into the same table — but rate-limited, because an unthrottled backfill hammers the cluster and blows the live read-path p99 SLO. That tension (backfill throughput vs serving latency) is a real principal design problem: throttle the backfill, schedule it off-peak, and protect live reads.

Lab Walkthrough Guidance

Lab 01 — Cassandra Data Modeler, suggested order:

  1. quorum / consistency_is_strong / level_nodes / tolerated_failures — the consistency dial (R+W>RF; what each level survives).
  2. partition_cells / partition_health — ok/warn/danger by cells + bytes.
  3. hot_partition_factor — skew detection (P01, again).
  4. tombstone_scan_ok — the 100k fail threshold.
  5. choose_compaction_strategy — TWCS (time-series) / LCS (read) / STCS (write).
  6. bucket_key — bound time-series partition growth.

Success Criteria

You are ready for Phase 12 when you can, from memory:

  1. Contrast serving vs analytics and say why each needs the other.
  2. Explain Cassandra's masterless ring, RF, and tunable consistency.
  3. State the query-first modeling rule and why there are no joins.
  4. Diagnose large vs hot partitions and give the fixes (bucket, salt, re-key).
  5. Explain the LSM write path, tombstones, and the TTL/queue tombstone trap.
  6. Pick a compaction strategy for write-heavy / read-heavy / time-series workloads.
  7. Choose a consistency level (incl. LOCAL_QUORUM multi-region) for a query.
  8. Wire a Flink upsert + rate-limited Spark backfill without blowing the read SLO.

Interview Q&A

Q: How do you model a "messages" feature in Cassandra? By the reads, not the entity. If the app needs "latest messages for a user" and "messages in a channel," that's two tables: one partitioned by user_id (clustered by time desc) and one by channel_id (clustered by time), each storing the message — denormalized, written twice. No joins, no ad-hoc filters. I'd also bound partition growth: if a user or channel can accumulate unbounded messages, bucket the partition key by time (e.g. channel_id + month) so no partition exceeds the size limits. The whole design starts from the lookup and works backward to the partition key.

Q: Reads were fine, then started failing with no deploy. What happened? Classic tombstone threshold. A delete- or TTL-heavy table (often a queue-like access pattern) accumulated tombstones faster than compaction removed them, and a read had to scan past more than ~100,000 of them, so Cassandra aborted the query. The fix depends on the cause: switch time-series-with-TTL tables to TWCS (drops whole expired SSTables without per-row tombstones), redesign away from the queue/range-over-deleted-data pattern, and add a tombstones-per-read alert (P13) so you see it building before it fails. The maddening part is that nothing in the app changed — the data shape crossed a threshold.

Q: When do you use QUORUM vs ONE vs ALL? Per the data's needs. QUORUM (with R+W>RF, e.g. RF=3) is the default for data that needs read-your-writes while tolerating a node down — strong and available. ONE for data where staleness is fine and latency/availability matter most (a "recently viewed" list). ALL almost never — it's the least available (one node down = the operation fails). Multi-region uses LOCAL_QUORUM to get quorum within the local DC without paying cross-region round-trips. The point: consistency is a per-query dial (P01), not a global "safe" setting.

Q: Stream computes features → Cassandra, and you also need to backfill history. Risks? Two. First, idempotency: the Flink upserts must be safe to replay (they are, since Cassandra is last-write-wins — a re-delivered upsert rewrites the same value), so the exactly-once sink and the store compose cleanly. Second, and the real trap: the Spark backfill must be rate-limited. An unthrottled backfill writing history at full cluster speed will saturate Cassandra and blow the live read-path p99 SLO — you'd take down serving to load history. So I throttle the backfill, run it off-peak, possibly to a separate token-aware writer, and watch the read latency the whole time. Backfill throughput vs serving latency is the tension to design around.

References

  • Jeff Carpenter & Eben Hewitt, Cassandra: The Definitive Guide — modeling, consistency, ops
  • Apache Cassandra docs — data modeling, compaction, consistency levels, tombstones
  • DataStax data-modeling guides (query-first methodology, chebotko diagrams)
  • Amazon DynamoDB developer guide — capacity, adaptive capacity, single-table design, Streams
  • Kleppmann, DDIA Ch. 3 (LSM-trees / SSTables) & Ch. 5–6 (replication, partitioning) — P01/P08