Cheat Sheet — The Numbers, Formulas & Decision Rules

One page of the things you should be able to recite. Each links to the phase that derives it. Memorising the number is worthless without the why — but in an interview, having both is the difference between "sounds senior" and "is senior."

Delivery semantics (P01)

at-most-once   = fire & forget                 → may lose data
at-least-once  = retry until acked             → may duplicate   ← practical default
exactly-once   = at-least-once + idempotency/transactions = exactly-once EFFECT

Rule: "exactly-once" is a property of the effect on state, re-established at every boundary — not a network guarantee. The moment you write to an external store, you re-earn it (upsert by key / dedup id / outbox).

Watermarks & windows (P04)

watermark W      = max_event_time_seen − bounded_delay
window fires when  W ≥ window.end
window expires when W ≥ window.end + allowed_lateness   → late events → side output
tumbling start   = ts − (ts % size);  interval is [start, start+size)  (end exclusive)

The triangle you trade: first-result delay ↔ completeness ↔ state size.

Kafka / Kinesis sizing (P02)

Spark tuning (P06)

#tasks in a stage      = #input partitions (post-shuffle: spark.sql.shuffle.partitions, default 200)
target partition size  ≈ 128–256 MB
broadcast join when     small side ≲ spark.sql.autoBroadcastJoinThreshold (default 10 MB; raise w/ care)
executor memory split  = heap × spark.memory.fraction(0.6) → execution+storage (unified)
spill cause            = partition data > available execution memory → tune partitions, not just memory
skew fix               = AQE skew join (split big partitions) / salting / broadcast
small-file fix         = repartition/coalesce to target size before write; compaction job after

Idempotent write recipe: write to a staging path → atomic rename/commit → or use a table format (Iceberg/Delta) whose commit is atomic.

EMR / cloud cost (P07, P14)

Parquet / file layout (P08)

file → row groups (≈128 MB) → column chunks → pages (≈1 MB)
skipping: footer + row-group stats (min/max/null count) → predicate pushdown
encodings: dictionary (low-cardinality) → RLE/bit-packing; then compression (zstd/snappy)
target file size: 128 MB–1 GB;  avoid << 128 MB (small-file tax) and >> 1 GB (poor parallelism)

Cassandra (P11)

strong consistency: R + W > RF   (e.g. RF=3, W=QUORUM(2), R=QUORUM(2) → 2+2>3 ✓)
partition size: keep < 100 MB and < ~100k cells; model around the read query
tombstones: deletes/TTL create tombstones → read amplification; alert on tombstones/read
compaction: STCS (write-heavy) | LCS (read-heavy, more write amp) | TWCS (time-series/TTL)
hot partition: too-coarse partition key → add a bucketing component to the key

Iceberg / lakehouse (P09)

commit = new metadata.json → manifest list → manifests → data files (immutable snapshot)
isolation = optimistic concurrency; conflicting commits retry against the new snapshot
time travel = query AS OF snapshot-id / timestamp
maintenance = rewrite_data_files (compaction) + expire_snapshots + remove_orphan_files
partition evolution = change spec without rewriting history (hidden partitioning)

Consistency / distributed systems (P01)

CAP: under a network Partition, choose Availability or Consistency
PACELC: Else (no partition), choose Latency or Consistency  ← the everyday tradeoff
quorum: R + W > N gives read-your-writes on that register
clocks: never trust wall clocks for ordering → use logical/Lamport/vector clocks or sequence ids

Reliability targets (P13) — example SLOs from the JD

99.9%  of critical streams process events within 60s
99.99% of accepted events are durably persisted
critical lakehouse tables queryable within 15 min of event arrival
no breaking schema change reaches prod without a compatibility check
every production dataset has an owner, SLA, lineage, and quality policy

The tradeoff dials you will always be asked to turn (P00, P15)

low latency ⟷ low cost
exactly-once ⟷ operational simplicity
flexible schema ⟷ safe contract
self-service ⟷ governance
streaming freshness ⟷ replay/batch correctness
cloud-native lock-in ⟷ open-source portability
developer freedom ⟷ platform standardization

Principal move: name the dial, quantify both ends with the workload's numbers, then choose — and write it in an ADR.

Definitions for every term above are in GLOSSARY.md.