🛸 Hitchhiker's Guide — Phase 01: Distributed Systems Foundations

Read this if: you want the distributed-systems vocabulary and numbers fast, before the WARMUP derives them slowly. These are the ideas that make Kafka, Flink, Spark, Cassandra, and Iceberg stop being five manuals and become one.

0. The 30-second mental model

Everything is a log (append-only, offset-addressed). You scale it by partitioning (per-key order only). You survive machine death by replicating (R + W > N math). The network lies (loses, dups, reorders, delays), so you never trust wall clocks and you make re-processing a no-op (idempotency) — which is the only "exactly-once" that exists. One sentence: assume reorder + duplication + partial failure as normal, and design so they don't change the answer.

1. The five engines, one idea each

Learn the idea, recognise the engine.

2. The numbers to tattoo

strong consistency:    R + W > N      (R + W = N is NOT strong — off by one)
failure tolerance:     N − quorum     (RF=3, W=2 → survive 1)
skew factor:           max_load / mean_load   (1.0 = balanced)
consistent hashing:    add a node → ~1/N keys move  (vs % N → ~all move)
Little's Law:          L = λ × W      (in-flight = rate × latency)
ordering:              per-partition only; never total at scale

3. Time: the rule that saves you

Never order distributed events by wall clock. Clocks drift, NTP jumps backwards. Order by: offsets (within a partition), Lamport (causality ⇒ order), vector clocks (also detects concurrent), or event-time + watermarks (P04). Wall clocks are for human display and SLA stopwatches only.

• Lamport: local = max(local, received) + 1. Gives order; can't prove causality.
• Vector: per-node counters. Tells you happens-before vs concurrent = conflict detection.

4. Partitioning gotchas (the same bug, four costumes)

• Salted hash bug: using hash() (salted per process) for partitioning → keys reshuffle on restart → per-key ordering destroyed. Use FNV/Murmur/CRC.
• Hot partition: low-cardinality / Zipfian key → one shard melts. Same in Kafka, Kinesis, Cassandra, Spark. Fix: better key, salt/bucket, or consistent hashing.
• % N resharding: changing N remaps almost every key. Consistent hashing moves ~1/N.

5. CAP is a trap; say PACELC

• CAP: during a partition, choose C or A. (Rare event.)
• PACELC: if Partition → C/A; Else → Latency or Consistency. The Else clause is the tradeoff you tune every day (leader read = consistent+slow; follower read = fast+maybe stale).

6. The impossibility results (and the practical escape)

• Two Generals: can't guarantee agreement over a lossy channel → can't do exactly-once delivery. Escape: change the goal to exactly-once effect (idempotency).
• FLP: async consensus can't guarantee termination with a faulty node. Escape: real systems (Raft/KRaft) use timeouts to make progress in practice.
• Gray failure: a node that's slow, not dead, still passing health checks. The worst outages. Most of on-call is catching these.

7. Exactly-once, decoded

at-most-once   = no retries        → may lose
at-least-once  = retry till acked  → may dup   ← practical default
exactly-once   = at-least-once + (idempotency | transactions) = exactly-once EFFECT

Buy it with: dedup-by-stable-id (bounded! late dups past the window slip through — a real tradeoff), idempotent sinks (upsert/set-union/max), or the outbox pattern. Re-earned at every boundary.

8. Backpressure: unbounded buffer = bug

When downstream is slow, slow the producer (bounded queue + backpressure) — don't grow an unbounded buffer (that's an OOM with extra steps). When you can't slow the source, shed load on purpose with a metric. Rising lag and rising checkpoint time are backpressure in disguise (P04).

9. Beginner mistakes that mark you in interviews

1. Ordering events by event.timestamp (wall clock) and trusting it.
2. Saying "exactly-once" as if it's a network/platform freebie.
3. hash() (or any salted/unstable hash) for partitioning.
4. "Strong + highly available" with no mention of partitions / PACELC.
5. Quorum off-by-one: thinking R + W = N is strong.
6. Unbounded queues "for performance."
7. Assuming a node is either up or down (forgetting gray/slow failure).

10. How this phase pays off later

• Partitioning + skew → P02 (Kafka keys, Kinesis shards), P06 (Spark skew), P11 (Cassandra partitions). One mental model, four phases.
• Exactly-once effect → P04 (Flink sinks), P05 (CDC), P09 (Iceberg commits), P11 (upserts).
• Quorum math → P02 (ISR/acks), P11 (consistency levels), P14/P15 (multi-region).
• Backpressure → P04 (the central streaming health signal).
• Logical time → P04 (watermarks), P11 (conflict resolution).

Read the WARMUP slowly, build the kit, then go to P02 where the log becomes Kafka for real.