Phase 01 — Distributed Systems Foundations for Data
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (30–40 hours) Prerequisites: Phase 00 (the lifecycle and the five forces)
Why This Phase Exists
The JD describes a person who can "debug at the byte, partition, checkpoint, offset, file, and query-plan level" and reason about "data correctness, ordering, replayability, idempotency, deduplication, schema compatibility … failure recovery." Every one of those words is a distributed systems word. Kafka, Flink, Spark, Cassandra, and Iceberg are not five things to learn — they are five expressions of the same handful of distributed systems ideas. Learn the ideas once, here, and the rest of the track is "how does this specific engine implement the thing I already understand."
This is the phase that separates engineers who use distributed systems from engineers who reason about them. The former are surprised by reordering, duplication, partial failure, and clock skew. The latter expect all four and design so they don't matter.
Concepts
- The log as the one abstraction under everything — append-only, ordered, offset-addressed; batch and streaming are two read modes of it (the Kreps/Kleppmann framing).
- Time: wall-clock vs logical clocks (Lamport, vector); why you must never order distributed events by wall time; happens-before vs concurrent; causality.
- Partitioning: stable hashing, per-key ordering, skew / hot partitions, and why
% Nis a resharding trap (consistent hashing as the fix). - Replication: leader/follower, quorums (R + W > N), ISR, sync vs async, and the durability/availability arithmetic (tolerated failures = N − quorum).
- Consistency models: linearizable → sequential → causal → eventual; read-your-writes; CAP (under a partition: C or A) and PACELC (else: latency or consistency — the everyday tradeoff CAP omits).
- Failure: the eight fallacies of distributed computing; partial failure; the Two Generals / FLP impossibility results and what they mean practically; gray failures.
- Delivery & effect: at-most/at-least/exactly-once; idempotency, dedup, the outbox pattern; exactly-once-effect as the only achievable "exactly-once."
- Backpressure & flow control: the queueing-theory reason unbounded buffers are bugs (Little's Law, bounded queues, load shedding).
Labs
Lab 01 — Partitioner, Idempotency & Exactly-Once Kit (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build the four primitives — deterministic partitioning + skew measurement, Lamport/vector clocks, bounded dedup → exactly-once effect, and quorum/replication math — and prove that reordered + duplicated delivery yields identical state |
| Concepts | Stable hashing, hot partitions, logical time, causality vs concurrency, idempotency, R + W > N, failure tolerance |
| How to Test | pytest test_lab.py -v — 14 tests incl. order-independence and exactly-once under an unreliable channel |
| Talking Points | Why never hash() for partitioning? What does a vector clock tell you that a Lamport clock can't? Why is bounded dedup a correctness tradeoff? |
| Resume bullet | Implemented partitioning, logical-clock causality, bounded idempotent dedup, and quorum-durability primitives; verified exactly-once effect under reordered/duplicated delivery |
→ Lab folder: lab-01-exactly-once-kit/
Extension Project A — Consistent Hashing Reshard (spec)
Implement a hash ring with virtual nodes; measure the fraction of keys that move when a
partition is added vs % N (the difference is the whole reason resharding is survivable).
Extension Project B — Two Generals Demonstrator (spec)
Simulate a lossy channel and prove empirically that no fixed number of acknowledgements achieves certain agreement — then show how at-least-once + idempotency sidesteps the impossibility by changing the goal from "agree" to "same effect."
Integrated-Scenario Hooks
- Hot partition (this phase's skew math) → Kafka key choice (P02), Kinesis shards (P02), Cassandra partition keys (P11), Spark data skew (P06). It is the same problem four times.
- Exactly-once effect (this phase's dedup) → Flink sinks (P04), CDC apply (P05), Iceberg commits (P09), Cassandra upserts (P11).
- Quorum math → Kafka ISR/
acks(P02), Cassandra consistency levels (P11), multi-region (P14/P15).
Guides in This Phase
Key Takeaways
- Five engines, one set of ideas: partitioning, logical time, replication, consistency, failure, idempotency, backpressure.
- Wall clocks lie; order with logical clocks or sequence ids.
- "Exactly-once" is exactly-once effect, re-earned at every boundary.
- Durability and consistency are arithmetic (R + W > N; N − quorum), not adjectives.
Deliverables Checklist
-
Lab 01 implemented; all 14 tests pass against
solution.py - You can state CAP and PACELC and give a data example of each choice
- You can explain the same "hot partition" problem in Kafka, Cassandra, and Spark terms
- Extension A or B attempted