🛸 Hitchhiker's Guide — Production Systems: Consensus, Partitioning, Builds

Read this if: You can write Java but want to explain why distributed systems make the design choices they do, and how those choices appear in Apache codebases.

0. The 30-second mental model

                    ┌─────────────────────────────────────────────┐
                    │              PRODUCTION SYSTEM               │
                    └─────────────────────────────────────────────┘
                            │               │              │
                     ┌──────▼──────┐ ┌──────▼──────┐ ┌────▼────────┐
                     │  CONSENSUS  │ │ PARTITIONING│ │    BUILDS   │
                     │             │ │             │ │             │
                     │ Who is in   │ │ Which shard │ │ Who owns    │
                     │ charge?     │ │ gets this   │ │ this code?  │
                     │             │ │ key?        │ │             │
                     │ Raft: terms │ │ murmur2 →   │ │ Maven multi-│
                     │ + majority  │ │ partition   │ │ module POM  │
                     │ vote        │ │ index       │ │             │
                     └─────────────┘ └─────────────┘ └─────────────┘

These three concerns appear together in every large-scale Apache project:

• ZooKeeper / Kafka controller / HDFS NameNode all use consensus to elect a leader
• Kafka producers use murmur2 to deterministically route records to partitions
• Apache Hadoop / Flink / Kafka all use multi-module Maven builds with shared BOMs

1. Raft Leader Election

1.1 The Problem Raft Solves

A cluster of N nodes must agree on which single node is the leader at any time. The leader is authoritative for all writes. If the leader fails, the cluster must elect a new one without human intervention — and without electing two leaders simultaneously (split brain).

1.2 Raft's Three Node States

      ┌──────────────────────────────────────────────┐
      │                                              │
      │  FOLLOWER  ──election timeout──▶  CANDIDATE  │
      │      ▲                               │       │
      │      │◀──── receives heartbeat ──────┘       │
      │      │                               │       │
      │      └───────── wins majority ──▶  LEADER    │
      │                                     │       │
      │  (all nodes start as FOLLOWER)      │       │
      └─────────────────────────────────────────────┘

1.3 Terms: Raft's Logical Clock

A term is a monotonically increasing integer. Every election attempt increments the term. Terms solve the "stale leader" problem:

Term 1: node-1 elected leader
Term 2: network partition; node-2 starts election, node-3 votes for it
         → node-2 is leader at term 2
Term 1 again?: node-1 comes back, still thinks it's leader at term 1
         → node-1 sees term=2 in any RPC → immediately steps down

Rule: If a node sees a term > its own, it must immediately update its term and revert to FOLLOWER. This guarantees there is never more than one leader per term.

1.4 Majority Vote

Why floor(N/2) + 1? Because any two majorities must overlap by at least one node. That overlapping node prevents two leaders from being elected in the same term.

5-node cluster:  majority = 3
Two candidates can each get at most:
  Candidate A wins nodes {1, 2, 3}   ← 3 = majority
  Candidate B wants nodes {1, 4, 5}  ← node 1 already voted for A!
  → Candidate B gets at most {4, 5} = 2 < 3 = majority → cannot win

1.5 Key Interview Points

• "Why random election timeout?" → Reduces probability of simultaneous elections (split vote)
• "What if two candidates split the vote?" → Both timeout, increment term, try again with new random timeout
• "What happens to a partitioned leader's writes?" → They are lost if not replicated to a majority before the partition. The new leader has the most up-to-date log by virtue of winning the vote.

2. Kafka Partitioning Algorithms

2.1 Why Deterministic Partitioning Matters

Kafka guarantees ordering only within a partition. If you send order-123 events and they must be processed in order, they must all go to the same partition. The producer must map the key "order-123" to the same partition every time, on every producer instance, across restarts.

This requires a deterministic, key-based partition function.

2.2 murmur2 Hash

Kafka's DefaultPartitioner uses murmur2, a non-cryptographic hash designed for hash-table lookups. It has two properties Java's String.hashCode() does not:

The algorithm processes 4 bytes at a time with a multiply-and-XOR mixing step:

// Simplified inner loop
int k = read4Bytes(data, i);
k *= M;
k ^= k >>> 24;   // ← avalanche: every bit of input affects every bit of output
k *= M;
h *= M;
h ^= k;

The full implementation is in MurmurPartitioner.java.

2.3 Round-Robin Partitioning

For messages with no key (e.g., log events where ordering doesn't matter), round-robin distributes load evenly across partitions using a single atomic counter:

(counter.getAndIncrement() & Integer.MAX_VALUE) % numPartitions

The & Integer.MAX_VALUE clears the sign bit to prevent negative partition indices when the counter wraps past Integer.MAX_VALUE.

2.4 Range Assignment (Consumer Group Protocol)

When a consumer group subscribes to a topic, Kafka must assign partitions to consumers. The Range strategy (Kafka's default) works as follows:

N = 5 partitions, C = 3 consumers, sorted: [c0, c1, c2]

rangeSize = 5 / 3 = 1   (integer division)
extras    = 5 % 3 = 2   (first 2 consumers get one extra)

Consumer 0: start=0, count=1+1=2 → partitions [0, 1]
Consumer 1: start=2, count=1+1=2 → partitions [2, 3]
Consumer 2: start=4, count=1+0=1 → partitions [4]

Why Range? It minimizes the number of partitions moved during a rebalance when consumers are added/removed one at a time (common pattern: rolling restart).

Tradeoff: With many topics, the first consumer tends to get more partitions (the "first consumer bias"). The Sticky and CooperativeSticky assignors solve this at the cost of more complex rebalance logic.

3. Multi-module Maven Builds

3.1 The Problem: Dependency Version Drift

Imagine 10 Maven modules all depending on jackson-databind. Over time:

• Module A declares version 2.14.2
• Module B declares version 2.15.0
• Module C forgets to declare it and inherits a transitive version 2.13.1

The resulting classpath is unpredictable. Integration tests pass on the developer's machine (which has version X in ~/.m2/repository) but fail in CI.

Solution: A parent POM with dependencyManagement pins all versions in one place. Child modules declare <groupId>/<artifactId> without a version, and Maven resolves the version from the parent's dependencyManagement section.

3.2 POM Inheritance Hierarchy

queue-parent/pom.xml     ← packaging=pom, defines <modules> and <dependencyManagement>
├── queue-api/pom.xml    ← <parent> → queue-parent; packaging=jar; no runtime deps
├── queue-core/pom.xml   ← <parent> → queue-parent; depends on queue-api
└── queue-app/pom.xml    ← <parent> → queue-parent; depends on queue-core; has exec jar config

3.3 Reactor Build Order

Running mvn verify from the parent triggers a reactor build:

1. Maven reads all pom.xml files declared in <modules>
2. Builds a dependency graph
3. Builds modules in topological order: api → core → app
4. If any module fails, the reactor stops

cd lab-03-multimodule-maven
mvn verify

# Output:
# [INFO] Reactor Build Order:
# [INFO]   queue-parent [pom]
# [INFO]   queue-api [jar]
# [INFO]   queue-core [jar]
# [INFO]   queue-app [jar]

3.4 BOM vs Parent POM

Apache projects typically publish a BOM artifact (e.g., kafka-bom, hadoop-bom) so downstream users can import consistent versions without inheriting the Apache project's parent POM configuration.

4. What Can Go Wrong in Production

5. Interview Questions — Concepts You Must Explain Out Loud

6. References

• Raft Paper — In Search of an Understandable Consensus Algorithm (Ongaro & Ousterhout, 2014)
• Apache Kafka Protocol Guide — partition assignment RPC specs
• Kafka DefaultPartitioner source
• Apache Maven — Introduction to the POM
• Maven: Guide to Working with Multiple Modules