Distributed Systems — Questions with Complete Answers

Staff/Principal data-infrastructure loops test whether you can reason about consensus, replication, partitioning, and failure from mechanism, not vocabulary. Every answer below ties back to a lab or a real Apache system.

Table of Contents

Consensus

Q: Explain Raft leader election precisely. What prevents two leaders in the same term?

Nodes are followers until an election timeout (randomized, e.g. 150–300 ms) fires without a heartbeat. The follower increments its term, becomes candidate, votes for itself, and requests votes. A node grants at most one vote per term (persisted), and only to a candidate whose log is at least as up-to-date (last term, then last index). A candidate needs a majority; two leaders in one term would require two disjoint majorities — impossible in one quorum set. Two leaders in different terms can transiently coexist, but the stale leader's AppendEntries are rejected by followers on term comparison, forcing it to step down. Randomized timeouts make split votes rare rather than impossible; a split vote just re-elects in the next term. You built and tested all of this in Phase 07 Lab 01.

Q: Why can't a Raft leader commit entries from previous terms by counting replicas?

The Figure-8 scenario in the Raft paper: an entry from an old term can be replicated to a majority and still be overwritten by a later leader whose log diverged. A leader may only advance commitIndex for entries of its own current term; older entries commit implicitly once a current-term entry on top of them commits. This is the subtlest rule in Raft and a favorite follow-up question.

Q: Where does ZooKeeper fit in the Hadoop/Kafka ecosystem, and what replaced it in Kafka?

ZooKeeper provides a replicated, totally-ordered, small-data store (ZAB protocol) used for coordination: HDFS/YARN HA failover via ephemeral-node leader locks, HBase meta, Kafka's old controller metadata. Kafka replaced it with KRaft — a Raft-based internal metadata quorum — eliminating the external dependency, the dual-system consistency gap (controller state vs ZK state), and scaling metadata to millions of partitions.

Replication and consistency

Q: Compare HDFS replication with Kafka ISR replication.

HDFS: pipeline replication — client writes a block through a chain of 3 datanodes; the namenode tracks block locations; acks flow back through the pipeline. Consistency is simple because blocks are immutable once closed. Kafka: leader-based log replication — followers fetch from the leader; the leader maintains the ISR (in-sync replica set); a message is committed when all ISR members have it; acks=all + min.insync.replicas=2 gives durability against one broker loss. The key design difference: HDFS replicates immutable blocks, Kafka replicates an ordered mutable log, so Kafka needs the high-watermark and leader-epoch machinery to prevent divergence on failover.

Q: What does "exactly-once" mean in Kafka/Spark, really?

There is no exactly-once delivery; there is exactly-once processing effect. Kafka: idempotent producers (sequence numbers per partition) remove duplicates from retries, and transactions make produce+offset-commit atomic across partitions. Spark Structured Streaming: replayable sources + deterministic computation + idempotent/transactional sinks gives end-to-end effective-once. If asked, always decompose into source replay, processing determinism, and sink idempotence.

Partitioning and rebalancing

Q: How does Kafka's default partitioner work and why murmur2?

partition = murmur2(keyBytes) % numPartitions (with the sign bit masked). Murmur2 is fast, non-cryptographic, and uniform across realistic key distributions — quality matters because a biased hash creates hot partitions that cap throughput at one broker. Keyless records use sticky batching (fill one partition's batch, then switch) to amortize batching overhead. You implemented and chi-square-tested this in Phase 07 Lab 02.

Q: What goes wrong when you increase the partition count of a keyed topic?

hash(key) % N changes for almost every key, so key→partition affinity breaks: ordering per key is no longer continuous across the boundary, and any state keyed by partition (consumer-local stores, Kafka Streams state stores) is suddenly wrong. The fixes are unpleasant — re-key into a new topic, or over-provision partitions up front. This is the classic "modulo is not consistent hashing" interview probe.

Q: Range vs round-robin consumer assignment — tradeoffs?

Range assigns contiguous partition blocks per topic; co-partitioned topics get aligned assignments (needed for joins) but it skews when partitions per topic ≪ consumers × topics. Round-robin spreads evenly but breaks co-partition alignment. Cooperative sticky assignment (modern default) minimizes movement during rebalances, which matters because every moved partition flushes caches and stalls consumption.

Scheduling and resource management

Q: Explain YARN's architecture and the capacity/fair scheduler tradeoff.

ResourceManager (scheduler + ApplicationsManager) arbitrates cluster resources; NodeManagers report and enforce per-node containers; each application runs its own ApplicationMaster that negotiates containers. The scheduler is policy-pluggable: capacity scheduler partitions the cluster into hierarchical queues with guaranteed capacities and elastic sharing (multi-tenant orgs); fair scheduler converges all running apps toward equal (weighted) shares (ad-hoc analytic clusters). The recurring design tension: utilization (let queues borrow) vs isolation (preemption to reclaim guarantees) — preemption cost is wasted work in killed containers.

Q: How does Spark's DAG scheduler decide stage boundaries?

Narrow dependencies (map, filter — each parent partition feeds one child partition) are pipelined into a single stage. Wide dependencies (shuffle: groupByKey, repartition, joins without co-partitioning) cut a stage boundary: the parent stage materializes shuffle files and the child stage fetches them. Stage failure granularity, speculative re-execution, and shuffle cost all follow from this cut. The interview follow-up is invariably "and what does AQE change?" — answer: post-shuffle statistics let Spark coalesce partitions, switch join strategies, and split skewed partitions at runtime.

Failure handling

Q: A long-running job fails on one task repeatedly on the same node. Walk through your reasoning.

Differentiate deterministic data problems from environmental ones. Same task id failing on different nodes → bad record/skew/code path (fix data handling, look at the partition's key range). Failing only on one node → environment (disk, memory, JVM version, local library); blacklist/decommission the node and check NodeManager health logs. Speculative execution masks slow nodes but not failing ones. Mention the observability you would check in order: application logs → executor/container logs → node health → cluster metrics for the time window.

Q: Why is retry-with-backoff insufficient as a fault-tolerance strategy?

Retries amplify load exactly when the system is least able to take it (retry storms), can violate idempotence assumptions (duplicate side effects), and convert partial failures into latency cliffs. Production-grade handling adds jittered exponential backoff, retry budgets, circuit breaking, idempotency tokens, and — most importantly — distinguishes retryable (transient, safe) from non-retryable (deterministic, unsafe) errors at the type level.

References