Phase 03 — Lessons from Large Apache Systems
Why this phase exists: Before you contribute to a large system, you need to understand how large systems fail — not to be pessimistic, but because the committers and PMC members you'll be working with carry this institutional memory. They've seen the same mistakes made across generations of distributed systems projects, and they'll hold your patches to a higher standard because of it.
What You'll Learn
- How Hadoop 1.x's single NameNode became a systemic SPOF and why fixing it took years
- How Hive's metastore became a metadata bottleneck at scale and the architectural decisions that caused it
- How Tez was designed to fix MapReduce's DAG limitations — and what it got right and wrong
- Why Spark originally avoided Hadoop's abstractions, and how compatibility pressure brought some of them back
- The recurring patterns of distributed systems failure: over-abstraction, centralization, backward-compatibility debt, coordination overhead
- How to read a technical design failure as a learning document, not a post-mortem
Concepts Covered
| Concept | Where it appears |
|---|---|
| Single point of failure (SPOF) and its cost at scale | Hadoop 1.x NameNode |
| HDFS HA: Active/Standby NameNode, JournalNodes, ZooKeeper fencing | Hadoop 2.x |
| Metadata bottleneck: why a single in-memory namespace doesn't scale | HDFS Federation |
| YARN: the resource management separation from MapReduce | Hadoop 2.x |
| MapReduce's two-phase limitation and disk materialization | Tez motivation |
| DAG of tasks vs map-reduce pairs | Tez vs MapReduce |
| Hive metastore: what it stores, why it serializes, why it becomes a bottleneck | Hive scaling |
| SerDe interface and the cost of backward compatibility | Hive |
| Catalyst vs Hive's query planner | Spark SQL motivation |
| Backward compatibility as an engineering constraint | All projects |
Labs
| # | Lab | Type | Output |
|---|---|---|---|
| 01 | Post-Mortem: A Known Design Failure | Research / Document | Structured post-mortem |
| 02 | Compatibility Burden Analysis | Code Reading / Document | Interface compatibility analysis |
Deliverables Checklist
- Post-mortem document for one chosen design failure (Hadoop NameNode SPOF or Hive metastore bottleneck)
- Compatibility burden analysis for one chosen interface (Hadoop FileSystem API or Hive SerDe)
- You can explain the chosen failure from first principles — what was built, why it failed, what the fix required, what lessons generalize
Interview Relevance
Staff/Principal Engineer interviews at companies running Spark/Hadoop at scale (Databricks, Cloudera, LinkedIn, Uber, Netflix, Meta) will ask:
- "What are the biggest architectural mistakes you've seen in distributed systems?"
- "How do you think about backward compatibility when designing an API?"
- "What's the cost of making a distributed system highly available? What are the tradeoffs?"
- "Why do metadata services become bottlenecks? How do you avoid it?"
The engineers interviewing you have often been personally involved in these failures. This phase gives you the vocabulary to engage at their level.