Lab 01 — Post-Mortem: A Known Design Failure

Phase: 03 — Lessons from Large Apache Systems
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 3–4 hours
Type: Research / Document
Output: post-mortem-[failure-name].md

Goal

Produce a structured post-mortem for one real design failure in the Apache Hadoop ecosystem. The post-mortem must explain: what was built, why it was built that way, at what scale it failed, what the failure cost, how it was fixed, and what engineering lessons generalize beyond this specific case.

This is not a blame exercise. It is a learning exercise. The engineers who built these systems were excellent. The failures are interesting precisely because they were not caused by incompetence — they were caused by constraints that seemed reasonable at design time becoming wrong at scale.

Choose One Failure

OptionSystemFailureDifficulty
AHadoop HDFSSingle NameNode as both availability SPOF and namespace scale ceiling⭐⭐
BHiveMetastore RDBMS as write contention and partition enumeration bottleneck at scale⭐⭐⭐

Recommendation: If you are less familiar with Hadoop internals, choose Option A. The NameNode SPOF is well-documented, the fix (HA) is concrete, and the failure mode is easy to reason about. Option B requires understanding more of the query execution stack.

Research Sources

Option A — Hadoop NameNode SPOF:

Option B — Hive Metastore Bottleneck:

Steps

Step 1 — Read the Primary Sources (60–90 min)

Read at least 2 primary sources for your chosen failure. Take notes on:

  1. When was this system built, and what problem did it solve?
  2. What was the original scale assumption?
  3. At what scale did the failure become apparent?
  4. What were the symptoms that users experienced?
  5. How long did it take to fix? What was the fix?

Step 2 — Understand the Technical Root Cause (30 min)

Go beyond "the NameNode was a SPOF." Understand:

  • For Option A: Why does a single JVM process create both availability and scale problems? What is the heap ceiling? What does JournalNode quorum buy you?
  • For Option B: Why does partition enumeration hit the metastore so hard? What SQL queries does Hive run against the metastore schema? What is the contention point?

Step 3 — Analyze the Fix (30 min)

Read the fix (HA for Option A, or partition pruning/LLAP/table formats for Option B):

  • What architectural change was required?
  • What was the migration cost for existing clusters?
  • Did the fix fully solve the problem, or introduce new constraints?

Step 4 — Extract the Generalizable Lessons (20 min)

What does this failure teach you that applies beyond this specific system? Think about:

  • Centralization and its limits
  • Scale changes the dominant constraint
  • Backward compatibility and migration cost
  • The cost of tight coupling between components

Step 5 — Write the Post-Mortem (30–45 min)

Fill out post-mortem-template.md for your chosen failure. Rename it to post-mortem-namenode-spof.md or post-mortem-hive-metastore.md.

Expected Output

lab-01-post-mortem/
├── README.md                           ← this file
└── post-mortem-[failure-name].md       ← your completed post-mortem

Verification

Your post-mortem is complete when:

  1. You can explain the failure from first principles in 3 minutes without notes
  2. Your technical root cause section contains a specific class, process, or data structure — not just "it didn't scale"
  3. Your generalizable lessons are stated as patterns applicable to other systems — not just observations about Hadoop
  4. You can answer: "What would you have done differently at design time, given the information available then?"

Talking Points

  • "Why did it take so long to fix the NameNode SPOF?" The HA fix required a new JournalNode quorum service, ZooKeeper integration, DataNode dual-reporting, and fencing — all while maintaining backward compatibility with existing clients. The engineering was not hard; the risk management was.
  • "Why not just use a distributed metadata store from the start?" In 2006, Chubby (Google's distributed lock service) was not open source and ZooKeeper didn't exist yet. The single-node design was reasonable given what was available.
  • "What replaced the Hive Metastore?" Modern table formats (Apache Iceberg, Delta Lake, Apache Hudi) store table metadata alongside the data files, eliminating the central metastore dependency for reads. But the metastore is still used for discoverability and catalog integration.

Resume Bullet

"Conducted structured post-mortem analysis of a production-scale Apache system design failure (Hadoop NameNode SPOF or Hive Metastore bottleneck); documented root cause, failure timeline, fix architecture, and generalizable distributed systems lessons."