Lab 02 — Compatibility Burden Analysis

Phase: 03 — Lessons from Large Apache Systems
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 4–5 hours
Type: Code Reading / Document
Repository: apache/hadoop or apache/hive
Output: compatibility-analysis-[interface-name].md

Goal

Analyze a real, long-lived Apache interface and document the compatibility burden it has accumulated — the constraints it imposes on new implementations, the semantic guarantees it cannot enforce on all backends, and the design choices that would have reduced this burden.

After this lab you will understand why committers are conservative about API changes and why "just add a method" is rarely simple in a project with thousands of downstream users.

Choose One Interface

Recommendation: Option A (FileSystem) is better for engineers who know POSIX filesystems. Option B (SerDe) is better for engineers who've worked with Hive or custom data formats.

Setup

# Option A — Hadoop FileSystem
git clone https://github.com/apache/hadoop.git
cd hadoop
git checkout rel/release-3.3.6   # stable recent release

# The interface file:
# hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

# Option B — Hive SerDe
git clone https://github.com/apache/hive.git
cd hive
git checkout rel/release-3.1.3

# The interface file:
# serde/src/java/org/apache/hadoop/hive/serde2/Deserializer.java
# Related: AbstractSerDe.java, ObjectInspector.java

Steps

Step 1 — Read the Interface in Full (45 min)

Open your chosen interface file. Read every method:

1. For each abstract/non-default method, write one sentence describing what it is supposed to do
2. Note the parameter types and return types
3. Identify which methods are the "core contract" (must be implemented) vs which are optional extensions
4. Note the @Deprecated annotations — each one is a story

# Count abstract methods (Java)
grep -c "abstract" FileSystem.java

# Find deprecated methods
grep -n "@Deprecated" FileSystem.java

# Find methods added in specific Hadoop versions (look for @since in Javadoc)
grep -n "@since" FileSystem.java

Save an annotated method list in ## Interface Methods.

Step 2 — Count and Categorize the Implementations (30 min)

For FileSystem: find all production implementations in the Hadoop ecosystem:

# In the Hadoop repo:
find . -name "*.java" -exec grep -l "extends FileSystem" {} \;

# Key implementations to find:
# - LocalFileSystem (POSIX local disk)
# - DistributedFileSystem (HDFS)
# - S3AFileSystem (AWS S3)
# - GCSFileSystem (Google Cloud Storage — in hadoop-connectors repo)
# - AzureBlobFileSystem (Azure — in hadoop-azure module)
# - FTPFileSystem
# - ViewFileSystem (federation namespace)

For each implementation, note:

• Which methods are overridden
• Whether the implementation fully honors the documented semantics

Save this in ## Implementations.

Step 3 — Find the Semantic Gaps (60–90 min)

This is the core task. Find methods where the documented semantics cannot be fully honored by some implementations.

For FileSystem — key semantic gaps to find:

1. rename(Path src, Path dst): Documented as atomic (POSIX semantics). S3 implements as copy + delete. Find the Hadoop JIRA where this was debated.
2. delete(Path f, boolean recursive): Find the consistency semantics on S3 vs HDFS.
3. append(Path f, ...): Find which implementations return UnsupportedOperationException.
4. listStatus(Path f): On S3, there's no such thing as a "directory" — listing is eventually consistent. Find how S3AFileSystem handles this.

For each semantic gap:

• What does the Javadoc promise?
• What does the S3 (or other) implementation actually do?
• What breaks if caller code relies on the promised behavior?
• Is there a JIRA or mailing list thread about this?

For SerDe — key semantic gaps to find:

1. Writable input type: The deserialize(Writable blob) signature assumes the data arrives as a Hadoop Writable. How does this constrain column-oriented formats like ORC and Parquet?
2. ObjectInspector pattern: The getObjectInspector() method returns an ObjectInspector tree describing the schema. How does this interact with schema evolution?
3. initialize(Configuration conf, Properties tbl): All configuration is passed as a flat Properties map. What is lost?

Save your findings in ## Semantic Gaps.

Step 4 — Find the Compatibility Constraints (45 min)

Look at the interface's git history to understand the evolution:

# Interface evolution over time
git log --oneline -- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java | head -40

# What changed in a specific commit
git show <commit-hash> -- [file]

# Find when a specific method was added
git log -S "public abstract FileStatus getFileStatus" --oneline

For each meaningful change in the last 5–10 years:

• Was the change backward-compatible?
• Was it a new abstract method (breaking), a default method (safe), or a deprecation?
• What required a major version bump?

Save in ## Evolution History.

Step 5 — Write the Analysis (30–45 min)

Fill out compatibility-analysis-template.md. Your analysis should answer:

1. What is this interface's core abstraction?
2. What semantic guarantees does it make that cannot be honored by all backends?
3. What would you change if you could start fresh, with only the knowledge available when it was designed?
4. What is the cost of those semantic gaps in production systems?

Expected Output

lab-02-compatibility-burden/
├── README.md                                        ← this file
└── compatibility-analysis-[interface-name].md       ← your completed analysis

Verification

Your analysis is complete when:

1. You can name at least 3 methods with semantic gaps and explain the gap precisely
2. You can explain why adding a new abstract method to FileSystem would break thousands of programs
3. You can describe an alternative interface design that would reduce the compatibility burden
4. You can answer: "Why doesn't Apache just release a v2 of this interface?"

Talking Points

• "Why not just release FileSystem v2?" S3AFileSystem is in production at AWS, Google, Netflix, Databricks, and every major cloud vendor. Each has their own patched version. A v2 that breaks the interface means all of them have to migrate simultaneously — coordination that's effectively impossible to organize.
• "What's wrong with Writable?" It's a Hadoop-specific serialization format from 2003. Every modern system (Arrow, Protobuf, Avro, ORC, Parquet) uses a different format. A SerDe interface that takes Writable cannot efficiently read Parquet without converting through Writable first.
• "What would a better interface design look like?" Apache Arrow's RecordBatch as the exchange format; backends implement RecordBatchReader. No Writable. Schema as a first-class object, not Properties. Default no-op implementations for optional capabilities. This is essentially what Apache Iceberg's FileIO interface does for storage.

Resume Bullet

"Analyzed backward compatibility constraints in a long-lived Apache interface (Hadoop FileSystem or Hive SerDe); documented semantic gaps between the API contract and real-world backend behavior, evolution history, and alternative design approaches for a lower-maintenance interface."