Phase 02 — JVM Codebase Navigation
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 1–2 weeks
Roles supported: All tracks — mandatory prerequisite for Phases 07–11.
Primary project: Apache Spark
Why This Phase Exists
Becoming an Apache committer requires the ability to navigate codebases you did not write, in subsystems you have never touched, under time pressure, without a guide. This is a skill almost never taught explicitly and almost always assumed.
The engineers who become committers are the ones who can open a 2 million-line Java/Scala codebase, find the relevant entry point in 10 minutes, trace the call path through four modules, understand the failure mode, and write a correct fix — all before the issue gets stale. This phase teaches that skill methodically.
You will set up Apache Spark for local development, learn to read its Maven module hierarchy, trace a complete execution path from user-facing API to internal executor, and then perform a full issue trace on a real historical bug: reading the JIRA, locating the broken code path, understanding the fix, and explaining why it was correct.
Concepts
- Maven multi-module project structure in large Apache codebases
- Reading
pom.xmlhierarchies: parent POMs, dependency management, module declarations - IntelliJ IDEA project import for Apache Spark (Scala + Java mixed modules)
- JVM codebase navigation strategies: API-surface-first, interface-following, test-reading
- Apache Spark module topology:
core/,sql/catalyst/,sql/core/,resource-managers/yarn/,network/,common/ - The Spark execution model: SparkSession → Logical Plan → Physical Plan → RDD → Task → Executor
- Key Spark classes and their responsibilities:
SparkContext,DAGScheduler,TaskScheduler,ShuffleManager,BlockManager - Reading git history to understand a bug fix:
git log,git show,git blame - Connecting a JIRA issue to a commit: PR description, JIRA link, release note entry
- Writing a "bug journey" document: problem, trigger, code path, fix, correctness argument
Labs
Lab 01 — Codebase Spelunking
| Field | Value |
|---|---|
| Goal | Clone Apache Spark, understand its Maven module structure, and produce an annotated trace of spark.read.csv("path") from the user API surface down to FileScanRDD.compute(). |
| Concepts | Maven multi-module structure; Spark module topology; logical → physical plan pipeline; IntelliJ IDEA navigation. |
| Project | apache/spark — branch master or latest stable tag (e.g., v3.5.1) |
| Steps | 1) Clone and build index (see setup guide in lab README). 2) Read the top-level pom.xml and map the 15 primary modules. 3) Start at SparkSession.read() — trace through DataFrameReader, DataSource, CSVFileFormat, FileSourceScanExec, FileScanRDD. 4) Annotate each class with one sentence explaining its responsibility. 5) Draw the module dependency graph for the classes you touched. |
| Output | A completed execution-trace.md with: annotated class list, module dependency diagram (ASCII), and a 3-sentence explanation of the logical→physical plan boundary. |
| How to Verify | Your trace should match the actual class hierarchy — verify by running Find Usages on DataFrameReader.load() in IntelliJ and confirming the call chain. |
| Talking Points | Why does Spark separate logical from physical plans? What is the Catalyst optimizer's role in this path? Why is FileScanRDD the right entry point for storage-side debugging? |
| Resume Bullet | "Navigated Apache Spark's 2M-line multi-module codebase; produced an annotated execution trace from SparkSession.read().csv() through Catalyst optimization to FileScanRDD.compute(), documenting module boundaries and responsibility of each layer." |
| Extensions | Trace a spark.sql("SELECT ...") query through the same pipeline. Where does it rejoin the CSV read path? |
Lab 02 — Issue Trace
| Field | Value |
|---|---|
| Goal | Select a real closed SPARK JIRA issue, locate its root cause in the Spark source tree, trace the broken code path, understand the fix, and produce a structured "bug journey" document. |
| Concepts | git log / git blame / git show; connecting JIRA to commits; reading diffs with context; root cause vs symptom. |
| Steps | 1) Select one of the pre-approved issues listed in the lab README. 2) Read the full JIRA description and all comments. 3) Find the fix commit via the JIRA "Commits" tab or PR link. 4) Read the diff: what exactly changed? 5) Navigate backward from the changed lines to find the root cause. 6) Complete the bug-journey-template.md. |
| Output | One completed bug-journey-[SPARK-XXXXX].md with: problem summary, trigger conditions, broken code path (class → method → line), fix description, correctness argument, and test coverage analysis. |
| How to Verify | Your root cause location should match the lines changed in the fix commit. Your correctness argument should explain why the fix resolves the root cause and not just the symptom. |
| Talking Points | How do you distinguish root cause from symptom in a distributed system bug? Why do Apache committers often ask "does this fix the root cause or paper over it?" What information in a JIRA issue tells you how severe a bug is? |
| Resume Bullet | "Performed end-to-end issue trace on a closed Apache Spark JIRA: located root cause in Spark internals using git blame and diff analysis, documented the broken code path, and produced a structured correctness argument for the accepted fix." |
| Extensions | Find the test added in the fix commit. Can you identify the boundary condition it covers? Would additional tests have caught this earlier? |
Deliverables Checklist
- Apache Spark cloned and project index built in IntelliJ
-
Top-level
pom.xmlmodule map documented (lab-01) -
Annotated execution trace:
SparkSession.read().csv()→FileScanRDD.compute()(lab-01) - Module dependency diagram drawn (lab-01)
- Logical→physical plan boundary explained in 3 sentences (lab-01)
- JIRA issue selected and fully read including all comments (lab-02)
- Fix commit located and diff read (lab-02)
- Root cause located to specific class/method/line (lab-02)
-
bug-journey-*.mdcompleted (lab-02)
Interview Relevance
- "Walk me through how a
spark.read.csv()call executes." (you can answer end-to-end) - "What is the Catalyst optimizer?" (you can explain its role in the logical→physical boundary)
- "How do you approach debugging an unfamiliar distributed codebase?" (you can describe the API-surface-first strategy)
- "What is a DAGScheduler?" (you can describe its role relative to TaskScheduler)
- "How do you find the root cause of a bug in a 2-million-line codebase you didn't write?" (you can describe the git blame → class trace → JIRA strategy)