🛸 Hitchhiker's Guide — Phase 02: JVM Codebase Navigation

Read this if: You can write Java/Scala but you've never opened a 2-million-line production codebase and felt confident navigating it. By the end you should be able to clone Apache Spark, find any class from a user-facing symptom, trace it to its implementation, and explain the execution model from SparkSession to Executor without consulting the documentation.

0. The 30-second mental model

Apache Spark is a multi-module Maven project. Each module has a clear responsibility, and the modules form a dependency DAG (not a tree). Understanding this DAG is the first step to navigating the codebase.

          User API
             │
     SparkSession / SparkContext
             │
     ┌───────┴───────┐
     │               │
  SQL Path       RDD Path
     │               │
  Catalyst       DAGScheduler
  (optimizer)        │
     │          TaskScheduler
  Physical Plan       │
     │           TaskSetManager
  FileScanRDD         │
     │            Executor
  compute()           │
     │         TaskRunner.run()
  Iterator[Row]       │
                 task.run()

Three invariants to keep in mind throughout:

1. The boundary between logical and physical plans is the optimizer. Anything above FileSourceScanExec is logical. Anything from it downward is physical/runtime.
2. Scheduling is separate from execution. DAGScheduler breaks the job into stages. TaskScheduler assigns tasks to executors. The executor runs them.
3. Storage is abstracted behind RDDs. Whether you read CSV, Parquet, or JDBC, you eventually get an RDD[InternalRow] with a compute() method.

1. Apache Spark's Maven Module Structure

1.1 Reading the Top-Level pom.xml

Every Maven project has a pom.xml. In a multi-module project, the top-level pom.xml is the parent POM: it declares all child modules, sets shared properties (Java version, Scala version, plugin versions), and defines the dependency management section — but does not itself produce a JAR.

<!-- Simplified excerpt from Spark's top-level pom.xml -->
<modules>
  <module>core</module>
  <module>sql/catalyst</module>
  <module>sql/core</module>
  <module>sql/hive</module>
  <module>streaming</module>
  <module>mllib</module>
  <module>resource-managers/yarn</module>
  <module>resource-managers/kubernetes</module>
  <module>launcher</module>
  <module>network/common</module>
  <module>network/shuffle</module>
  <module>common/unsafe</module>
  <module>common/sketch</module>
  <module>common/kvstore</module>
  ...
</modules>

Each <module> is a subdirectory containing its own pom.xml with a <parent> pointing back to the root. This is the standard Maven multi-module pattern.

1.2 The Primary Spark Modules

1.3 The Dependency DAG

Modules depend on each other. The key rule: core has no dependency on sql/. The SQL layer depends on core, not the other way around. This means:

• You can read an RDD bug purely in core/ without touching SQL
• A DataFrame bug almost always involves both sql/catalyst/ and sql/core/
• A shuffle bug may touch core/, network/shuffle/, and resource-managers/yarn/

                         ┌─────────────┐
                         │  sql/hive   │
                         └──────┬──────┘
              ┌──────────────────┘
              ▼
         ┌─────────┐    ┌──────────────┐
         │sql/core │───▶│ sql/catalyst │
         └────┬────┘    └──────┬───────┘
              └────────────────┘
                       │
                       ▼
              ┌────────────────┐
              │      core      │◀── resource-managers/yarn
              └────────┬───────┘◀── resource-managers/k8s
                       │
              ┌────────┴──────────────┐
              │                       │
              ▼                       ▼
     ┌─────────────────┐    ┌──────────────────┐
     │  network/common │    │  common/unsafe   │
     └─────────────────┘    └──────────────────┘

2. Setting Up Apache Spark for Local Development

2.1 Prerequisites

# Required tools
java --version    # Java 11 or 17 (not 21+ for older branches)
mvn --version     # Maven 3.8+
scala -version    # Scala 2.12 or 2.13 (match your Spark branch)
git --version

# IntelliJ IDEA (Community edition is sufficient)
# Plugin required: Scala plugin

2.2 Clone and Build

# Clone the repository
git clone https://github.com/apache/spark.git
cd spark

# Check out a stable release tag (faster than building main)
git checkout v3.5.1

# Build only the modules you need (skip tests and docs to save time)
./build/mvn -DskipTests -Phive -Pyarn package -pl core,sql/catalyst,sql/core \
    -am 2>&1 | tail -50
# -pl : only these modules
# -am : also build modules they depend on
# This takes 8–15 minutes on a modern laptop

# Full build (takes 30–60 minutes; only do this once):
./build/mvn -DskipTests package

2.3 Import into IntelliJ IDEA

1. File → Open → select the spark/ directory
2. IntelliJ detects the Maven project and auto-imports modules
3. Wait for indexing to complete (5–15 minutes for initial index)
4. File → Project Structure → SDKs → confirm Java 11 or 17 is selected
5. Install Scala plugin if not already present (IntelliJ will prompt)

After indexing, you have:

• Go to Declaration (Cmd+B / Ctrl+B): jump from usage to definition
• Find Usages (Opt+F7 / Alt+F7): find every call site for a method
• Navigate to Class (Cmd+O / Ctrl+N): open any class by name
• Type Hierarchy (Ctrl+H): see all implementations of an interface

2.4 Running a Single Test

# Run one test class without building the whole project
./build/mvn test -pl sql/core -Dtest=DataFrameReaderSuite -DfailIfNoTests=false

3. JVM Codebase Navigation Strategies

3.1 The API-Surface-First Strategy

When you receive a bug report like "reading a CSV file with a null value in column 3 throws NPE," start at the public API, not at the crash stack trace.

1. Find the public API entry point
   └── SparkSession.read().csv("path")

2. Follow the method chain until you reach an interface or abstract class
   └── DataFrameReader.csv() → DataFrameReader.load() → DataSource.apply()

3. Find all implementations of the interface/abstract class
   └── CSVFileFormat implements FileFormat
   └── Use Type Hierarchy (Ctrl+H) on FileFormat to see all implementations

4. Narrow to the implementation your bug is in
   └── The NPE points at CSVFileFormat.buildScan() line 174

5. Now read the implementation in context

This strategy gets you to the right place faster than starting from a stack trace, because stack traces only show one path — but the real problem may be that the path shouldn't have been taken at all.

3.2 The Test-Reading Strategy

Before reading implementation code, read the tests for the class you're investigating. Tests tell you:

• What the expected behavior is (the passing tests)
• What edge cases were considered (the named test methods)
• What the error handling contract is (tests with assertThrows)
• What assumptions the implementor made (what is NOT tested)

# Find tests for a class
find . -name "*DataFrameReader*Test*" -o -name "*DataFrameReaderSuite*"
# Apache Spark uses ScalaTest, so test files end in "Suite"

3.3 The Interface-Following Strategy

Apache projects are heavily interface-driven. When tracing a call path:

SparkSession.sessionState.executePlan()
         │
         ▼
  QueryExecution.executedPlan   ← this is a SparkPlan
                                  SparkPlan is an abstract class
                                  Use Type Hierarchy to see all physical plans

Every time you hit an abstract method or interface, stop and ask: "Which implementation is actually running for my use case?" The answer is usually in:

• The factory method that constructs the object
• A pattern-match in a Rule class
• An injection point in SparkContext or SparkSession.Builder

3.4 The git Blame Strategy

When you find the specific line that causes a bug, use git blame to understand its history:

# Who last changed this line, and why?
git blame sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Show the full commit that changed a specific line
git show <commit-hash>

# Find all commits that touched a file
git log --oneline sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Find when a specific string was introduced
git log -S "nullValue" --oneline

git blame is underused by engineers who are new to large codebases. It tells you: the person who wrote this line, the commit that introduced it, and usually the JIRA number in the commit message.

4. The Spark Execution Model in Depth

4.1 The Full Call Path: spark.read.csv("path")

SparkSession.read()
    └── Returns a new DataFrameReader instance
            │
DataFrameReader.csv(path: String)
    └── Calls format("csv").load(path)
            │
DataFrameReader.load(path)
    └── DataSource.apply(sparkSession, "csv", paths = Seq(path))
    └── df = DataSource(sparkSession, className, paths).resolveRelation()
            │
DataSource.resolveRelation()
    └── fileFormat = DataSource.lookupDataSource("csv")  → CSVFileFormat
    └── Returns: HadoopFsRelation(location, ..., fileFormat = CSVFileFormat)
    └── Wraps in: LogicalRelation(HadoopFsRelation)
            │
LogicalRelation  ← this is the LOGICAL PLAN node
            │
            ▼  (when an Action is called, e.g., .show() or .count())
            │
QueryExecution.executedPlan
    └── Calls optimizer rules on the logical plan
    └── Produces: FileSourceScanExec  ← PHYSICAL PLAN node
            │
FileSourceScanExec.inputRDD
    └── Returns: FileScanRDD
            │
FileScanRDD.compute(partition, context)
    └── Opens file via Hadoop InputFormat
    └── Returns: Iterator[InternalRow]
            │
            ▼
  Iterator consumed by downstream operators
  (Project, Filter, Aggregate, etc.)

4.2 The Logical → Physical Plan Boundary

The Catalyst optimizer sits between the analyzed logical plan and the physical plan. It applies a sequence of rules (each a pattern-matching transform over the plan tree) to produce a more efficient plan before planning begins.

4.3 The Scheduling Model

Once an action triggers execution, this is what happens:

SparkContext.runJob()
    │
    ▼
DAGScheduler.runJob()
    └── Builds DAG of stages from the RDD dependency chain
    └── Determines stage boundaries (shuffle boundaries = stage boundaries)
    └── Submits stages in topological order
            │
TaskScheduler.submitTasks(taskSet)
    └── Assigns tasks to executors based on locality (PROCESS_LOCAL > NODE_LOCAL > RACK_LOCAL > ANY)
    └── Manages task retries on failure
            │
CoarseGrainedSchedulerBackend
    └── Sends task binary to executor via Netty transport
            │
Executor.launchTask()
    └── Deserializes the task
    └── Runs TaskRunner in a thread pool
            │
TaskRunner.run()
    └── task.run(taskAttemptId, attemptNumber, metricsSystem)
    └── Task is either a ShuffleMapTask or a ResultTask
            │
ShuffleMapTask.runTask()  OR  ResultTask.runTask()
    └── Calls rdd.iterator(partition, context)
    └── Writes output to shuffle files or returns result to driver

Key insight: DAGScheduler operates on RDD lineages. TaskScheduler operates on task sets. They communicate through a clean interface (DAGScheduler.taskStarted(), taskEnded(), etc.). Bugs often live at this boundary.

4.4 The BlockManager

Every executor has a BlockManager. It is responsible for:

• Caching RDD partitions in memory or disk (persist() / cache())
• Broadcasting variables (driver sends to executors via the block manager)
• Shuffle write (ShuffleMapTask writes shuffle blocks)
• Shuffle read (tasks fetch shuffle blocks from peer executors)

The BlockManager is the part of Spark most involved in out-of-memory errors, shuffle fetch failures, and broadcast issues. Many production bugs live here.

5. Reading a JIRA Issue and Connecting It to Code

5.1 Anatomy of a JIRA Issue

SPARK-XXXXX
├── Summary          ← one-line description (the symptom, not the cause)
├── Description      ← reproducer, stack trace, context
├── Comments         ← discussion, investigation, design debate
├── Attachments      ← patches (older workflow) or PR links (modern)
├── Fix Version/s    ← which release included the fix
└── Linked Issues    ← related JIRAs (parent, blocker, duplicate)

The comments section is the most valuable. It often contains:

• Stack traces with line numbers (now stale after the fix, but useful for locating the area)
• Bisect results ("the regression was introduced in commit X")
• Design debate ("Option A vs Option B — here's why we chose B")
• Reviewer concerns that shaped the final patch

5.2 Finding the Fix Commit

# Method 1: From the JIRA, click "Commits" tab → links to GitHub PR
# Method 2: Search git log for the JIRA number
git log --oneline --all | grep "SPARK-XXXXX"

# Method 3: Search merged PRs on GitHub
# github.com/apache/spark/pulls?q=SPARK-XXXXX

# Method 4: If you know the fix version (e.g., 3.2.0):
git log v3.1.3..v3.2.0 --oneline | grep "SPARK-XXXXX"

5.3 Reading the Diff

When you find the commit, read the diff with context:

git show <commit-hash>          # full diff
git show <commit-hash> --stat   # which files changed
git show <commit-hash> -U10     # 10 lines of context per change

Look for:

• What was added: new null checks, new synchronized blocks, new early-exit conditions
• What was removed: the broken code path
• What was not changed: the calling code — which tells you the fix is internal
• Test changes: what new case was added to the test suite

5.4 The Root Cause vs Symptom Distinction

This is what committers care about most in a patch review:

Example: A NullPointerException in CSVFileFormat.buildScan() when schema inference finds a column of all nulls.

• Symptom fix: Add if (field == null) continue; at the crash site
• Root cause fix: Understand why field can be null (schema inference returns null for all-null columns in some code paths), fix the schema inference to return NullType instead of null, add a test for all-null columns

Committers will always ask: "Why is this field null? Should it be? What other code paths hit this same condition?"

6. Interview Questions — Concepts You Must Explain Out Loud

7. References

• Apache Spark GitHub — apache/spark
• Spark Internals (Databricks slides) — overview of the execution model
• Deep Dive: Apache Spark Memory Management (Databricks)
• Mastering Apache Spark (Jacek Laskowski — online, free) — the most detailed public Spark internals reference
• Spark SQL Paper — Armbrust et al., SIGMOD 2015 — original Catalyst/DataFrame paper
• Effective Java, 3rd ed. — Joshua Bloch — Java idioms used throughout Spark's Java interop layers
• Maven: Guide to Multi-Module Projects
• IntelliJ IDEA: Navigating Source Code