Lab 01 — Codebase Spelunking

Phase: 02 — JVM Codebase Navigation
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 4–6 hours
Type: Code Navigation / Document
Repository: apache/spark — tag v3.5.1

Goal

Produce an annotated execution trace of spark.read.csv("path") from the first line of user-facing API code down to FileScanRDD.compute(). Map the Maven module dependencies touched along the way. Explain the logical→physical plan boundary in your own words.

After this lab you will be able to open any Apache Spark bug report and navigate to the relevant code in under 10 minutes.

Setup (Do This Once)

# 1. Clone the repository
git clone https://github.com/apache/spark.git
cd spark
git checkout v3.5.1

# 2. Build the modules used in this lab (skip tests to save time)
./build/mvn -DskipTests -Phive -Pyarn package \
    -pl core,sql/catalyst,sql/core -am \
    2>&1 | tail -30

# 3. Open in IntelliJ IDEA
# File → Open → select the spark/ directory
# Wait for Maven import and indexing (5–20 minutes first time)
# Confirm: File → Project Structure → SDKs → Java 11 or 17

# 4. Verify navigation works:
# Press Cmd+O (Mac) or Ctrl+N (Linux/Windows)
# Type: SparkSession
# You should see org.apache.spark.sql.SparkSession

Steps

Step 1 — Map the Module Structure (45 min)

Open spark/pom.xml (the root). Find the <modules> block.

  1. List every <module> entry. There are approximately 30–40.

  2. For each of the following 8 modules, write one sentence describing its responsibility:

    • core
    • sql/catalyst
    • sql/core
    • sql/hive
    • resource-managers/yarn
    • network/common
    • network/shuffle
    • common/unsafe

  3. Draw a simple dependency diagram (ASCII is fine) showing which of these 8 modules depend on which others.

Hint: Open each module's pom.xml and look at its <dependencies>. The <artifactId> values show you the module names.

Save this in execution-trace.md under ## Module Map.

Step 2 — Trace the Call Path (2–3 hours)

Starting at SparkSession.read(), trace the code path to FileScanRDD.compute(). For each class in the chain, record:

  • The module it lives in (e.g., sql/core)
  • The fully qualified class name
  • One sentence describing its responsibility
  • The specific method where control transfers to the next class in the chain

Use IntelliJ's navigation tools:

  • Cmd+B / Ctrl+B — Go to Declaration
  • Opt+F7 / Alt+F7 — Find Usages
  • Ctrl+H — Type Hierarchy (for abstract classes and interfaces)
  • Cmd+Shift+F / Ctrl+Shift+F — Full-text search across the project

Starting point:

// In IntelliJ: Cmd+O → "SparkSession"
// Find the method: def read: DataFrameReader
val df = spark.read.csv("path/to/file.csv")

Waypoints to find and record (you fill in the details):

1. SparkSession.read()                          → sql/core
2. DataFrameReader.csv(path)                    → sql/core
3. DataFrameReader.load()                       → sql/core
4. DataSource.apply() / DataSource.resolveRelation()  → sql/core
5. CSVFileFormat  (or TextInputCSVDataSource)   → sql/core
6. LogicalRelation                              → sql/catalyst
7. FileSourceScanExec                           → sql/core
8. FileScanRDD                                  → sql/core
9. FileScanRDD.compute()                        → sql/core

When does the physical plan get built? The logical plan is built immediately when you call csv(). The physical plan is built lazily when you call an action (.show(), .count(), .collect()). Find where QueryExecution.executedPlan is called — this is the boundary.

Save this as a table in execution-trace.md under ## Execution Trace.

Step 3 — The Logical → Physical Plan Boundary (30 min)

Find and read these two classes:

  1. QueryExecution (in sql/core/)
  2. SparkPlanner (in sql/core/)

Answer these questions in execution-trace.md under ## Plan Boundary:

  1. What method in QueryExecution triggers physical plan creation? What does it call?
  2. What is SparkPlanner's role? What does it do with an optimized logical plan?
  3. In 3 sentences or fewer: explain the logical → physical plan transition in your own words, as if explaining it to an engineer who has never seen Spark internals.

Step 4 — Draw the Module Dependency Diagram (30 min)

For the classes you touched in Step 2, draw an ASCII diagram showing:

  • Which module each class belongs to
  • Which module depends on which (draw an arrow from dependent → dependency)

Example format:

sql/core ──depends on──▶ sql/catalyst
sql/core ──depends on──▶ core
sql/catalyst ──depends on──▶ core

Save this in execution-trace.md under ## Module Dependencies.

Expected Output

lab-01-codebase-spelunking/
├── README.md                   ← this file
└── execution-trace.md          ← your completed trace document

execution-trace.md structure:

## Module Map
[8 module descriptions + dependency diagram]

## Execution Trace
[Table: class name | module | responsibility | handoff method]

## Plan Boundary
[3-question answers + 3-sentence explanation]

## Module Dependencies
[ASCII dependency diagram for classes touched]

Verification

Your trace is correct if:

  1. Every class in your table is a real class you can open in IntelliJ
  2. The handoff methods connect correctly (output of one is input to next)
  3. LogicalRelation appears before FileSourceScanExec in your list
  4. Your plan boundary explanation correctly identifies that the physical plan is built lazily

Optional verification: set a breakpoint in FileScanRDD.compute() and run a small Spark app locally. Confirm the call stack matches your trace.

Talking Points

  • Why is DataSource a pivot point? It is the registry that maps a format name ("csv") to its FileFormat implementation. This is the extensibility point for custom data sources.
  • Why is the plan built lazily? Lazy evaluation lets the optimizer see the entire plan before committing. It also means you can build a plan in a driver application and serialize it to executors without running anything.
  • What is InternalRow vs Row? InternalRow is the unboxed binary format used inside the execution engine. Row is the user-facing boxed API. The conversion happens at the API boundary (e.g., Dataset.collect()).

Resume Bullet

"Navigated Apache Spark's multi-module Maven codebase from scratch; produced an annotated execution trace from SparkSession.read().csv() through Catalyst optimization to FileScanRDD.compute(), documenting module boundaries, class responsibilities, and the logical→physical plan transition."