Lab 02 — Issue Trace

Phase: 02 — JVM Codebase Navigation
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 4–6 hours
Type: Code Investigation / Document
Repository: apache/spark

Goal

Perform an end-to-end trace of a real closed Apache Spark JIRA issue: read the full issue, locate the root cause in the source tree using git blame and git log, read and understand the fix commit, and produce a structured "bug journey" document.

This lab trains the most important skill in open-source contribution: the ability to take an unfamiliar bug report and navigate a large codebase to its root cause — independently, without guidance.

Pre-Selected Issues

Choose one of the following closed issues. All are real SPARK bugs with clean, single-commit fixes and complete JIRA descriptions. Start with Level 1 if you are new to Spark internals.

If a JIRA link is unavailable: The Apache JIRA is at https://issues.apache.org/jira/browse/SPARK-XXXXX. The corresponding GitHub PR can be found by searching github.com/apache/spark/pulls?q=SPARK-XXXXX.

Steps

Step 1 — Read the Issue Completely (45 min)

Open your chosen JIRA. Read:

1. The Summary line — this is the symptom
2. The Description — this is the reproducer and context
3. Every comment from top to bottom — this is where the diagnosis happens
4. The Fix Version/s field — which Spark release included the fix
5. Any linked issues — blockers, duplicates, parents

While reading, take notes on:

• What is the observable symptom?
• What conditions trigger it?
• Is there a stack trace? What line does it point to?
• What was the first hypothesis? Was it correct?
• Where did the investigation narrow down to?

Save your notes in bug-journey-[SPARK-XXXXX].md under ## Issue Analysis.

Step 2 — Find the Fix Commit (20 min)

cd spark

# Method 1: JIRA "Commits" tab (most reliable)
# JIRA → Commits tab → click GitHub PR link

# Method 2: Search git log
git log --oneline --all | grep "SPARK-XXXXX"
# Replace XXXXX with your issue number

# Method 3: GitHub PR search
# github.com/apache/spark/pulls?q=SPARK-XXXXX+is%3Amerged

# Once you have the commit hash or PR:
git show <commit-hash> --stat     # which files changed
git show <commit-hash> -U5        # diff with 5 lines of context

Record in your bug journey document:

• The commit hash
• The PR number (if applicable)
• Which files were changed
• How many lines were changed (added/removed)

Step 3 — Locate the Root Cause (1–2 hours)

This is the core skill.

Do NOT start by reading the fix. First, try to locate the root cause independently:

1. Use the stack trace or description to identify the first suspicious class/method
2. Open that class in IntelliJ. Read the method in question.
3. Use git blame on the specific lines — who wrote this, and when?
4. Read 50 lines above and below the crash site for context
5. Ask: "Under what condition does this code misbehave?"

# git blame with line range
git blame -L 120,160 sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Show what a specific commit changed in context
git show <commit-hash> -- path/to/File.scala

Then, after you've formed your own hypothesis, read the fix commit and compare.

Record in ## Root Cause Analysis:

• The specific class, method, and approximate line where the bug lives
• The exact condition that triggers incorrect behavior
• Your hypothesis (before reading the fix): what you thought the fix would do
• Whether your hypothesis matched the actual fix

Step 4 — Trace the Broken Code Path (30 min)

Reconstruct the execution path from user-facing API call to the point of failure. This is the same skill as Lab 01, but starting from a bug instead of a known entry point.

For example, if the bug is in CSVFileFormat.buildScan(), trace backward:

• Who calls buildScan()?
• Under what conditions is the null input passed?
• Could the caller have validated earlier?

Record a simplified call chain (5–8 entries is enough) in ## Broken Code Path.

Step 5 — Read and Understand the Fix (30 min)

Read the fix diff carefully:

git show <commit-hash> -U10   # 10 lines of context

For every change in the diff, answer:

1. What was the broken behavior?
2. What does the fix do instead?
3. Is this fixing the root cause or the symptom? How can you tell?
4. What test was added? What boundary condition does it cover?
5. Are there related conditions the fix does NOT cover? (Look for TODOs or known limitations in the commit message)

Record this in ## Fix Analysis.

Step 6 — Write the Correctness Argument (20 min)

A correctness argument answers: "Why does this fix work? Why doesn't it break anything else?"

This is exactly what a reviewer writing "+1" on the fix PR would say. Write yours in ## Correctness Argument:

• Why the fix resolves the root cause (not just the symptom)
• Why the fix doesn't introduce regressions in the non-bug case
• What the test added in the fix covers and why that's sufficient

Template: bug-journey-[SPARK-XXXXX].md

# Bug Journey — SPARK-XXXXX

## Issue Summary
- **JIRA**: [SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)
- **Symptom**: (one sentence)
- **Trigger conditions**: (what user action / data / configuration causes this)
- **Affected versions**: 
- **Fix version**: 
- **Severity** (your assessment): Blocker / Critical / Major / Minor

## Issue Analysis
[Notes from reading the full JIRA: investigation timeline, key comments]

## Root Cause Analysis
- **Location**: `ClassName.methodName()` in module `xxx`
- **Root cause**: (one paragraph — what is wrong with the code)
- **Your hypothesis before reading the fix**:
- **Did your hypothesis match?**: Yes / No / Partially — explain

## Broken Code Path
[Simplified call chain from user API to point of failure]
1. `SparkSession.X()` → ...
2. ...
N. `BuggyClass.method()` ← bug lives here

## Fix Analysis
- **Commit**: [link to GitHub commit]
- **Files changed**: 
- **Lines changed**: +X / -Y
- **What changed**: (per-file summary)
- **Root cause fix or symptom fix**: (your assessment + reasoning)
- **Test added**: (describe the new test case)
- **Known limitations**: (what cases does this NOT fix)

## Correctness Argument
[Why the fix works and why it doesn't break anything else — 2–4 paragraphs]

Expected Output

lab-02-issue-trace/
├── README.md                              ← this file
└── bug-journey-SPARK-XXXXX.md            ← your completed bug journey

Verification

Your bug journey is complete when:

1. Your root cause location matches the lines changed in the fix commit
2. Your correctness argument explains why the fix works — not just what it does
3. Your broken code path connects to the root cause location
4. You can explain the bug and fix out loud in 2 minutes without notes

Talking Points

• What is the difference between root cause and symptom? The symptom is the NPE. The root cause is that the schema inference returned null for an all-null column when it should have returned NullType.
• Why do reviewers push back on "add a null check here"? Because it papers over the symptom. The real question is: should null be reachable here at all? If not, why is it reachable?
• How do you know when a bug is "yours" to fix? Any committer can fix any bug. But fixing a bug in a subsystem you don't know well requires more review scrutiny. The more you understand the surrounding context, the more confidently you can fix root causes rather than symptoms.

Resume Bullet

"Performed end-to-end issue trace on a closed Apache Spark JIRA: located root cause in Spark internals via git blame and diff analysis, reconstructed the broken execution path across modules, and produced a structured correctness argument for the accepted fix."