Warmup Guide — JVM Codebase Navigation

Zero-to-expert primer for Phase 02: how to land in a million-line Apache JVM codebase (Spark, Hadoop, Kafka) and become productive without a guide — build systems, module anatomy, IDE archaeology, and execution-path tracing.

Table of Contents

Chapter 1: The Shape of a Large Apache JVM Project

From zero: Spark is ~1.5M lines across ~30 Maven/SBT modules; Hadoop ~3M. Nobody reads it linearly — you build a map of intent, then descend only where needed. The recurring module anatomy (learn it once, recognize it everywhere):

  • common/core: shared utilities, config machinery, RPC — everything depends on it.
  • API modules (sql/api, hadoop-client-api): the public surface, deliberately thin, annotated with audience/stability markers (Phase 04's compat machinery).
  • Engine modules (sql/core, sql/catalyst, core/): the actual machinery.
  • Connector/integration modules: optional, depend outward, good first-patch terrain (small blast radius).
  • *-tests / test-jars: shared test fixtures — also where you learn how to test in this codebase, which matters more than how to code in it.

The 80/20 of any module: README/package-info → the 2–3 interfaces everything implements → the config keys it registers → its largest test class. That ordering — docs, contracts, knobs, behaviors — beats reading implementation files in alphabetical order by an order of magnitude.

Chapter 2: Maven (and SBT/Gradle) — Enough to Navigate

You need build-system literacy to navigate, not to administer:

  • The reactor: mvn -pl sql/core -am package builds one module And its upstream deps (-am) — the difference between a 5-minute and a 2-hour loop. Learn each project's documented fast paths (Spark: ./build/sbt ~compile for incremental; -DskipTests everywhere).
  • dependency:tree: where "which jar does this class come from / why two versions" questions die. Shaded jars (relocated packages — org.sparkproject.guava) explain classes whose package names don't exist in any source tree.
  • Profiles (-Phadoop-3.4, -Pyarn): the same tree builds different artifact sets; when code seems unused, check whether a profile compiles it.
  • Where generated code lives (target/generated-sources — ANTLR parsers, protobuf): "I can't find this class's source" usually ends here.
  • Run a single test fast: mvn test -pl core -Dtest=DAGSchedulerSuite (or ./build/sbt "core/testOnly *DAGSchedulerSuite -- -z \"stage failure\"").

Chapter 3: Setting Up the Archaeology Lab — IDE and Tools

  • IntelliJ IDEA opens the Maven/SBT model directly; give it heap (-Xmx4g+ in the VM options) for Spark-sized indexes. The navigation muscle memory that matters: go-to-implementation (⌥⌘B — interfaces have many), find-usages (⌥F7), type hierarchy (⌃H), and structural search for patterns.
  • Decompiled views: IDEA decompiles dependency bytecode on click — how you read shaded/generated classes.
  • ripgrep beside the IDE: rg -t scala "spark.sql.shuffle.partitions" across the repo answers "where is this config consumed" faster than any index. Config-key strings are the single best entry points into unfamiliar engines — they appear at exactly the decision sites.
  • Debugger over print: a conditional breakpoint inside a real test run (-Dmaven.surefire.debug / IDEA's test runner) teaches call structure faster than a day of reading. Capture stacks: a breakpoint's stack trace is the call-path diagram for your notes.

Chapter 4: Top-Down Navigation — From Public API Inward

Use when the question is "how does feature X work":

  1. Find the public entry point (the method users call — df.collect(), producer.send()).
  2. Follow it through interfaces, not implementations — note each interface and ask "what are the other implementations and why do they exist?" (the answer encodes the design space).
  3. At each hop, record only: component name, responsibility in one phrase, the handoff. Resist reading bodies — you are drawing the subway map, not the streets.
  4. Stop at process/thread boundaries (RPC, event loops, queues) and mark them — they're where stack traces will later be discontinuous (Chapter 5's problem).

The deliverable form (Lab 01 uses it): a numbered hop list, API → ComponentA(role) → [event queue] → ComponentB(role) → …, each hop with file:line. Ten hops of this beats fifty pages of prose notes.

Chapter 5: Bottom-Up Navigation — From Symptom to Cause

Use when the question is "why did this break" (Lab 02's mode):

  1. Start from exact text: error messages and log lines are grep-able anchors — rg "Container killed on request" lands you at the throw/log site immediately.
  2. Read the stack trace as a path, but know its gaps: executor-side traces don't show the driver decision that scheduled them; async boundaries (Netty handlers, event loops, Future callbacks) restart stacks. At each gap, find the enqueue site for the dequeue you're looking at.
  3. Reproduce in a test: find the nearest existing test of the failing component (test class names mirror source names) and bend it toward the bug. A failing test is both your understanding made checkable and 80% of your eventual patch.
  4. Then bisect versions if needed: git bisect with that test as the oracle.

Chapter 6: git Archaeology — History as Documentation

In mature projects, why lives in history, not comments:

  • git log --follow -p -- path/File.scala — evolution of one file, renames included.
  • git blame -w -C (ignore whitespace, track copies) on the exact lines you don't understand → commit message → JIRA id (Apache commit discipline: [SPARK-12345] ...) → the JIRA thread holds the design discussion, the rejected alternatives, and the reviewers (= the experts to tag someday).
  • git log -S"methodName" (pickaxe) — when was this introduced/removed, even if blame is obscured by refactors.
  • The compound move you'll use weekly: blame line → JIRA → linked PR → review comments. That chain is the project's institutional memory, fully public.

Chapter 7: A Worked Example — Tracing a Spark Action

The Lab-01-shaped trace, as it should read when done (Spark ~3.5, names stable for years):

  1. RDD.collect() → SparkContext.runJob — the action entry; everything funnels here.
  2. → DAGScheduler.submitJob — posts JobSubmitted onto an event loop (DAGSchedulerEventProcessLoop) — first async boundary; driver-side.
  3. handleJobSubmitted → stage graph built by walking RDD lineage backward, cutting at shuffle dependencies (wide deps = stage boundaries — the Phase 07/interview staple).
  4. → submitMissingTasks → TaskScheduler.submitTasks → SchedulerBackend — tasks serialized, offered to executors via RPC (second boundary: process hop).
  5. Executor: Executor.TaskRunner.run → deserialize → Task.runTask → ShuffleMapTask/ResultTask — the closure finally executes; results travel back via block manager / direct result depending on size.

Every hop is findable by the Chapter 4 method in under an hour; the asymmetry between "impossible wall of code" and "ten findable hops" is the entire lesson of this phase.

Lab Walkthrough Guidance

Order: Lab 01 (spelunking) → Lab 02 (issue trace).

  • Lab 01: build the project first (the build always teaches: profiles, generated code, module graph). Produce the hop-list trace with file:line anchors; timebox each hop to 15 minutes — if stuck, you're reading bodies instead of following interfaces.
  • Lab 02: pick a resolved bug with a linked PR (you'll have ground truth). Do the full bottom-up: symptom → anchor grep → path → reproducing test idea → then compare against the actual fix. Score your diagnosis honestly in the template; the deltas are the curriculum.

Success Criteria

You are ready for Phase 03 when you can, from memory:

  1. Name the recurring module anatomy and the 80/20 reading order for a new module.
  2. Build one module with upstream deps and run one named test, in one command each.
  3. Execute the blame→JIRA→PR→review chain on an arbitrary confusing line.
  4. Reproduce the five-hop Spark action trace (or your project's equivalent) on a whiteboard.
  5. State the two navigation modes and which evidence each starts from.

Interview Q&A

Q: You've joined a team owning a 1M-line service you've never seen. First week? Build it and run its tests (the build reveals the module graph and the lies in the README); read the top 3 interfaces of the core module; trace ONE main request path top-down into a hop list; then take a real ticket bottom-up — navigation is learned by need, not by survey. Deliverable by Friday: the hop-list map and one small merged fix.

Q: How do you find why a config flag exists and whether it's safe to remove? rg the key for registration + consumption sites; blame the registration → JIRA → original motivating incident; check usage telemetry/defaults history (git log -S on the default value); then the Phase 04 compat lens — a public config is API surface, so removal is a deprecation cycle, not a delete.

Q: A stack trace ends inside an event loop. How do you keep tracing? Find the dequeue site's corresponding enqueue: search for the event class's constructor/post call. Async frameworks make causality discontinuous in stacks but explicit in event types — the event class name is the thread-crossing breadcrumb. (Same skill for Future callbacks: find where the future was created/completed.)

References