Phase 06 — Apache Spark Internals
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Prerequisites: Phase 01 (partitioning, skew), Phase 08 helps but isn't required yet
Why This Phase Exists
Spark is the workhorse of batch (and a serious streaming engine, P05), and it is the subject of interview Round 3: "a Spark job on EMR runs for 9 hours, spills heavily, produces millions of small Parquet files, and causes Athena queries to scan excessive data." The JD demands you "understand Spark execution internals: DAGs, stages, tasks, shuffles, broadcast joins, adaptive query execution, partitioning, memory management, spill behavior, executor sizing, speculative execution, and failure recovery" and "debug slow Spark jobs, shuffle explosions, data skew, executor OOMs, excessive spill, bad partitioning, small-file problems."
This phase makes those words mechanical. The performance of a Spark job is almost entirely determined by shuffle, skew, partitioning, memory, and file layout — and once you can reason about those five, a "9-hour job" becomes a checklist, not a mystery.
Concepts
- The execution hierarchy: application → jobs (per action) → stages (split at shuffle) → tasks (one per partition) → executors/cores; the DAG scheduler.
- Narrow vs wide dependencies: narrow (map/filter — pipelined) vs wide (groupBy/join/ repartition — shuffle); why the shuffle is the cost center.
- The shuffle: map-side write (sorted/spilled to disk) → reduce-side fetch; shuffle service; why shuffles dominate runtime, network, and disk; shuffle explosions.
- Data skew: a few hot keys → fat partitions → straggler tasks; detection (median multiple) and fixes (salting, AQE skew-join, broadcast).
- Joins: broadcast-hash vs sort-merge vs shuffle-hash; the broadcast threshold; AQE runtime join switching and skew-join splitting; dynamic partition pruning.
- Adaptive Query Execution (AQE): coalescing shuffle partitions, switching join strategies, optimizing skew — using runtime statistics.
- Memory & spill: unified memory (
spark.memory.fraction), execution vs storage, Tungsten off-heap, spill to disk, executor OOM, executor sizing (cores × memory). - File layout: the small-file problem, target file sizes,
repartition/coalescebefore write, output partitioning (and over-partitioning), Parquet (P08). - Reliability: task retries, speculative execution, stage recomputation, idempotent writes (staging + atomic commit / table formats), safe backfills and reprocessing.
- Spark SQL & Catalyst: logical → optimized → physical plan; predicate pushdown;
cost-based optimization (P10); reading
explain().
Labs
Lab 01 — Spark Execution Simulator (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Model stage planning at shuffle boundaries, skew detection, AQE coalescing, broadcast decisions, shuffle sizing, spill estimation, and small-file reporting |
| Concepts | Stages/shuffle, skew, AQE, broadcast joins, executor memory/spill, small files |
| How to Test | pytest test_lab.py -v — 16 tests |
| Talking Points | Why does a shuffle create a stage? When broadcast vs sort-merge? Why is spill a partition-count problem? |
| Resume bullet | Built a Spark execution model (stage planning, skew detection, AQE coalescing, broadcast/shuffle sizing, spill + small-file diagnostics) for tuning batch jobs |
→ Lab folder: lab-01-spark-simulator/
Lab 02 — Spark Execution Model in Scala (implemented, sbt test)
| Field | Value |
|---|---|
| Goal | The Scala twin of Lab 01 (Spark is a Scala/JVM system): stage planning, skew detection, AQE coalesce, broadcast/shuffle sizing, spill, small-file report — verified with ScalaTest |
| Concepts | Same Spark internals, in idiomatic Scala (a real SparkSession needs JDK 17/21; the model runs anywhere) |
| How to Test | sbt test — 6 ScalaTest specs (runs here); real-Spark path is the extension |
| Resume bullet | Modeled Spark's execution/tuning logic (stages, skew, AQE, spill, file sizing) in Scala with ScalaTest coverage |
→ Lab folder: lab-02-scala-spark-model/
Extension Project A — Tune a real Spark job (spec)
Generate a skewed dataset; run a groupBy/join locally (PySpark); read the Spark UI;
fix it with AQE, repartitioning, broadcast, and salting; measure the runtime and small-file
count before/after.
Extension Project B — Safe backfill framework (spec; → P13)
Design (and sketch in code) a backfill that is idempotent (staging + atomic commit or a table-format write), partitioned by date, resumable, and reproducible — the JD's "reproducible and auditable backfills."
Integrated-Scenario Hooks
- This phase's small-file report is the cause of the Athena scan-cost explosion (P10) and the motivation for compaction (P09).
- Its idempotent write pattern is realized by table-format commits (P09).
- Its skew is P01's skew, fourth costume; its shuffle sizing reappears for EMR cost (P07).
Guides in This Phase
Key Takeaways
- Spark performance = shuffle + skew + partitioning + memory + file layout. Master those five.
- A stage boundary is a shuffle; tasks = partitions; the shuffle is where time and money go.
- Spill is a partition-count problem as much as a memory one; AQE coalesces small partitions.
- Small files are the silent killer that turns into Athena's bill (P10).
- Idempotent writes (staging + atomic commit / table formats) make backfills safe.
Deliverables Checklist
- Lab 01 implemented; all 16 tests pass
- You can narrate the Round-3 9-hour-job diagnosis end to end
- You can explain AQE's three tricks (coalesce, join switch, skew-join)
- Extension A real-tune or Extension B backfill design