Lab 02 — Spark Execution Model in Scala

Phase: 06 — Apache Spark Internals | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Spark is a Scala/JVM system, so its tuning logic is naturally Scala. This lab is the Scala twin of Lab 01's Python simulator — stage planning at shuffle boundaries, skew detection, AQE coalescing, broadcast/shuffle sizing, spill, and the small-file report — as pure functions verified with ScalaTest (sbt test).

Why a model, not a real SparkSession? Spark 3.5 targets JDK 8/11/17/21; this environment runs JDK 26, where Spark's heavy sun.misc.Unsafe/JDK-internal reflection won't start. The execution model captures the reasoning the interview tests (Round 3) without the runtime — and runs anywhere. The "run it on real Spark" path is the extension.

What you build (SparkModel)

  • planStages / numStages — split a plan into stages at wide (shuffle) ops
  • detectSkew — partitions > factor × median (the straggler detector)
  • coalescePartitions — AQE greedy merge to target (big partition stands alone)
  • shouldBroadcast / shufflePartitions — join + parallelism sizing
  • estimateSpill — partition vs execution-memory share
  • smallFileReport — the compaction pitch (reduction ratio)

Run

sbt test

Expected: 6 tests, all green in SparkModelSpec.

Success criteria

  • sbt test → all 6 specs pass.
  • You can narrate the Round-3 "9-hour job" diagnosis using these functions (WARMUP).
  • You can explain why spill is a partition-count problem and why coalesce never splits a fat partition (that's the skew-join path).

Extensions

  • Run it on real Spark (JDK 17/21): add "org.apache.spark" %% "spark-sql" % "3.5.1", build a skewed DataFrame, groupBy/join, read explain(), and confirm AQE coalesces and switches the join — then compare to this model's predictions.
  • Add salting (split a hot key k#0..k#n, re-aggregate) and show the skew factor drop.
  • Model AQE dynamic join switching (sort-merge → broadcast once runtime stats arrive).