🛸 Hitchhiker's Guide — Phase 06: Apache Spark Internals

Read this if: you want to debug a slow/expensive Spark job fast and ace the "9-hour-job" interview round. Skim, read WARMUP, build the simulator.

0. The 30-second mental model

Spark splits your code into stages at every shuffle; each stage runs one task per partition; a stage finishes when its slowest task does. So performance = shuffle + skew + partitioning + memory + file layout. One sentence: fewer shuffles, evenly-sized partitions (128–256 MB), broadcast the small side, let AQE coalesce, and don't write a million tiny files.

1. The hierarchy

app → job (per action) → STAGE (split at shuffle) → TASK (1 per partition) → core/executor
#stages = #shuffles + 1        stage time ≈ slowest task (the straggler)

2. Narrow vs wide

narrow (map/filter)         → pipeline in-stage, no network
wide (groupBy/join/repartition/distinct/sort) → SHUFFLE → new stage, hits disk+network
first instinct on a slow job: "how many shuffles, can I remove one (broadcast)?"

3. The five performance levers

LeverSymptom if wrongFix
Shuffle countlong runtime, network heavybroadcast, bucketing, fewer wide ops
Skew1 task runs for hourssalt, AQE skew-join, broadcast, fix NULL keys
Partition sizespill / OOM / tiny taskssize by bytes: ceil(total/128MB)
Memoryspill, executor OOMmore partitions first, then memory; ~4–5 cores/exec
File layoutmillions of small files, Athena costrepartition/coalesce to 128MB–1GB; compact

4. The numbers

spark.sql.shuffle.partitions default = 200   ← almost always wrong; size by bytes
target partition / file size = 128 MB – 1 GB
autoBroadcastJoinThreshold default = 10 MB   ← raise deliberately for medium dims
spark.memory.fraction = 0.6 (exec+storage share of heap)
spill ≈ max(0, partition_bytes − heap×0.6)   ← more partitions shrinks this

5. Join picker

small side ≤ threshold → BROADCAST hash join (no shuffle)   ← biggest single speedup
two big tables         → SORT-MERGE (shuffle both, sort, merge)
stale stats hiding a small side → ANALYZE TABLE / broadcast hint
star schema            → dynamic partition pruning

6. AQE = three runtime tricks

1. coalesce small shuffle partitions (never splits a big one)
2. switch sort-merge → broadcast when a side is actually small
3. skew-join: split skewed partitions into parallel sub-partitions
turn it on (default modern Spark); still understand it to debug when it can't help

7. Spill: the counterintuitive one

spill happens when a TASK's working set > its execution-memory share
fix = MORE PARTITIONS (smaller working set) at least as often as more RAM

8. Small files = the silent bill

write emits 1 file / output-partition / task → millions of tiny files
→ huge metadata, slow listing, bad scans, S3 request cost (P08/P10)
fix: repartition/coalesce to target BEFORE write; compact table formats (P09)

9. Beginner mistakes that mark you

  1. Leaving shuffle.partitions=200 for any data size.
  2. "Add more memory" for spill instead of more partitions.
  3. Sort-merge joining when one side fits a broadcast.
  4. Writing without controlling file count → small-file explosion.
  5. Ignoring the NULL-join-key skew bomb.
  6. Non-idempotent writes → duplicate data on retry/backfill.
  7. Trusting stale table stats (no ANALYZE) so AQE/broadcast misfire.

10. War stories

  • "One task at 100%, 199 idle." → skew. Salt the hot key or AQE skew-join.
  • "OOM on a join." → broadcasting a not-actually-small side, or too few partitions.
  • "Athena bill 20×'d after the new pipeline." → the Spark job writes millions of small files; compact + partition (P09/P10).
  • "Backfill double-counted." → non-idempotent write; staging+atomic commit or table format.

11. How this phase pays off later

  • Small-file report → P08 (Parquet sizing), P09 (compaction), P10 (Athena cost).
  • Idempotent writes → P09 table-format commits.
  • Shuffle/executor sizing → P07 EMR cost & YARN.
  • Skew → P01, P02, P11 — same enemy, fifth costume.

Read WARMUP, build the simulator, then P07: where Spark runs — Hive/Tez/MapReduce lineage and EMR operations.