Lab 01 — EMR/YARN Cost & Scheduling Model

Phase: 07 — Hive, Tez, MapReduce & EMR | Difficulty: ⭐⭐⭐☆☆ | Time: 4–5 hours

Owning EMR means owning the cluster's cost and placement arithmetic. This lab builds it: YARN container packing (and which resource binds), EMR-on-EC2 cost with a spot mix, EMR Serverless pay-per-use, spot-interruption probability, the core/task node split that makes spot safe, and Hive partition pruning.

What you build

  • yarn_plan — bin-pack tasks into containers across nodes; report whether memory or vcores is the binding resource (the "64 cores but memory caps me" lesson)
  • cluster_cost_usd — EMR-on-EC2 cost with a spot fraction
  • emr_serverless_cost_usd — pay per vCPU-hour + GB-hour (the spiky-workload win)
  • spot_interruption_probability — 1 − (1−p)^n (why 50 spot nodes ≈ guaranteed loss)
  • safe_spot_plan — on-demand core nodes (HDFS/AM) + spot task nodes
  • partitions_scanned — Hive partition pruning (the cost/perf lever; full scan when no filter)

Key concepts

ConceptWhat to understand
YARN binding resourcea node fits min(mem//task, vcores//task) containers
Spot economicsup to ~70–90% off, but interruptible (2-min warning)
Core vs task nodescore (HDFS/AM) on-demand; task (compute) on spot
EMR Serverlessno idle cluster; pay only while running — great for spiky jobs
Partition pruningno filter = full scan = the cost explosion

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 14 tests pass.
  • You can explain why spot_interruption_probability(50, 0.02) ≈ 0.64 and why that means "don't put HDFS or the driver on spot."
  • You can decide EMR-on-EC2 vs EMR Serverless vs EMR-on-EKS for a stated workload.

Extensions

  • Add an EMR-on-EKS vs EMR-on-EC2 vs Serverless total-cost comparator for a given job shape (steady vs spiky) and recommend with the deciding dial (P00).
  • Model managed scaling: given a job's task curve over time, compute the node count and cost vs a fixed cluster.
  • Add bootstrap + idle cost: include cluster spin-up time and idle minutes — the hidden cost that makes Serverless win for short, frequent jobs.