Phase 07 — Hive, Tez, MapReduce & EMR
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 06 (Spark — runs on the same YARN/EMR substrate)
Why This Phase Exists
The JD lists Hive, Tez, MapReduce legacy systems, EMR, EMR Serverless, EMR on EKS, YARN and demands you "understand Hive and Tez execution … metastore behavior … legacy warehouse migration" and "EMR operations: cluster lifecycle, managed scaling, spot interruptions, bootstrap actions, instance fleets, YARN tuning, Spark-on-YARN, EMR Serverless, EMR on EKS, IAM, logging, and cost controls." It also lists migrating off Hive/Tez (Round 5). So you must understand the Hadoop lineage you're modernizing and operate the EMR clusters your Spark/Flink jobs run on — including the cost arithmetic that makes you the "cost efficiency" person.
Concepts
- The Hadoop lineage: HDFS (P08) + MapReduce (
map → shuffle/sort → reduce, every stage to disk) → Tez (DAG execution, pipelined, no disk between steps) → why Spark won (in-memory DAGs). Knowing this is knowing what you're migrating from. - Hive: the metastore (table → files + partitions + stats), the query compiler (HiveQL → MapReduce/Tez DAG), partitions & partition pruning, file formats (ORC native), statistics, bucketing, and the SerDe model. The ancestor of the lakehouse catalog (P10).
- Tez: DAG vertices/edges, container reuse, dynamic parallelism — the bridge between MapReduce and Spark-style execution.
- YARN: ResourceManager/NodeManager, containers (memory + vcores), the ApplicationMaster, schedulers (capacity/fair), and resource binding (the lab) — the cluster OS under Spark/ Hive/Tez on EMR.
- EMR operations: cluster lifecycle (provision → bootstrap → run → terminate), instance fleets/groups, managed scaling, core vs task nodes, spot interruptions & safe placement, bootstrap actions, logging to S3, IAM roles (EMRFS), and cost controls.
- EMR flavours: EMR-on-EC2 (you manage the cluster/YARN), EMR Serverless (no cluster, pay per vCPU/GB-hour), EMR on EKS (Spark on your Kubernetes) — and when each wins.
- Cost: spot economics, right-sizing, the small-file/partition-pruning levers (P06/P10), cost attribution by team (a JD deliverable).
Labs
Lab 01 — EMR/YARN Cost & Scheduling Model (flagship, implemented)
| Field | Value |
|---|---|
| Goal | YARN container packing + binding resource, EMR-on-EC2 cost with spot mix, EMR Serverless cost, spot-interruption probability, core/task split, and Hive partition pruning |
| Concepts | YARN scheduling, spot economics, EMR cost models, partition pruning |
| How to Test | pytest test_lab.py -v — 14 tests |
| Talking Points | What binds container count? When EMR Serverless vs EC2 vs EKS? Why core nodes on-demand? |
| Resume bullet | Built an EMR/YARN capacity + cost model (container packing, spot risk, serverless vs cluster cost, partition pruning) for cost-efficient batch operations |
→ Lab folder: lab-01-emr-cost-model/
Extension Project A — Hive → Spark/Iceberg migration plan (spec; → P15)
Map a legacy Hive/Tez warehouse to a modern lakehouse: preserve history, convert ORC layouts, keep the metastore working through the transition, eliminate small files, and decommission MapReduce/Tez — the JD's Problem 3 and Round 5.
Extension Project B — EMR cost review (spec)
Take a (synthetic) month of EMR usage; find waste (idle clusters, on-demand where spot is safe, over-provisioned nodes, full scans from missing partition filters) and produce a prioritized savings list with dollar estimates and cost attribution by team.
Integrated-Scenario Hooks
- This phase's partition pruning is the same lever as Athena cost (P10) and Iceberg partitioning (P09).
- Its spot/core-task model is how the capstone's batch tier (P16) runs cheaply.
- Its migration plan is the spine of P15's Round-5 design review.
Guides in This Phase
Key Takeaways
- MapReduce → Tez → Spark is a story of less disk between steps; know it to migrate well.
- Hive's metastore + partitions are the ancestor of the lakehouse catalog (P10).
- YARN packs containers by the binding resource (memory or vcores) — size tasks to fit.
- Spot is cheap but interruptible: core nodes on-demand, task nodes on spot.
- EMR Serverless wins for spiky/short jobs; EC2/EKS for steady or K8s-native ones.
Deliverables Checklist
- Lab 01 implemented; all 14 tests pass
- You can explain the MapReduce → Tez → Spark progression and why each step was faster
- You can choose EMR-on-EC2 vs Serverless vs EKS with the deciding dial
- Extension A migration plan or Extension B cost review