🛸 Hitchhiker's Guide — Phase 07: Hive, Tez, MapReduce & EMR

Read this if: you need the Hadoop lineage you're modernizing and the EMR ops + cost arithmetic, fast. Skim, read WARMUP, build the cost model.

0. The 30-second mental model

MapReduce → Tez → Spark is a story of removing disk between stages. Hive gave us the metastore (table→files+partitions, the lakehouse catalog's ancestor). YARN is the cluster OS that packs jobs into memory+vcore containers. EMR runs all of it on AWS — and your job there is mostly cost: spot, right-sizing, partition pruning, no idle clusters. One sentence: know the legacy to migrate it; operate EMR to make it cheap and reliable.

1. The lineage

MapReduce: map → shuffle/sort → reduce, DISK between every stage (slow, robust)
Tez:       DAG, pipelined, container reuse, dynamic parallelism (Hive got fast)
Spark:     in-memory DAG + lineage recovery (P06)  ← what you migrate TO
through-line: each step removed disk I/O between stages

2. Hive = the metastore

Hive Metastore (or Glue Catalog) = table → S3/HDFS path + partitions + schema + stats
partitions = directories (dt=2026-06-14/region=us); predicate → PRUNE to matching dirs
traps: too-many-partitions (high-cardinality col → metastore meltdown), small files
managed table (drop=delete data!) vs external (drop=metadata only)
ANALYZE TABLE for stats (CBO needs them, P10); native columnar = ORC (P08)

3. YARN containers

container = memory + vcores slice of a node
per node = min(node_mem // task_mem, node_vcores // task_vcores)
the BINDING resource (runs out first) is what you tune
"64 cores but only 8 containers" = memory bound

4. EMR node roles (memorize)

master : 1, runs ResourceManager / AM
core   : NodeManager + HDFS DataNode → losing one risks DATA + the job → ON-DEMAND
task   : compute only, no HDFS → safe to lose → SPOT

5. EMR flavours

EMR on EC2     → you run the cluster+YARN; steady heavy batch; idle costs money
EMR Serverless → no cluster, pay per vCPU/GB-hour WHILE running; spiky/short jobs win
EMR on EKS     → Spark on your Kubernetes; K8s-native orgs, one platform
hidden cost: cluster boot + idle minutes → Serverless eliminates it

6. The cost numbers

spot savings ≈ 70–90%, 2-min reclaim warning
P(≥1 interruption) = 1 − (1−p)^n   ← 50 nodes @2% ≈ 0.64 → expect it
Athena/Trino scan = $5/TB → partition prune + columnar + no small files = the bill
EMR Serverless ≈ $0.052/vCPU-hr + $0.0058/GB-hr

7. Cost-cut playbook (in order of usual impact)

1. attribute (tag by team) — can't cut what you can't see
2. kill idle/bootstrap waste (Serverless / transient clusters)
3. spot the task nodes (~70% off safe compute)
4. scan cost: partition prune + columnar + compaction (often 10×)
5. right-size containers/nodes to the binding resource

8. Beginner mistakes that mark you

  1. Core nodes on spot → reclaim kills HDFS + the job.
  2. High-cardinality partition column → millions of dirs, metastore meltdown.
  3. Dropping a managed table thinking it's external → data gone.
  4. Paying for warm idle clusters between hourly jobs (use Serverless).
  5. Sizing containers without checking which resource binds.
  6. No partition filter → full-table scan → the cost explosion.

9. War stories

  • "Spot reclaim killed the cluster." → core node was on spot. Core=on-demand always.
  • "Metastore is timing out." → someone partitioned by user_id (millions of dirs).
  • "The bill doubled and nobody knows whose job." → no cost-allocation tags.
  • "DROP TABLE deleted the lake." → it was a managed table, not external.

10. How this phase pays off later

  • Partition pruning → P09 (Iceberg partitioning), P10 (Athena cost).
  • Metastore → P10 (Glue Catalog, Lake Formation).
  • Migration plan → P15 Round-5 design review.
  • Cost arithmetic → P13 cost reviews, P00 tradeoff dials.

Read WARMUP, build the cost model, then P08: where the data actually lives — S3, HDFS, and the columnar formats (Parquet/ORC) that make scans cheap.