Warmup Guide — Hive, Tez, MapReduce & EMR

Zero-to-principal primer for Phase 07: the Hadoop lineage you're modernizing (MapReduce → Tez → Spark), Hive's metastore and partition model, YARN as the cluster OS, and EMR operations + cost — the substrate your batch jobs run on and the bills you must control.

Table of Contents

Chapter 1: The Hadoop Lineage — MapReduce → Tez → Spark

You will spend part of this role migrating off the Hadoop stack, so you must understand it:

  • MapReduce (2004): the original model. A job is map → shuffle/sort → reduce, and crucially every stage boundary materializes to HDFS (disk). Multi-step jobs chain many MR jobs, each writing/reading disk — robust but slow. It's the conceptual root of everything (P06's stages are MR's map/reduce generalized).
  • Tez (2013): replaced MR under Hive/Pig with a DAG engine — vertices and edges, pipelined, no forced disk write between every step, container reuse, dynamic parallelism. A big speedup for Hive without rewriting queries.
  • Spark (2014): in-memory DAGs with lineage-based recovery (P06) — caches intermediate data in memory, generalizes beyond map/reduce, unifies batch + SQL + streaming + ML.

The through-line: each step removed disk I/O between stages. When you justify a Hive/Tez → Spark/Iceberg migration (Round 5), this is the performance story.

Chapter 2: Hive — Metastore, Partitions, Formats

Hive is "SQL over files," and its lasting contribution is the metastore:

  • Hive Metastore (HMS): a relational DB mapping logical tables → physical locations (HDFS/S3 paths), partitions, schemas, SerDes, and statistics. Every engine — Spark, Trino, Presto, Flink — can use the HMS (or the Glue Data Catalog, its AWS-managed equivalent, P10) as the source of truth for "what tables exist and where." The lakehouse catalog (P09/P10) is the metastore's descendant.
  • Partitions: a Hive table is partitioned by columns (e.g. dt=2026-06-14/region=us), which become directory paths. A query with a predicate on a partition column prunes to the matching directories (the lab's partitions_scanned) — the single biggest cost/perf lever. The classic failure: too many partitions (high-cardinality partition column → millions of tiny directories → metastore meltdown) or the small-file problem within partitions (P06/P08).
  • Statistics: ANALYZE TABLE populates row counts and column stats the optimizer needs (P10 CBO); stale stats = bad plans.
  • Formats: Hive's native columnar format is ORC (P08), with built-in indexes and bloom filters; Parquet is equally supported. The SerDe abstracts row encoding.
  • External vs managed tables: managed = Hive owns the data (drop = delete); external = Hive only catalogs it (drop = metadata only). Get this wrong and you delete the lake.

Chapter 3: Tez Execution

Tez is the DAG engine that made Hive fast before Spark took over. Worth knowing because legacy warehouses still run on it and the JD lists "Tez container failures" as a debugging target:

  • A Tez job is a DAG of vertices (each like a map or reduce stage) connected by edges (data movement: one-to-one, broadcast, scatter-gather/shuffle). It pipelines vertices and reuses containers across them (avoiding MR's per-stage JVM startup).
  • Dynamic parallelism: Tez can adjust the number of reducer tasks at runtime based on data size (a precursor to Spark's AQE, P06).
  • Common failures: container OOM (vertex memory misconfigured), shuffle/skew (same as P06), and metastore bottlenecks under many concurrent queries.

Chapter 4: YARN — The Cluster OS

YARN (Yet Another Resource Negotiator) is the resource manager under MapReduce/Tez/Hive/Spark on EMR-on-EC2:

  • ResourceManager (cluster-wide scheduler) + NodeManagers (per-node agents) + per-job ApplicationMaster (negotiates resources, tracks the job). Spark-on-YARN runs the Spark driver in (or alongside) the AM and executors in YARN containers.
  • A container = a slice of a node's memory + vcores. How many containers fit on a node = min(node_mem // container_mem, node_vcores // container_vcores) — and the binding resource (whichever runs out first) is what you tune (the lab's yarn_plan). The classic surprise: "this node has 64 cores but only fits 8 containers" because memory bound first.
  • Schedulers: Capacity (queues with guaranteed shares) and Fair (equalize over time) — how you isolate tenants on a shared cluster.
  • Tuning: container sizing, yarn.nodemanager.resource.memory-mb, overhead for off-heap, and matching Spark executor memory to container memory (with overhead headroom, P06 Ch. 7).

Chapter 5: EMR Operations

EMR is AWS's managed Hadoop/Spark/Flink/Hive. What you operate:

  • Cluster lifecycle: provision → bootstrap actions (install deps, configure) → run steps → terminate (transient clusters) or stay (long-running). Transient per-job clusters are a cost pattern; long-running shared clusters are a multi-tenancy pattern.
  • Instance fleets / groups: define node roles — master (1, runs the RM/AM), core (run NodeManagers and HDFS DataNodes — losing one risks data + the job), task (compute only, no HDFS — safe to lose). Fleets let you mix instance types and spot/on-demand with target capacities.
  • Managed scaling: EMR auto-adds/removes nodes based on YARN demand within min/max bounds — elastic cost without manual resizing.
  • Storage: EMRFS lets Hadoop read/write S3 as if it were HDFS (the lake lives in S3, P08); local HDFS is for shuffle/temp. IAM roles (instance profile + EMRFS role) govern S3 access (P14).
  • Logging: push YARN/Spark logs to S3 so they survive cluster termination — essential for post-mortems (P13).

Chapter 6: EMR Flavours — EC2 vs Serverless vs EKS

EMR on EC2EMR ServerlessEMR on EKS
You managethe cluster + YARNnothing (just submit)a K8s cluster
Scalingmanaged scalingautomatic per-jobK8s autoscaling
Cost shapenode-hours (idle costs!)per vCPU/GB-hour while runningnode-hours on EKS
Best forsteady, large, HDFS-heavyspiky/short/intermittent jobsK8s-native orgs, shared platform
Spin-upminutes (cluster boot)secondspod scheduling

The decision (lab's cost models): steady heavy batch → EC2; bursty/unpredictable → Serverless (no idle cost); already running everything on Kubernetes → EKS (one platform, one scheduler). The hidden factor is idle + bootstrap cost: a cluster that boots for 5 minutes and sits idle between hourly jobs is pure waste — Serverless eliminates it.

Chapter 7: Spot, Cost, and Cost Attribution

The "cost efficiency" mandate is arithmetic:

  • Spot instances: up to ~70–90% cheaper than on-demand, but reclaimable with a 2-minute warning. The probability that at least one of your spot nodes is interrupted is 1 − (1−p)^n (the lab) — which grows fast with node count, so at 50 nodes you should expect interruptions and design for them.
  • Safe spot placement: put core nodes (HDFS + AM — losing them kills the job/data) on on-demand, and task nodes (stateless compute) on spot (the lab's safe_spot_plan). Checkpoint long jobs so a reclaimed task node just reschedules its work.
  • Other levers (cross-phase): partition pruning (this lab) + columnar formats (P08) + compaction/no-small-files (P06/P09) cut scan and storage cost; right-sized containers cut compute; managed scaling + transient clusters cut idle.
  • Cost attribution (a JD deliverable): tag every cluster/job/bucket with an owner and a cost-allocation tag, so the bill is a map by team — the thing that actually changes behavior (teams optimize what they're charged for).

Chapter 8: Migrating Off the Legacy Stack

The JD's Problem 3 / Round 5: migrate Hive/Tez/Flume → Kafka/Flink/Spark/Iceberg/Athena. The principal pattern (full treatment in P15):

  1. Preserve history: keep the metastore working; the lakehouse can read the same S3 data via the catalog during transition.
  2. Convert layouts: ORC/Parquet conversion as needed; fix broken partitions; eliminate small files (compaction).
  3. Introduce the table format (Iceberg/Delta, P09): register existing data, then add ACID/time-travel/streaming writes.
  4. Dual-run & validate: run old (Hive/Tez) and new (Spark/Iceberg) in parallel; diff outputs (P13 data-diffing) until parity holds.
  5. Cut over queries (Athena/Trino, P10) and writers (Flink/Spark); decommission MapReduce/Tez/Flume only after a soak period with rollback available.

The non-negotiables: keep history reproducible, validate parity, and always have a rollback — the same migration discipline as schema evolution (P03) and backfills (P06).

Lab Walkthrough Guidance

Lab 01 — EMR/YARN Cost Model, suggested order:

  1. yarn_plan — min(mem//task, vcores//task); report the binding resource (the lesson).
  2. cluster_cost_usd — on-demand + spot mix.
  3. emr_serverless_cost_usd — per vCPU/GB-hour; scales linearly with time.
  4. spot_interruption_probability — 1−(1−p)^n; grows with node count.
  5. safe_spot_plan — core on-demand, task spot.
  6. partitions_scanned — equality-filter pruning; no filter = full scan.

Success Criteria

You are ready for Phase 08 when you can, from memory:

  1. Explain MapReduce → Tez → Spark as a disk-I/O-reduction story.
  2. Explain the Hive metastore, partitions, pruning, and managed-vs-external tables.
  3. Explain YARN containers and the binding-resource computation.
  4. Explain EMR node roles (master/core/task) and why spot goes on task nodes.
  5. Choose EMR-on-EC2 vs Serverless vs EKS for a workload, with the deciding factor.
  6. Compute spot interruption probability and a safe core/task split.
  7. Outline a Hive/Tez → lakehouse migration with validation and rollback.

Interview Q&A

Q: You're asked to cut the EMR bill 40%. Where do you look first? Attribution first — tag clusters/jobs by team so the bill is a map; you can't cut what you can't see. Then the mechanical levers in order of usual impact: (1) idle/bootstrap waste — move spiky jobs to EMR Serverless or transient clusters so you stop paying for warm idle clusters; (2) spot — put task nodes on spot (core nodes stay on-demand) for ~70% off the compute that's safe to interrupt; (3) scan cost — partition pruning + columnar + no small files (P06/P08/P10) often cuts the Athena/Spark scan bill 10×; (4) right-sizing — containers and node types matched to the binding resource. Each is a number I can estimate, so I'd present a prioritized list with dollar figures, not a vibe.

Q: A core node on spot got reclaimed and the job died. What went wrong? Core nodes run HDFS DataNodes and (often) the ApplicationMaster — losing one can lose shuffle/temp data and kill the application, so they must be on-demand. Spot belongs on task nodes, which are stateless compute: a reclaimed task node just reschedules its work elsewhere. The fix is the core/task split, plus checkpointing long jobs so a reclamation is a reschedule, not a restart. At any real node count, spot interruptions are a certainty (1−(1−p)^n), so the architecture has to assume them.

Q: When is EMR Serverless the right call over a cluster? When the workload is spiky, short, or intermittent, because a managed cluster charges you for boot time and idle minutes between jobs, while Serverless bills only the vCPU/GB-hours you actually consume and starts in seconds. For a steady, all-day, HDFS-heavy pipeline a right-sized EC2 cluster with managed scaling can be cheaper and gives more control; for "a few jobs an hour" Serverless usually wins on the idle-cost elimination alone. If the org already runs everything on Kubernetes, EMR-on-EKS unifies the platform and scheduler.

References