Glossary — Every Term in the JD, Defined Precisely

The JD names ~120 technologies and concepts. This glossary defines each one to the depth an interviewer probes, and points to the phase where you build it. If a term in the JD is not here yet, that is a bug — open it as a gap.

Legend: → PNN = the phase that implements/teaches it.

A. Compute & Processing Engines

  • Apache Spark — a distributed, in-memory batch+micro-batch engine. Work is a DAG of stages split at shuffle boundaries; each stage is a set of tasks (one per partition) run on executors. Key internals: shuffle, broadcast joins, Adaptive Query Execution (AQE), Tungsten memory management, spill. → P06
  • Apache Flink — a true streaming (record-at-a-time) engine with managed keyed/operator state, event-time processing, checkpoints/savepoints, and exactly-once sinks. The reference engine for low-latency stateful stream processing. → P04
  • Apache Hive — SQL-on-Hadoop: a metastore (table→files mapping) + a query compiler that historically generated MapReduce, then Tez, DAGs. The ancestor of the lakehouse catalog. → P07
  • Apache Tez — a DAG execution framework that replaced MapReduce under Hive/Pig: avoids writing intermediate results to HDFS between steps, enabling pipelined DAGs. → P07
  • MapReduce — the original Hadoop batch model: map → shuffle/sort → reduce, with every stage boundary materialised to disk. Slow but the conceptual root of all the above. → P07
  • Amazon EMR — AWS's managed Hadoop/Spark/Flink/Hive cluster service. Flavours: EMR-on-EC2 (YARN), EMR Serverless (no cluster to manage), EMR on EKS (run Spark on a Kubernetes cluster). → P07
  • AWS Glue — serverless Spark (Glue ETL) + the Glue Data Catalog (a managed Hive metastore) + crawlers. → P07, P10
  • Apache Beam — a unified batch+stream programming model (the Dataflow model) that runs on multiple engines (Flink, Spark, Dataflow) via runners. → P05
  • Trino / Presto — a distributed MPP query engine (no storage of its own) that federates queries over S3/Hive/Iceberg/Cassandra/etc. Trino is the actively-developed fork of PrestoDB. → P10
  • Athena — AWS's serverless, pay-per-TB-scanned managed Trino over S3 + Glue Catalog. → P10
  • Kubernetes-native data workloads — running Spark/Flink as K8s pods (Spark-on-K8s, Flink Kubernetes Operator) instead of YARN. → P07, P14

B. Streaming & Messaging

  • Apache Kafka — a distributed, partitioned, replicated commit log. Topics → partitions; ordering is guaranteed within a partition only; consumers track offsets in consumer groups; durability via replication (ISR) + acks=all. → P02
  • Amazon Kinesis — AWS's managed streaming service. Shards are the unit of throughput (1 MB/s in, 2 MB/s out, 1000 records/s in); the partition key maps records to shards. Data Streams vs Firehose vs Data Analytics. → P02
  • Amazon MSK — managed Kafka on AWS (MSK Provisioned / MSK Serverless). → P02
  • Apache Pulsar — a Kafka-class log with a segment/broker separation (BookKeeper storage) and built-in tiered storage + multi-tenancy. The "third option." → P02
  • Kafka Streams — a JVM library (not a cluster) for stateful stream processing on top of Kafka, using changelog topics + RocksDB local state. → P05
  • Flink DataStream API / Flink SQL — the low-level (operators, state, timers) and declarative (SQL over streams) Flink programming surfaces. → P04
  • Spark Structured Streaming — micro-batch (and continuous) streaming on Spark's SQL engine, with checkpointing to a durable store. → P05
  • Akka Streams — a Reactive-Streams implementation on the Akka actor runtime; back-pressured, typed stream graphs in Scala/Java. → P05, P12
  • Flume — a legacy log/event ingestion agent (source → channel → sink). The thing you migrate off of into Kafka/Kinesis. → P02
  • CDC (Change Data Capture) — turning a database's row changes (via binlog/WAL, e.g. Debezium) into an event stream. The bridge from OLTP to the lakehouse. → P05
  • Schema Registry — a service that stores versioned schemas and enforces compatibility rules so producers and consumers evolve safely. → P03
  • Dead-letter queue (DLQ) / quarantine — a side destination for messages that fail validation/processing, with the reason attached, so the pipeline keeps flowing and bad data is preserved for diagnosis. → P02, P03
  • Replay — re-reading the log from an older offset to reprocess history (the property that makes backfills and bug-recovery possible). → P02, P06
  • Consumer group / rebalance — a set of consumers sharing a subscription; partitions are divided among members and rebalanced on join/leave/failure. → P02
  • ISR (in-sync replicas) — the replica set caught up to the leader; acks=all + min.insync.replicas defines durability. → P02
  • Leader election — choosing the partition replica that serves reads/writes; happens on broker failure (KRaft/ZooKeeper-coordinated). → P02
  • Idempotent producer / transactions — Kafka features giving exactly-once append per partition and atomic offset+output commits within Kafka. → P02, P03
  • Poison message — a malformed/unprocessable record that, without a DLQ, blocks a partition forever. → P02

C. Streaming Semantics

  • Event time vs processing time — when the event happened vs when the system saw it. Aggregate in event time; use processing time only for pipeline metrics. → P04
  • Watermark — a monotonic estimate flowing with the stream asserting "no events with timestamp ≤ W are still coming"; drives window emission. → P04
  • Windows — tumbling (fixed, non-overlapping), sliding/hopping (overlapping), session (gap-based, dynamic). → P04
  • Allowed lateness — how long after a window fires it stays open to absorb late events (re-emitting corrections) before expiring. → P04
  • Checkpoint vs savepoint — automatic, engine-owned recovery snapshots vs manual, user-owned, portable snapshots for upgrades/migrations. → P04
  • Barrier / checkpoint alignment — Flink injects barriers into streams; aligning them across inputs is what makes checkpoints consistent (and can stall under skew). → P04
  • RocksDB state backend — an embedded LSM key-value store Flink uses to keep large keyed state on local disk (spilling beyond heap). → P04, P08
  • Backpressure — downstream slowness propagating upstream as flow control; the central streaming health signal. → P01, P04
  • Exactly-once / at-least-once / at-most-once — delivery+effect guarantees. "Exactly-once" = at-least-once delivery + idempotent or transactional processing = exactly-once effect. → P01
  • Sessionization — grouping a user's events into sessions by inactivity gaps. → P05

D. Languages, Runtime & Functional Programming

  • Scala — a JVM language blending OO and FP; the lingua franca of Spark/Flink/Kafka Streams internals and data-platform SDKs. → P12
  • Java interop / JVM — Scala compiles to JVM bytecode and interops with Java; you must understand JVM memory, GC, thread pools, serialization. → P12
  • Akka — an actor-model runtime for concurrent, distributed, supervised systems. → P12
  • Cats — a Scala library of functional abstractions: Functor, Applicative, Monad, Traverse, Validated, EitherT, OptionT, Resource. → P12
  • Cats Effect — an effect system (IO) giving referentially-transparent, cancelable, resource-safe concurrency. → P12
  • ZIO — an alternative effect system: ZIO[R, E, A] (environment, typed error, value) with built-in dependency injection and structured concurrency. → P12
  • FS2 — Functional Streams for Scala: pure, back-pressured streaming built on Cats Effect. → P12
  • Validated — an applicative that accumulates errors (unlike Either's fail-fast) — the right tool for data-contract validation. → P03, P12
  • Resource / bracket — guaranteed acquire/release of resources even under failure or cancellation. → P12
  • Type-level modeling / data contracts in types — making illegal states unrepresentable so whole bug classes can't compile. → P12

E. Storage & Data Formats

  • Amazon S3 — object storage; the lake's substrate. Strong read-after-write consistency (since 2020), prefix-based request scaling, request costs, lifecycle policies, encryption, cross-region replication. → P08
  • HDFS — the Hadoop Distributed File System; block-based, NameNode metadata + DataNode blocks. The on-prem ancestor of S3-as-lake. → P08
  • Parquet — a columnar file format: file → row groups → column chunks → pages, with per-chunk statistics, dictionary/RLE encoding, and predicate pushdown. → P08
  • ORC — Optimised Row Columnar: stripes → row index → streams, with built-in indexes and bloom filters; Hive-native. → P08
  • Avro — a row-oriented, schema-carrying binary format ideal for streaming/ingestion and schema evolution. → P03, P08
  • Protobuf — Protocol Buffers: compact tag-based binary encoding with strict field-number-based evolution rules. The JD's preferred event payload. → P03
  • JSON / CSV — text formats: ubiquitous, self-describing (JSON) or positional (CSV), but expensive and weakly typed for analytics. → P03, P08
  • Cassandra — a wide-column, masterless (Dynamo-style) distributed database. Model around queries; partition key chooses distribution; tombstones, TTL, compaction strategies, tunable consistency. → P11
  • DynamoDB — AWS's managed key-value/document store; partition-key + sort-key model, on-demand/provisioned capacity, GSIs/LSIs, streams. → P11
  • Iceberg — an open table format: metadata files → manifest lists → manifests → data files, giving snapshot isolation, hidden partitioning, partition evolution, time travel. → P09
  • Delta Lake — a table format using a transaction log (_delta_log JSON + checkpoints) for ACID, time travel, and MERGE. → P09
  • Apache Hudi — a table format optimised for upserts/incremental: Copy-on-Write vs Merge-on-Read, timeline, record-level indexes. → P09
  • Hive Metastore — the catalog mapping logical tables/partitions to physical files + statistics. → P07, P10
  • Glue Data Catalog — AWS's managed, Hive-compatible metastore. → P10
  • RocksDB — an embeddable LSM-tree KV store; Flink/Kafka-Streams state backend. → P04

F. Lakehouse / Table-Format Concepts

  • Snapshot / time travel — each commit creates an immutable snapshot; you can query "as of" a snapshot/timestamp. → P09
  • Snapshot isolation / optimistic concurrency — readers see a consistent snapshot; concurrent writers detect conflicts at commit and retry. → P09
  • Hidden partitioning / partition evolution — Iceberg derives partition values from columns (no extra columns) and lets you change the partition spec without rewriting old data. → P09
  • Compaction / clustering / Z-order / bucketing / sorting — rewriting many small files into fewer large ones and laying data out so queries skip more of it. → P06, P09
  • Small-file problem — too many tiny files → metadata + open-cost explosion + bad scan performance. The lakehouse's most common disease. → P06, P09

G. Query & Analytics Layer

  • Predicate pushdown — pushing filters down to the scan so the engine reads fewer files/row-groups/pages. → P08, P10
  • Partition pruning — skipping whole partitions whose values can't match the filter. → P10
  • Dynamic filtering — building a filter from one side of a join at runtime to prune the other side's scan. → P10
  • Broadcast join — shipping the small side to every node to avoid a shuffle. → P06, P10
  • Cost-based optimisation (CBO) — using table/column statistics to choose join order and strategy. → P10
  • CTE materialisation — computing a common sub-expression once and reusing it. → P10
  • Lake Formation — AWS's fine-grained (table/column/row) access control + governance layer over the Glue Catalog and S3. → P10, P14
  • Semantic layer / data mart — a curated, business-logic-bearing view layer over raw tables. → P10

H. Cloud-Native (AWS) Infrastructure

  • IAM — Identity and Access Management: principals, policies, roles, least privilege, cross-account access via AssumeRole. → P14
  • KMS — Key Management Service: envelope encryption, key policies, grants. → P14
  • CloudWatch / CloudTrail — metrics+logs+alarms / API audit logging. → P13, P14
  • EKS / Lambda / Step Functions — managed Kubernetes / serverless functions / serverless workflow orchestration. → P07, P13, P14
  • VPC networking / PrivateLink — network isolation / private service endpoints that keep traffic off the public internet. → P14
  • Terraform — declarative infrastructure-as-code; the way platform infra is versioned, reviewed, and promoted across environments. → P14

I. Reliability, Correctness & Governance

  • SLO / SLI / error budget — the target (objective), the measured indicator, and the allowed failure budget that gates change velocity. → P13
  • Lineage — the dataset/field-level graph of "what was computed from what." → P13
  • Data quality dimensions — freshness, volume, uniqueness, referential integrity, schema validity, semantic validity, cardinality drift. → P13
  • Idempotency / dedup / outbox — making re-processing safe (no double effects). → P01, P05
  • Backfill / reprocessing — recomputing historical outputs correctly and reproducibly. → P06, P13
  • Data contract — an enforced agreement (schema + semantics + SLA + ownership) between a producer and its consumers. → P03
  • PII classification — tagging fields by sensitivity to drive masking, access, and retention. → P14
  • Blast radius / failure domain — the set of things that fail together; the first question in any incident. → P01, P13
  • ADR (Architecture Decision Record) — a short, durable record of a decision and its context/tradeoffs/consequences. → P15

Cross-reference: the one-page numbers and decision rules live in CHEATSHEET.md.