« All Roles

Principal / Distinguished Data Infrastructure Engineer — Curriculum

Target Role: Principal / Distinguished Data Infrastructure Engineer — Real-Time Distributed Data Platform, Streaming Systems, and Lakehouse Architecture (the reference JD — jd.md)

Also prepares you for:

• Netflix / Uber / Stripe / Databricks / Confluent — Staff/Principal Data Platform or Streaming Infrastructure Engineer
• Amazon / AWS — Principal Engineer, EMR / Kinesis / Glue / Lake Formation
• Any enterprise building a petabyte-scale, multi-region, real-time + batch data platform that hundreds or thousands of engineers depend on

Duration: 34 weeks core (≈8 months) — extendable to 12+ months with full capstones Seniority Target: Principal / Distinguished IC (the person who is the escalation point for the hardest failures and defines platform standards for the org) Goal: Make you the engineer who can architect the data infrastructure layer at the system level and debug it at the byte, partition, checkpoint, offset, file, and query-plan level — across streaming, batch, lakehouse, serving, governance, and the Scala/JVM platform code that ties it together.

Why This Curriculum Exists

This is not a "write ETL jobs" role and this is not a "learn one tool" course. The JD is explicit: you build the data infrastructure layer itself — the streaming substrate, the lakehouse architecture, the schema-governance system, the reliability controls, the developer SDKs, the observability, and the operational playbooks that hundreds or thousands of engineers, analysts, data scientists, and product teams stand on.

Most engineers specialise in one or two of: distributed systems, stream processing, batch processing, cloud infrastructure, functional programming, data modeling, query optimisation, data reliability, schema evolution, storage formats, operability, security, governance, cost, and developer-platform design. This role requires practical fluency across all of them — and, harder, the judgment to make the right tradeoff when they conflict (low latency vs low cost; exactly-once vs operational simplicity; flexible schemas vs safe contracts; self-service vs governance).

So this track is built around two non-negotiables:

1. Every technology in the JD gets implemented, not just described. Watermarks, exactly-once sinks, Parquet row-group skipping, Iceberg snapshot isolation, Spark's shuffle, Cassandra's tombstone math, a Cats-style Validated contract engine, IAM least-privilege evaluation — you build a working, tested miniature of each so the concept becomes muscle memory instead of folklore.
2. Everything is taught to the level interviewers actually probe — the numbers, the failure modes, the corner cases, the production tradeoffs. We assume you are aiming for the platform that serves millions of users with correctness guarantees, not a hobby pipeline.

How Each Phase Teaches (the "many ways of explaining")

Every phase deliberately explains the same material through several lenses, because principal-level understanding is the intersection of all of them:

When the JD demands the JVM (Flink, Scala/Cats/ZIO/FS2/Akka), the phase additionally ships a real, sbt test-verified Scala lab. Scala is first-class here, not an afterthought: it is the JVM language of Spark/Flink/Kafka-Streams and is in heavy demand for data-platform roles. Six runnable Scala labs + four runnable capstones (72 ScalaTest specs, all green via sbt test; JDK + sbt required):

What You Will Build

Every phase ships a tested, runnable artifact. By the end you will have hand-built a miniature of essentially every component of a modern data platform:

1. A failure-domain & exactly-once reasoning kit — partitioners, idempotency keys, a two-generals demonstration, and a dedup/outbox engine
2. An ingestion gateway — partition assignment, consumer-group rebalancing, offset management, retention/compaction, DLQ + poison-message quarantine, replay-from-offset
3. A schema-registry & data-contract engine — Protobuf/Avro/JSON compatibility checking (backward/forward/full), a Validated-style accumulating validator, CI gate
4. A Flink-grade stream processor — event-time windows, watermarks, allowed lateness, keyed state, checkpoint/savepoint + restore, exactly-once sink protocol
5. Streaming joins & CDC — stream-table enrichment, sessionization, a CDC apply loop with idempotent upserts
6. A Spark execution simulator — DAG → stages → tasks, shuffle + skew detection, AQE-style coalescing, broadcast-join planning, small-file analysis
7. An EMR/YARN cost & scheduling model — instance-fleet + spot-interruption planner, YARN container packing, EMR-Serverless vs EKS cost calculator
8. Parquet & ORC internals — a row-group/page reader with predicate pushdown and dictionary/RLE decoding from first principles
9. An Iceberg-style table format — snapshots, manifest lists, snapshot isolation, compaction, time travel, partition evolution, and conflict resolution on commit
10. A query optimiser — predicate pushdown, partition pruning, join reordering, a cost-based plan chooser, and an Athena scan-cost estimator
11. A Cassandra/DynamoDB data modeler — partition-key design, tombstone & TTL math, compaction-strategy chooser, consistency-level (R+W>N) calculator, hot-partition detector
12. A typed functional data SDK — Resource (bracket) safety, Validated/EitherT error modeling, an effect-system mini-runtime, FS2-style streaming, retry semantics
13. An orchestration & reliability core — a DAG scheduler with content-hash caching, SLO/error-budget tracker, freshness/volume/uniqueness data-quality checks, lineage graph
14. A governance & security kit — an IAM policy evaluator (least privilege), PII classifier + access policy, Lake-Formation-style column/row masking, audit log
15. Capstone platforms — the JD's five hard problems, composed end-to-end

Folder Structure

data-engineer/
├── README.md            ← you are here
├── jd.md                ← the target role profile (the source of truth for scope)
├── GLOSSARY.md          ← every term in the JD, defined precisely
├── CHEATSHEET.md        ← the numbers, formulas, and decision rules in one place
├── SUMMARY.md           ← mdBook table of contents
├── phase-00-principal-data-infra-engineer/
├── phase-01-distributed-systems-foundations/
├── phase-02-event-ingestion-messaging/
├── phase-03-serialization-schema-contracts/
├── phase-04-flink-stream-processing/
├── phase-05-streaming-alternatives-cdc/
├── phase-06-spark-internals/
├── phase-07-hive-tez-mapreduce-emr/
├── phase-08-storage-columnar-formats/
├── phase-09-lakehouse-table-formats/
├── phase-10-query-analytics-engines/
├── phase-11-serving-cassandra-dynamodb/
├── phase-12-scala-functional-data-engineering/
├── phase-13-orchestration-reliability-observability/
├── phase-14-security-privacy-governance/
├── phase-15-principal-architecture-strategy/
├── phase-16-capstone-systems/
├── interview-prep/      ← the JD's 5-round evaluation loop + per-domain Q&A banks
└── system-design/       ← full walkthroughs of the JD's example hard problems

Each phase contains README.md, WARMUP.md, HITCHHIKERS-GUIDE.md, BROTHER-TALK.md, and at least one lab-NN-*/ (README.md, lab.py, solution.py, test_lab.py, requirements.txt) where all tests pass against the solution.

34-Week Schedule

Prerequisites

• Strong programming — Python 3.10+ (the labs run in Python) and a willingness to read and write Scala (Phase 12 teaches it to platform-author level; you do not need prior Scala, but you need to not fear the JVM)
• SQL fluency — you will optimise queries at the plan level
• Comfort with a terminal, git, and Docker — the extension projects spin up real Kafka/Flink/Trino/Cassandra in containers
• No paid cloud account required — every flagship lab runs locally and simulates the distributed component; the concepts are cloud-portable by design. AWS-specific phases (07, 14) teach the model and provide Terraform you can read and adapt.

Interview Relevance

This track maps directly onto the JD's own five-round evaluation loop:

• Round 1 — Distributed data systems design ("ingest 5M events/sec → Flink → S3/Iceberg → Athena → Cassandra") → phases 01–04, 08–11 + system-design/
• Round 2 — Deep debugging (a Flink job with rising checkpoint duration, growing RocksDB state, delayed watermarks, growing lag) → phases 04, 06, 13
• Round 3 — Spark & lakehouse optimisation (a 9-hour job spilling and producing millions of small files) → phases 06–10
• Round 4 — Scala platform engineering (a type-safe contracts + stream-processing SDK) → phases 03, 12
• Round 5 — Architecture leadership (migrate Hive/Tez/Flume → Kafka/Flink/Iceberg) → phases 07, 09, 15

Start with Phase 00, then read each phase's HITCHHIKERS-GUIDE.md (fast), then its WARMUP.md (slow), then build the lab before moving on. The lab is where folklore becomes knowledge.

Principal / Distinguished Data Infrastructure Engineer

Real-Time Distributed Data Platform, Streaming Systems, and Lakehouse Architecture

Role Summary

We are looking for a Principal / Distinguished Data Infrastructure Engineer to design, build, operate, and evolve a mission-critical data platform that processes large-scale event streams, batch workloads, operational data, analytical data, and machine-learning-ready datasets across cloud and hybrid environments.

This role is not focused on writing ordinary ETL jobs. It is focused on building the data infrastructure layer itself: the streaming substrate, lakehouse architecture, schema governance systems, reliability controls, query optimization patterns, storage layout standards, ingestion frameworks, developer tooling, observability systems, and operational playbooks used by hundreds or thousands of engineers, analysts, data scientists, ML engineers, product teams, and business systems.

The ideal candidate has deep expertise in:

EMR, Hive, Tez, Spark, Flume, Flink, Scala, Akka, Cats,
Kafka, Kinesis, Cassandra, Parquet, Protobuf, Athena,
S3, Iceberg, Delta Lake, Hudi, Trino/Presto, Glue,
Airflow/Dagster, Kubernetes, Terraform, observability,
streaming semantics, schema evolution, distributed systems,
data correctness, platform reliability, and cost efficiency.

This is a senior technical leadership role for someone who can both architect at the system level and debug at the byte, partition, checkpoint, offset, file, and query-plan level.

Mission

Build and operate a high-scale, fault-tolerant, real-time and batch data platform that supports:

• Low-latency event ingestion.
• Real-time stream processing.
• Large-scale batch processing.
• Lakehouse analytics.
• Operational data serving.
• Feature generation for ML and AI systems.
• Data governance, lineage, privacy, and access control.
• High-confidence data correctness.
• Cost-efficient cloud execution.
• Developer-friendly platform abstractions.
• Safe self-service for internal engineering teams.

The platform must handle workloads ranging from simple analytics pipelines to advanced distributed systems problems involving stateful stream processing, exactly-once guarantees, late-arriving events, backpressure, skew, reprocessing, schema migration, partition evolution, checkpoint recovery, multi-region replication, high-cardinality observability, and petabyte-scale storage optimization.

Technical Environment

The engineer should be comfortable designing and operating systems involving most or all of the following technologies.

Compute and Processing

Apache Spark
Apache Flink
Apache Hive
Apache Tez
Amazon EMR
AWS Glue
Apache Beam
Trino / Presto
Athena
MapReduce legacy systems
Kubernetes-native data workloads

Expected depth:

• Understand Spark execution internals: DAGs, stages, tasks, shuffles, broadcast joins, adaptive query execution, partitioning, memory management, spill behavior, executor sizing, speculative execution, and failure recovery.
• Understand Flink internals: job graph, task slots, keyed state, operator state, checkpoints, savepoints, barriers, watermarks, event time, processing time, timers, state TTL, backpressure, RocksDB state backend, rescaling, and exactly-once sinks.
• Understand Hive and Tez execution: query planning, DAG execution, partition pruning, file formats, metastore behavior, statistics, ORC/Parquet optimization, and legacy warehouse migration.
• Understand EMR operations: cluster lifecycle, managed scaling, spot interruptions, bootstrap actions, instance fleets, YARN tuning, Spark-on-YARN, EMR Serverless, EMR on EKS, IAM, logging, and cost controls.

Streaming and Messaging

Apache Kafka
Amazon Kinesis
Kafka Streams
Flink DataStream API
Flink SQL
Spark Structured Streaming
Akka Streams
Flume
CDC platforms
Schema Registry
Dead-letter queues
Replay systems

Expected depth:

• Design durable event ingestion architectures.
• Choose between Kafka, Kinesis, Pulsar, and cloud-native queues.
• Understand partitioning, consumer groups, offset management, retention, compaction, replication, ISR, leader election, and broker failure modes.
• Design replayable pipelines.
• Handle ordering guarantees and out-of-order events.
• Implement event-time processing with watermarks.
• Control backpressure and overload.
• Design dead-letter queues and quarantine flows.
• Handle poison messages and malformed payloads.
• Build idempotent producers and consumers.
• Understand exactly-once versus effectively-once versus at-least-once semantics.
• Design transactional sinks and deduplication strategies.
• Build streaming joins, windowed aggregations, sessionization, enrichment, and stateful processing.
• Migrate legacy Flume ingestion into modern Kafka/Kinesis/Flink-based ingestion.

Programming Languages and Runtime Systems

Scala
Java
Python
SQL
Akka
Cats
Cats Effect
FS2
ZIO
Functional programming
Type-level modeling
JVM performance tuning

Expected depth:

• Build production-grade Scala services and libraries.
• Use functional programming to model data workflows safely.
• Use Cats abstractions such as Functor, Applicative, Monad, Traverse, Validated, EitherT, OptionT, and Resource.
• Understand effect systems such as Cats Effect or ZIO.
• Build concurrent, streaming, and actor-based systems with Akka or Akka Streams.
• Understand JVM memory, GC, thread pools, async boundaries, blocking calls, serialization, and profiling.
• Write APIs and frameworks that other engineers can use safely.
• Design strongly typed domain models for events, schemas, transformations, and data contracts.
• Build internal SDKs for ingestion, stream processing, data quality, schema validation, and pipeline deployment.

Storage and Data Formats

Amazon S3
HDFS
Cassandra
DynamoDB
Iceberg
Delta Lake
Apache Hudi
Hive Metastore
AWS Glue Data Catalog
Parquet
ORC
Avro
Protobuf
JSON
CSV
RocksDB state backend

Expected depth:

• Design table formats for lakehouse workloads.
• Understand Parquet internals: row groups, column chunks, pages, encodings, compression, statistics, predicate pushdown, dictionary encoding, and file sizing.
• Understand Protobuf schema evolution, field numbering, backward compatibility, unknown fields, optional fields, repeated fields, and binary payload design.
• Compare Avro, Protobuf, JSON, Parquet, and ORC for different ingestion and analytics use cases.
• Design partitioning strategies that avoid small files, skew, and high metadata overhead.
• Implement compaction, clustering, sorting, Z-ordering, bucketing, and file-size optimization.
• Design Cassandra data models around query patterns, partition keys, clustering keys, tombstones, TTL, compaction strategy, consistency levels, repair, hot partitions, and read/write amplification.
• Understand S3 consistency, object layout, request costs, prefix scaling, lifecycle policies, encryption, and cross-region replication.
• Build storage abstractions that support analytics, replay, data science, machine learning, and operational serving.

Query and Analytics Layer

Athena
Trino / Presto
Hive
Spark SQL
Flink SQL
Iceberg SQL
Glue Catalog
Lake Formation
BI tools
Ad hoc analytics

Expected depth:

• Optimize queries at the plan level.
• Understand join strategies, predicate pushdown, partition pruning, statistics, CTE materialization, broadcast joins, dynamic filtering, and cost-based optimization.
• Diagnose slow Athena/Trino/Hive/Spark SQL queries.
• Design lakehouse tables for high-concurrency analytics.
• Build semantic layers and data marts without corrupting lineage or duplicating business logic.
• Support both exploratory analytics and production-grade data products.
• Understand when Athena is sufficient and when a dedicated Trino/Spark/Flink architecture is needed.
• Establish standards for table ownership, schemas, SLAs, freshness, retention, privacy, and access control.

Core Responsibilities

1. Architect the End-to-End Data Platform

Design the complete data platform spanning:

event production
ingestion
schema validation
stream processing
batch processing
CDC
data lake storage
warehouse/lakehouse tables
serving stores
query engines
governance
observability
deployment
incident response
cost management

You will define how data flows from operational systems into Kafka/Kinesis, through Flink/Spark jobs, into S3/Iceberg/Parquet tables, into Athena/Trino/Spark SQL, and into downstream systems such as Cassandra, feature stores, dashboards, ML pipelines, and customer-facing applications.

You must be able to reason about:

• Data correctness.
• Ordering.
• Replayability.
• Idempotency.
• Deduplication.
• Schema compatibility.
• Data freshness.
• Latency.
• Throughput.
• Cost.
• Operational complexity.
• Blast radius.
• Privacy.
• Security.
• Failure recovery.

2. Build Real-Time Streaming Infrastructure

Design and implement real-time pipelines using Kafka, Kinesis, Flink, Spark Structured Streaming, and Akka Streams.

Examples of expected systems:

• Real-time user activity ingestion.
• Payment/event/fraud detection pipelines.
• Clickstream processing.
• IoT/event telemetry processing.
• Real-time personalization features.
• Online aggregation systems.
• Low-latency operational dashboards.
• CDC-based replication pipelines.
• Streaming joins between event streams and slowly changing dimensions.
• Exactly-once transactional sinks into lakehouse tables.
• Streaming upserts into Cassandra or DynamoDB.
• Real-time alerting and anomaly detection streams.

You should deeply understand:

• Event-time processing.
• Watermarks.
• Late events.
• Windowing.
• Sessionization.
• Stateful operators.
• Checkpointing.
• Savepoints.
• Backpressure.
• Offset commits.
• Rebalances.
• Consumer lag.
• Idempotent writes.
• Deduplication.
• Reprocessing.
• Failure recovery.
• State migration.
• Stream versioning.

3. Own Batch and Lakehouse Architecture

Design large-scale batch processing using Spark, Hive, Tez, EMR, Athena, Iceberg, Delta Lake, or Hudi.

Expected systems include:

• Petabyte-scale data lake tables.
• Incremental ETL and ELT pipelines.
• Backfill frameworks.
• Reprocessing frameworks.
• Batch feature generation.
• Historical event reconstruction.
• Data quality validation.
• Lakehouse compaction systems.
• Dataset publishing workflows.
• Multi-tenant data platform APIs.

You should be able to debug:

• Slow Spark jobs.
• Shuffle explosions.
• Data skew.
• Executor OOMs.
• Excessive spill.
• Bad partitioning.
• Small-file problems.
• Broken metastore partitions.
• Athena scan-cost explosions.
• Hive query regressions.
• Tez container failures.
• S3 throttling.
• EMR spot interruption failures.
• Corrupt Parquet files.
• Schema drift.
• Inconsistent late-arriving data.

4. Design Data Contracts and Schema Governance

Own schema governance across event streams, lakehouse tables, and analytical datasets.

Responsibilities:

• Define event schema standards.
• Manage Protobuf, Avro, and JSON schema evolution.
• Build compatibility checks.
• Integrate schema registry workflows into CI/CD.
• Prevent breaking changes in event contracts.
• Define data contracts between producers and consumers.
• Build schema migration playbooks.
• Track field-level ownership and lineage.
• Support versioned datasets and historical reprocessing.
• Enforce PII classification and access rules.
• Design policy-aware event ingestion.

Expected technical depth:

• Protobuf field compatibility.
• Backward and forward compatibility.
• Nullable versus optional fields.
• Enum evolution.
• Unknown field behavior.
• Binary encoding tradeoffs.
• Schema registry design.
• Contract testing.
• Producer/consumer compatibility matrices.
• Safe deprecation.
• Schema drift detection.

5. Build Data Quality, Correctness, and Reliability Systems

This role requires treating data pipelines like production software systems with explicit correctness guarantees.

You will define and implement:

• Data quality checks.
• Freshness checks.
• Volume anomaly checks.
• Referential integrity checks.
• Duplicate detection.
• Late-data reconciliation.
• Exactly-once or effectively-once strategies.
• Replay validation.
• Data diffing.
• Audit tables.
• Checkpoint verification.
• End-to-end lineage.
• Dataset certification.
• Data incident response.
• SLOs and SLIs for data products.

Example SLOs:

99.9% of critical streams process events within 60 seconds.
99.99% of accepted events are durably persisted.
Critical lakehouse tables are queryable within 15 minutes of event arrival.
No breaking schema change reaches production without compatibility checks.
Backfills are reproducible and auditable.
Every production dataset has an owner, SLA, lineage, and quality policy.

6. Operate Distributed Systems in Production

You are expected to be an escalation point for the hardest failures.

You should be able to diagnose:

• Kafka broker instability.
• Kinesis shard hot spots.
• Consumer lag spikes.
• Flink checkpoint failures.
• Flink RocksDB state growth.
• Spark executor failures.
• Spark shuffle service issues.
• EMR cluster instability.
• Hive metastore bottlenecks.
• Athena query failures.
• Cassandra tombstone storms.
• Cassandra hot partitions.
• S3 request throttling.
• Kubernetes pod eviction.
• JVM GC pauses.
• Serialization bottlenecks.
• Network saturation.
• Data skew.
• Backpressure cascades.
• Cascading pipeline failures.
• Incorrect event-time semantics.
• Silent data corruption.

You will design runbooks, dashboards, alerting, and automated mitigations for these systems.

7. Build Internal Developer Platforms

This role includes platform engineering, not just data engineering.

You will build tools that let other engineers safely create and operate pipelines.

Possible platform components:

• Pipeline templates.
• Scala SDKs for stream processing.
• Data contract generators.
• Protobuf validation tools.
• CI/CD checks for schemas and pipelines.
• Self-service pipeline deployment.
• Backfill orchestration framework.
• Replay framework.
• Data quality framework.
• Dataset registry.
• Lineage browser.
• Cost attribution dashboard.
• Query optimization advisor.
• Streaming job health dashboard.
• Data incident management workflow.
• Lakehouse table creation service.
• Automated compaction scheduler.
• EMR/Flink/Spark deployment abstractions.
• Local development and test harnesses.

The platform should reduce duplicated effort while preserving enough flexibility for advanced use cases.

8. Drive Cloud-Native Data Infrastructure

Own AWS-heavy architecture involving:

EMR
EMR Serverless
EMR on EKS
Kinesis
MSK
S3
Glue
Athena
Lake Formation
IAM
KMS
CloudWatch
CloudTrail
EKS
Lambda
Step Functions
DynamoDB
VPC networking
PrivateLink
Terraform

Expected depth:

• IAM least privilege.
• Cross-account access.
• Encryption at rest and in transit.
• KMS key policies.
• Network isolation.
• Private endpoints.
• S3 bucket policies.
• Lake Formation permissions.
• Glue catalog governance.
• Athena workgroup controls.
• Cost allocation.
• Autoscaling.
• Spot usage.
• Multi-region architecture.
• Disaster recovery.
• Infrastructure as code.
• CI/CD promotion across environments.

9. Lead Technical Strategy and Architecture Reviews

You will define the technical direction for the data platform.

Responsibilities:

• Create architecture decision records.
• Review major data platform designs.
• Define streaming and lakehouse standards.
• Establish data modeling patterns.
• Decide when to use Spark, Flink, Hive, Athena, Trino, Cassandra, Kinesis, Kafka, or EMR.
• Evaluate new technologies.
• Lead migrations away from obsolete systems.
• Mentor senior engineers.
• Raise the engineering bar.
• Create reference architectures.
• Build prototypes for risky designs.
• Present tradeoffs to senior engineering leadership.
• Translate business requirements into platform capabilities.

You should be able to defend architectural decisions using:

• Latency requirements.
• Throughput requirements.
• Failure modes.
• Team skill constraints.
• Cost model.
• Operational burden.
• Data correctness requirements.
• Security and compliance needs.
• Migration complexity.
• Long-term maintainability.

Example Hard Problems This Role Must Solve

Problem 1: Global Real-Time Event Ingestion

Design a multi-region event ingestion platform that accepts billions of events per day from product services, mobile clients, backend systems, and partner integrations.

Requirements:

• Kafka or Kinesis ingestion.
• Protobuf payloads.
• Schema validation at ingress.
• Dead-letter routing.
• Low-latency Flink processing.
• Durable raw event storage in S3.
• Partitioned Parquet/Iceberg tables.
• Athena and Trino query access.
• Reprocessing support.
• Regional failover.
• Exactly-once or effectively-once persistence.
• End-to-end observability.
• Cost attribution by producer team.

Problem 2: Stateful Fraud or Risk Stream Processing

Build a Flink-based stateful processing system that detects suspicious behavior in near real time.

Requirements:

• Event-time semantics.
• Out-of-order events.
• Sliding windows.
• Session windows.
• Pattern detection.
• Enrichment from Cassandra.
• Broadcast state for rules.
• RocksDB state backend.
• Checkpointing to S3.
• Savepoint-based deployment.
• Stateful job upgrades.
• Rule versioning.
• Low-latency alert output.
• Replay for model/rule validation.

Problem 3: Lakehouse Migration from Hive/Tez to Iceberg/Spark/Flink

Migrate a legacy Hive/Tez warehouse into a modern lakehouse.

Requirements:

• Preserve historical data.
• Convert ORC/Parquet layouts.
• Maintain backward-compatible schemas.
• Introduce Iceberg or Delta Lake.
• Support Athena/Trino/Spark queries.
• Eliminate small files.
• Implement compaction.
• Fix broken partitions.
• Replace batch-only pipelines with incremental processing.
• Support streaming writes.
• Maintain lineage.
• Reduce query cost.
• Decommission legacy Flume/Hive dependencies safely.

Problem 4: Cassandra Serving Layer for Real-Time Features

Design a low-latency serving store for derived real-time features.

Requirements:

• Cassandra schema design.
• Partition key design.
• Hot partition avoidance.
• TTL policy.
• Compaction strategy.
• Consistency-level selection.
• Write idempotency.
• Streaming upserts from Flink.
• Backfill from Spark.
• Read-path latency SLOs.
• Tombstone monitoring.
• Repair strategy.
• Multi-region replication.

Problem 5: Enterprise Data Governance Platform

Design governance for thousands of datasets and event streams.

Requirements:

• Dataset registry.
• Event schema registry.
• Field-level ownership.
• PII classification.
• Access-control integration.
• Lineage graph.
• Data quality checks.
• Contract testing.
• Audit logging.
• Retention policies.
• Dataset certification.
• Breaking-change prevention.
• Integration with CI/CD.
• Integration with Lake Formation, Glue, Athena, and IAM.

Required Experience

Distributed Data Systems

You should have extensive experience with:

• Apache Spark internals and production operations.
• Apache Flink internals and production operations.
• Kafka or Kinesis at high scale.
• EMR, Hive, Tez, Athena, Glue, and S3-based data lakes.
• Lakehouse table formats such as Iceberg, Delta Lake, or Hudi.
• Cassandra or another distributed serving database.
• Batch and streaming pipeline design.
• Large-scale data modeling.
• Production incident response.
• Query optimization.
• Data correctness and reliability engineering.

Scala and JVM Engineering

You should be strong in:

• Scala.
• Java interoperability.
• Functional programming.
• Akka or Akka Streams.
• Cats or Cats Effect.
• Type-safe API design.
• JVM performance tuning.
• Concurrency.
• Serialization.
• Library design.
• Testing distributed data applications.

This role expects more than basic Scala syntax. You should be able to build reusable frameworks and platform libraries that prevent entire classes of bugs.

Cloud and Platform Engineering

You should understand:

• AWS data services.
• Kubernetes.
• Terraform.
• CI/CD.
• Secrets management.
• IAM.
• Networking.
• Observability.
• Autoscaling.
• Cost optimization.
• Multi-environment deployment.
• Disaster recovery.
• Security controls.

Data Reliability and Operations

You should be comfortable owning:

• SLOs.
• SLIs.
• Error budgets.
• On-call rotations.
• Incident response.
• Postmortems.
• Runbooks.
• Capacity planning.
• Cost reviews.
• Reliability reviews.
• Operational maturity roadmaps.

Advanced Technical Expectations

A strong candidate should be able to answer and implement around questions like these.

Spark

• How does Spark split work into jobs, stages, and tasks?
• What causes a shuffle?
• How do you diagnose data skew?
• How does adaptive query execution change join behavior?
• When should you use broadcast joins?
• How do you tune executor memory?
• What causes executor OOM?
• How do you design a safe backfill framework?
• How do you avoid small files when writing Parquet?
• How do you make Spark writes idempotent?

Flink

• How do watermarks affect correctness?
• How do checkpoints work?
• What causes checkpoint alignment delays?
• How does RocksDB state grow?
• How do you upgrade a stateful Flink job?
• What is the difference between savepoints and checkpoints?
• How do you handle late events?
• How do you design exactly-once sinks?
• How do you debug backpressure?
• How do you rescale keyed state?

Kafka and Kinesis

• How do you choose partition keys?
• How do you handle hot partitions?
• How do you design retention?
• How do you perform replay safely?
• How do you manage consumer lag?
• What happens during consumer group rebalancing?
• How do Kafka transactions work?
• How do Kinesis shards affect throughput?
• How do you design cross-region replication?
• How do you prevent poison messages from blocking pipelines?

Lakehouse

• How do Iceberg snapshots work?
• How do partition evolution and hidden partitioning work?
• How do you compact small files?
• How do you design table layouts for Athena and Trino?
• How do you support streaming writes and batch reads safely?
• How do you handle schema evolution?
• How do you implement time travel?
• How do you recover from bad writes?
• How do you enforce retention and deletion policies?

Cassandra

• How do you design tables around query patterns?
• What creates tombstones?
• Why are large partitions dangerous?
• How do compaction strategies affect performance?
• What consistency levels are appropriate?
• How do you handle TTL-heavy workloads?
• How do you model time-series data?
• How do you support backfills without overwhelming the cluster?

Data Correctness

• How do you prove a pipeline is correct?
• How do you detect duplicates?
• How do you make writes idempotent?
• How do you reconcile batch and streaming outputs?
• How do you test late-arriving data?
• How do you validate replay results?
• How do you prevent schema-breaking changes?
• How do you detect silent data loss?

Deliverables Expected in This Role

Within the first 6-12 months, this engineer should be able to deliver systems such as:

1. Unified event ingestion platform using Kafka/Kinesis, Protobuf, schema validation, DLQs, and replay.
2. Flink-based stream processing framework with templates, observability, deployment automation, and correctness patterns.
3. Spark/EMR batch framework for reliable backfills, incremental processing, and cost-efficient execution.
4. Lakehouse architecture using Iceberg/Delta/Hudi, Parquet, Athena, Trino, and Glue.
5. Data contract system integrated into CI/CD.
6. Schema governance platform for Protobuf/Avro/JSON events.
7. Data quality framework for freshness, volume, uniqueness, referential integrity, and anomaly detection.
8. Streaming observability dashboard covering lag, throughput, watermark delay, checkpoint health, backpressure, DLQ rate, and failure rate.
9. Cost optimization program reducing waste from poor partitioning, inefficient queries, overprovisioned clusters, and excessive small files.
10. Migration strategy from legacy Hive/Tez/Flume systems to modern streaming and lakehouse systems.
11. Cassandra serving architecture for real-time features and low-latency lookups.
12. Incident response playbooks for Kafka, Flink, Spark, EMR, Athena, Cassandra, and S3 failures.

Success Metrics

This role is successful when:

• Critical pipelines have clear SLOs and meet them.
• Streaming latency is predictable and measurable.
• Batch backfills are reproducible and safe.
• Dataset ownership is explicit.
• Schema changes are controlled.
• Data quality issues are detected before customers or executives notice them.
• Engineers can create new pipelines without reinventing architecture.
• Athena/Trino/Spark query costs are reduced.
• Lakehouse tables are well-partitioned and compacted.
• Kafka/Kinesis/Flink/Spark incidents decrease in frequency and severity.
• The platform can replay historical data safely.
• Compliance and privacy controls are built into the platform.
• Senior engineers use the platform because it is better than custom one-off solutions.

Seniority Bar

This is not a senior data engineer role. This is a principal/expert-level infrastructure role.

The candidate should demonstrate:

• System-wide technical judgment.
• Ability to operate under ambiguity.
• Deep distributed systems debugging skill.
• Strong understanding of failure modes.
• Ability to influence multiple teams.
• Strong written architecture communication.
• Ability to simplify complex systems without making them fragile.
• Ability to build platforms, not just pipelines.
• Ability to mentor senior engineers.
• Ability to define engineering standards.
• Ability to balance correctness, latency, cost, and operational burden.

What Makes This Role Difficult

This role is difficult because it combines several hard domains:

distributed systems
stream processing
batch processing
cloud infrastructure
functional programming
data modeling
query optimization
data reliability
schema evolution
storage formats
operability
security
governance
cost optimization
developer platform design

Most engineers specialize in one or two of these. This role requires practical fluency across all of them.

The hardest part is not knowing the tools. The hardest part is making correct tradeoffs when the requirements conflict:

• Low latency versus low cost.
• Exactly-once semantics versus operational simplicity.
• Flexible schemas versus safe contracts.
• Self-service versus governance.
• Streaming freshness versus replay correctness.
• Cassandra serving speed versus lakehouse analytical flexibility.
• Batch accuracy versus real-time responsiveness.
• Fast delivery versus long-term platform maintainability.
• Cloud-native services versus open-source portability.
• Developer freedom versus platform standardization.

Example Interview / Evaluation Loop

Round 1: Distributed Data Systems Design

Design a real-time data platform that ingests 5 million events per second, validates Protobuf schemas, processes events with Flink, stores raw and curated data in S3/Iceberg, supports Athena queries, and serves derived features from Cassandra.

Expected discussion:

• Kafka versus Kinesis.
• Partitioning.
• Schema registry.
• DLQs.
• Flink topology.
• Checkpointing.
• Storage layout.
• Exactly-once semantics.
• Cassandra modeling.
• Replay.
• Observability.
• Failure recovery.
• Cost model.

Round 2: Deep Debugging

You are given a Flink job with rising checkpoint duration, increasing RocksDB state size, delayed watermarks, and growing Kafka lag.

Expected analysis:

• Backpressure.
• State growth.
• Late events.
• Key skew.
• Checkpoint alignment.
• RocksDB tuning.
• Network bottlenecks.
• Sink slowness.
• Watermark strategy.
• Partition distribution.
• Deployment changes.
• Rollback and savepoint strategy.

Round 3: Spark and Lakehouse Optimization

A Spark job on EMR runs for 9 hours, spills heavily, produces millions of small Parquet files, and causes Athena queries to scan excessive data.

Expected analysis:

• Spark DAG.
• Shuffle analysis.
• Partitioning.
• AQE.
• Join strategy.
• Executor sizing.
• File sizing.
• Compaction.
• Iceberg/Delta/Hudi table layout.
• Athena partition pruning.
• S3 cost model.
• Backfill strategy.

Round 4: Scala Platform Engineering

Design a Scala library that allows product teams to define event schemas, validate data contracts, build stream processors, write to lakehouse tables, and expose observability automatically.

Expected discussion:

• Type-safe APIs.
• Cats abstractions.
• Resource safety.
• Effect systems.
• Error modeling.
• Retry semantics.
• Backpressure.
• Testing.
• Serialization.
• Compatibility validation.
• Developer ergonomics.
• Binary compatibility.
• Versioning.

Round 5: Architecture Leadership

Lead a design review for migrating from legacy Hive/Tez/Flume pipelines to Kafka/Flink/Spark/Iceberg/Athena.

Expected discussion:

• Migration phases.
• Compatibility.
• Dual writes.
• Data validation.
• Replay.
• Cutover.
• Rollback.
• Cost.
• Governance.
• Team adoption.
• Risk management.
• Deprecation plan.

Book Baseline: Suggested Chapter Structure

You can turn this role into a serious expert-level book using the following structure.

Part 1: Foundations of Modern Data Infrastructure

1. What a principal data infrastructure engineer actually does.
2. Data platform versus data pipeline versus data product.
3. Batch, streaming, serving, and lakehouse architectures.
4. Distributed systems principles for data engineers.
5. Correctness, latency, throughput, cost, and operability.

Part 2: Event Ingestion and Messaging

6. Kafka internals.
7. Kinesis internals.
8. Partitioning, ordering, retention, and replay.
9. Protobuf, Avro, schema registry, and data contracts.
10. Dead-letter queues, poison messages, and ingestion safety.
11. Flume and legacy ingestion migration.

Part 3: Stream Processing

12. Flink architecture and execution model.
13. Event time, watermarks, windows, and late data.
14. State, checkpoints, savepoints, and RocksDB.
15. Exactly-once and effectively-once processing.
16. Spark Structured Streaming.
17. Akka Streams and functional streaming.
18. Real-time joins, enrichment, and aggregations.
19. Production debugging for streaming systems.

Part 4: Batch Processing and EMR

20. Spark internals.
21. Spark SQL and query optimization.
22. Hive and Tez internals.
23. EMR architecture and operations.
24. Backfills, reprocessing, and replay.
25. Data skew, shuffle, spill, memory, and file layout.
26. Cost optimization for batch workloads.

Part 5: Storage, Formats, and Lakehouse Design

27. S3 and object storage design.
28. Parquet internals.
29. ORC, Avro, JSON, and Protobuf tradeoffs.
30. Iceberg, Delta Lake, and Hudi.
31. Partitioning, clustering, compaction, and metadata.
32. Athena, Trino, Presto, and query-engine behavior.
33. Lakehouse governance and table ownership.

Part 6: Serving Systems

34. Cassandra data modeling.
35. DynamoDB and cloud-native serving stores.
36. Feature serving and real-time lookups.
37. TTL, tombstones, hot partitions, and consistency.
38. Connecting streams to serving stores safely.

Part 7: Scala and Functional Data Engineering

39. Scala for data infrastructure.
40. Akka and actor-based systems.
41. Cats and functional abstractions.
42. Cats Effect, ZIO, FS2, and resource safety.
43. Type-safe event modeling.
44. Building internal data platform SDKs.

Part 8: Reliability, Observability, and Operations

45. Data SLOs and SLIs.
46. Streaming observability.
47. Batch observability.
48. Lineage and auditability.
49. Incident response.
50. Data quality systems.
51. Detecting silent data corruption.
52. Capacity planning and cost management.

Part 9: Security, Privacy, and Governance

53. IAM and least privilege.
54. Encryption, KMS, and network isolation.
55. Lake Formation and Glue governance.
56. PII classification and retention.
57. Access control and audit trails.
58. Data contracts and organizational governance.

Part 10: Principal-Level Architecture

59. Architecture decision records.
60. Platform strategy.
61. Migration from legacy Hadoop systems.
62. Build versus buy decisions.
63. Multi-region data architecture.
64. Developer experience for data platforms.
65. Technical leadership and design reviews.
66. Operating as a principal engineer.

Part 11: Capstone Systems

67. Build a Kafka/Kinesis ingestion platform.
68. Build a Flink stream processing platform.
69. Build a Spark/EMR batch processing platform.
70. Build an Iceberg/Athena lakehouse.
71. Build a Cassandra feature-serving layer.
72. Build schema governance and data contracts.
73. Build observability and incident response.
74. Build a full production-grade data platform.

Final Positioning Statement

This job description represents a top-tier expert data infrastructure role.

A person who can perform this role well is not merely a data engineer. They are a hybrid of:

principal software engineer
distributed systems engineer
streaming platform engineer
data architect
cloud infrastructure engineer
Scala/JVM engineer
database/storage engineer
data reliability engineer
technical strategist

The role is intentionally hard because it requires the engineer to understand the entire lifecycle of data:

creation -> ingestion -> validation -> transport -> processing ->
storage -> query -> serving -> governance -> replay -> deletion

And to do that while satisfying production constraints:

correctness
latency
throughput
availability
security
privacy
cost
operability
developer experience
long-term maintainability

That is the bar for a truly challenging principal/expert-level data infrastructure role.

Glossary — Every Term in the JD, Defined Precisely

The JD names ~120 technologies and concepts. This glossary defines each one to the depth an interviewer probes, and points to the phase where you build it. If a term in the JD is not here yet, that is a bug — open it as a gap.

Legend: → PNN = the phase that implements/teaches it.

A. Compute & Processing Engines

• Apache Spark — a distributed, in-memory batch+micro-batch engine. Work is a DAG of stages split at shuffle boundaries; each stage is a set of tasks (one per partition) run on executors. Key internals: shuffle, broadcast joins, Adaptive Query Execution (AQE), Tungsten memory management, spill. → P06
• Apache Flink — a true streaming (record-at-a-time) engine with managed keyed/operator state, event-time processing, checkpoints/savepoints, and exactly-once sinks. The reference engine for low-latency stateful stream processing. → P04
• Apache Hive — SQL-on-Hadoop: a metastore (table→files mapping) + a query compiler that historically generated MapReduce, then Tez, DAGs. The ancestor of the lakehouse catalog. → P07
• Apache Tez — a DAG execution framework that replaced MapReduce under Hive/Pig: avoids writing intermediate results to HDFS between steps, enabling pipelined DAGs. → P07
• MapReduce — the original Hadoop batch model: map → shuffle/sort → reduce, with every stage boundary materialised to disk. Slow but the conceptual root of all the above. → P07
• Amazon EMR — AWS's managed Hadoop/Spark/Flink/Hive cluster service. Flavours: EMR-on-EC2 (YARN), EMR Serverless (no cluster to manage), EMR on EKS (run Spark on a Kubernetes cluster). → P07
• AWS Glue — serverless Spark (Glue ETL) + the Glue Data Catalog (a managed Hive metastore) + crawlers. → P07, P10
• Apache Beam — a unified batch+stream programming model (the Dataflow model) that runs on multiple engines (Flink, Spark, Dataflow) via runners. → P05
• Trino / Presto — a distributed MPP query engine (no storage of its own) that federates queries over S3/Hive/Iceberg/Cassandra/etc. Trino is the actively-developed fork of PrestoDB. → P10
• Athena — AWS's serverless, pay-per-TB-scanned managed Trino over S3 + Glue Catalog. → P10
• Kubernetes-native data workloads — running Spark/Flink as K8s pods (Spark-on-K8s, Flink Kubernetes Operator) instead of YARN. → P07, P14

B. Streaming & Messaging

• Apache Kafka — a distributed, partitioned, replicated commit log. Topics → partitions; ordering is guaranteed within a partition only; consumers track offsets in consumer groups; durability via replication (ISR) + acks=all. → P02
• Amazon Kinesis — AWS's managed streaming service. Shards are the unit of throughput (1 MB/s in, 2 MB/s out, 1000 records/s in); the partition key maps records to shards. Data Streams vs Firehose vs Data Analytics. → P02
• Amazon MSK — managed Kafka on AWS (MSK Provisioned / MSK Serverless). → P02
• Apache Pulsar — a Kafka-class log with a segment/broker separation (BookKeeper storage) and built-in tiered storage + multi-tenancy. The "third option." → P02
• Kafka Streams — a JVM library (not a cluster) for stateful stream processing on top of Kafka, using changelog topics + RocksDB local state. → P05
• Flink DataStream API / Flink SQL — the low-level (operators, state, timers) and declarative (SQL over streams) Flink programming surfaces. → P04
• Spark Structured Streaming — micro-batch (and continuous) streaming on Spark's SQL engine, with checkpointing to a durable store. → P05
• Akka Streams — a Reactive-Streams implementation on the Akka actor runtime; back-pressured, typed stream graphs in Scala/Java. → P05, P12
• Flume — a legacy log/event ingestion agent (source → channel → sink). The thing you migrate off of into Kafka/Kinesis. → P02
• CDC (Change Data Capture) — turning a database's row changes (via binlog/WAL, e.g. Debezium) into an event stream. The bridge from OLTP to the lakehouse. → P05
• Schema Registry — a service that stores versioned schemas and enforces compatibility rules so producers and consumers evolve safely. → P03
• Dead-letter queue (DLQ) / quarantine — a side destination for messages that fail validation/processing, with the reason attached, so the pipeline keeps flowing and bad data is preserved for diagnosis. → P02, P03
• Replay — re-reading the log from an older offset to reprocess history (the property that makes backfills and bug-recovery possible). → P02, P06
• Consumer group / rebalance — a set of consumers sharing a subscription; partitions are divided among members and rebalanced on join/leave/failure. → P02
• ISR (in-sync replicas) — the replica set caught up to the leader; acks=all + min.insync.replicas defines durability. → P02
• Leader election — choosing the partition replica that serves reads/writes; happens on broker failure (KRaft/ZooKeeper-coordinated). → P02
• Idempotent producer / transactions — Kafka features giving exactly-once append per partition and atomic offset+output commits within Kafka. → P02, P03
• Poison message — a malformed/unprocessable record that, without a DLQ, blocks a partition forever. → P02

C. Streaming Semantics

• Event time vs processing time — when the event happened vs when the system saw it. Aggregate in event time; use processing time only for pipeline metrics. → P04
• Watermark — a monotonic estimate flowing with the stream asserting "no events with timestamp ≤ W are still coming"; drives window emission. → P04
• Windows — tumbling (fixed, non-overlapping), sliding/hopping (overlapping), session (gap-based, dynamic). → P04
• Allowed lateness — how long after a window fires it stays open to absorb late events (re-emitting corrections) before expiring. → P04
• Checkpoint vs savepoint — automatic, engine-owned recovery snapshots vs manual, user-owned, portable snapshots for upgrades/migrations. → P04
• Barrier / checkpoint alignment — Flink injects barriers into streams; aligning them across inputs is what makes checkpoints consistent (and can stall under skew). → P04
• RocksDB state backend — an embedded LSM key-value store Flink uses to keep large keyed state on local disk (spilling beyond heap). → P04, P08
• Backpressure — downstream slowness propagating upstream as flow control; the central streaming health signal. → P01, P04
• Exactly-once / at-least-once / at-most-once — delivery+effect guarantees. "Exactly-once" = at-least-once delivery + idempotent or transactional processing = exactly-once effect. → P01
• Sessionization — grouping a user's events into sessions by inactivity gaps. → P05

D. Languages, Runtime & Functional Programming

• Scala — a JVM language blending OO and FP; the lingua franca of Spark/Flink/Kafka Streams internals and data-platform SDKs. → P12
• Java interop / JVM — Scala compiles to JVM bytecode and interops with Java; you must understand JVM memory, GC, thread pools, serialization. → P12
• Akka — an actor-model runtime for concurrent, distributed, supervised systems. → P12
• Cats — a Scala library of functional abstractions: Functor, Applicative, Monad, Traverse, Validated, EitherT, OptionT, Resource. → P12
• Cats Effect — an effect system (IO) giving referentially-transparent, cancelable, resource-safe concurrency. → P12
• ZIO — an alternative effect system: ZIO[R, E, A] (environment, typed error, value) with built-in dependency injection and structured concurrency. → P12
• FS2 — Functional Streams for Scala: pure, back-pressured streaming built on Cats Effect. → P12
• Validated — an applicative that accumulates errors (unlike Either's fail-fast) — the right tool for data-contract validation. → P03, P12
• Resource / bracket — guaranteed acquire/release of resources even under failure or cancellation. → P12
• Type-level modeling / data contracts in types — making illegal states unrepresentable so whole bug classes can't compile. → P12

E. Storage & Data Formats

• Amazon S3 — object storage; the lake's substrate. Strong read-after-write consistency (since 2020), prefix-based request scaling, request costs, lifecycle policies, encryption, cross-region replication. → P08
• HDFS — the Hadoop Distributed File System; block-based, NameNode metadata + DataNode blocks. The on-prem ancestor of S3-as-lake. → P08
• Parquet — a columnar file format: file → row groups → column chunks → pages, with per-chunk statistics, dictionary/RLE encoding, and predicate pushdown. → P08
• ORC — Optimised Row Columnar: stripes → row index → streams, with built-in indexes and bloom filters; Hive-native. → P08
• Avro — a row-oriented, schema-carrying binary format ideal for streaming/ingestion and schema evolution. → P03, P08
• Protobuf — Protocol Buffers: compact tag-based binary encoding with strict field-number-based evolution rules. The JD's preferred event payload. → P03
• JSON / CSV — text formats: ubiquitous, self-describing (JSON) or positional (CSV), but expensive and weakly typed for analytics. → P03, P08
• Cassandra — a wide-column, masterless (Dynamo-style) distributed database. Model around queries; partition key chooses distribution; tombstones, TTL, compaction strategies, tunable consistency. → P11
• DynamoDB — AWS's managed key-value/document store; partition-key + sort-key model, on-demand/provisioned capacity, GSIs/LSIs, streams. → P11
• Iceberg — an open table format: metadata files → manifest lists → manifests → data files, giving snapshot isolation, hidden partitioning, partition evolution, time travel. → P09
• Delta Lake — a table format using a transaction log (_delta_log JSON + checkpoints) for ACID, time travel, and MERGE. → P09
• Apache Hudi — a table format optimised for upserts/incremental: Copy-on-Write vs Merge-on-Read, timeline, record-level indexes. → P09
• Hive Metastore — the catalog mapping logical tables/partitions to physical files + statistics. → P07, P10
• Glue Data Catalog — AWS's managed, Hive-compatible metastore. → P10
• RocksDB — an embeddable LSM-tree KV store; Flink/Kafka-Streams state backend. → P04

F. Lakehouse / Table-Format Concepts

• Snapshot / time travel — each commit creates an immutable snapshot; you can query "as of" a snapshot/timestamp. → P09
• Snapshot isolation / optimistic concurrency — readers see a consistent snapshot; concurrent writers detect conflicts at commit and retry. → P09
• Hidden partitioning / partition evolution — Iceberg derives partition values from columns (no extra columns) and lets you change the partition spec without rewriting old data. → P09
• Compaction / clustering / Z-order / bucketing / sorting — rewriting many small files into fewer large ones and laying data out so queries skip more of it. → P06, P09
• Small-file problem — too many tiny files → metadata + open-cost explosion + bad scan performance. The lakehouse's most common disease. → P06, P09

G. Query & Analytics Layer

• Predicate pushdown — pushing filters down to the scan so the engine reads fewer files/row-groups/pages. → P08, P10
• Partition pruning — skipping whole partitions whose values can't match the filter. → P10
• Dynamic filtering — building a filter from one side of a join at runtime to prune the other side's scan. → P10
• Broadcast join — shipping the small side to every node to avoid a shuffle. → P06, P10
• Cost-based optimisation (CBO) — using table/column statistics to choose join order and strategy. → P10
• CTE materialisation — computing a common sub-expression once and reusing it. → P10
• Lake Formation — AWS's fine-grained (table/column/row) access control + governance layer over the Glue Catalog and S3. → P10, P14
• Semantic layer / data mart — a curated, business-logic-bearing view layer over raw tables. → P10

H. Cloud-Native (AWS) Infrastructure

• IAM — Identity and Access Management: principals, policies, roles, least privilege, cross-account access via AssumeRole. → P14
• KMS — Key Management Service: envelope encryption, key policies, grants. → P14
• CloudWatch / CloudTrail — metrics+logs+alarms / API audit logging. → P13, P14
• EKS / Lambda / Step Functions — managed Kubernetes / serverless functions / serverless workflow orchestration. → P07, P13, P14
• VPC networking / PrivateLink — network isolation / private service endpoints that keep traffic off the public internet. → P14
• Terraform — declarative infrastructure-as-code; the way platform infra is versioned, reviewed, and promoted across environments. → P14

I. Reliability, Correctness & Governance

• SLO / SLI / error budget — the target (objective), the measured indicator, and the allowed failure budget that gates change velocity. → P13
• Lineage — the dataset/field-level graph of "what was computed from what." → P13
• Data quality dimensions — freshness, volume, uniqueness, referential integrity, schema validity, semantic validity, cardinality drift. → P13
• Idempotency / dedup / outbox — making re-processing safe (no double effects). → P01, P05
• Backfill / reprocessing — recomputing historical outputs correctly and reproducibly. → P06, P13
• Data contract — an enforced agreement (schema + semantics + SLA + ownership) between a producer and its consumers. → P03
• PII classification — tagging fields by sensitivity to drive masking, access, and retention. → P14
• Blast radius / failure domain — the set of things that fail together; the first question in any incident. → P01, P13
• ADR (Architecture Decision Record) — a short, durable record of a decision and its context/tradeoffs/consequences. → P15

Cross-reference: the one-page numbers and decision rules live in CHEATSHEET.md.

Cheat Sheet — The Numbers, Formulas & Decision Rules

One page of the things you should be able to recite. Each links to the phase that derives it. Memorising the number is worthless without the why — but in an interview, having both is the difference between "sounds senior" and "is senior."

Delivery semantics (P01)

at-most-once   = fire & forget                 → may lose data
at-least-once  = retry until acked             → may duplicate   ← practical default
exactly-once   = at-least-once + idempotency/transactions = exactly-once EFFECT

Rule: "exactly-once" is a property of the effect on state, re-established at every boundary — not a network guarantee. The moment you write to an external store, you re-earn it (upsert by key / dedup id / outbox).

Watermarks & windows (P04)

watermark W      = max_event_time_seen − bounded_delay
window fires when  W ≥ window.end
window expires when W ≥ window.end + allowed_lateness   → late events → side output
tumbling start   = ts − (ts % size);  interval is [start, start+size)  (end exclusive)

The triangle you trade: first-result delay ↔ completeness ↔ state size.

Kafka / Kinesis sizing (P02)

Spark tuning (P06)

#tasks in a stage      = #input partitions (post-shuffle: spark.sql.shuffle.partitions, default 200)
target partition size  ≈ 128–256 MB
broadcast join when     small side ≲ spark.sql.autoBroadcastJoinThreshold (default 10 MB; raise w/ care)
executor memory split  = heap × spark.memory.fraction(0.6) → execution+storage (unified)
spill cause            = partition data > available execution memory → tune partitions, not just memory
skew fix               = AQE skew join (split big partitions) / salting / broadcast
small-file fix         = repartition/coalesce to target size before write; compaction job after

Idempotent write recipe: write to a staging path → atomic rename/commit → or use a table format (Iceberg/Delta) whose commit is atomic.

EMR / cloud cost (P07, P14)

Parquet / file layout (P08)

file → row groups (≈128 MB) → column chunks → pages (≈1 MB)
skipping: footer + row-group stats (min/max/null count) → predicate pushdown
encodings: dictionary (low-cardinality) → RLE/bit-packing; then compression (zstd/snappy)
target file size: 128 MB–1 GB;  avoid << 128 MB (small-file tax) and >> 1 GB (poor parallelism)

Cassandra (P11)

strong consistency: R + W > RF   (e.g. RF=3, W=QUORUM(2), R=QUORUM(2) → 2+2>3 ✓)
partition size: keep < 100 MB and < ~100k cells; model around the read query
tombstones: deletes/TTL create tombstones → read amplification; alert on tombstones/read
compaction: STCS (write-heavy) | LCS (read-heavy, more write amp) | TWCS (time-series/TTL)
hot partition: too-coarse partition key → add a bucketing component to the key

Iceberg / lakehouse (P09)

commit = new metadata.json → manifest list → manifests → data files (immutable snapshot)
isolation = optimistic concurrency; conflicting commits retry against the new snapshot
time travel = query AS OF snapshot-id / timestamp
maintenance = rewrite_data_files (compaction) + expire_snapshots + remove_orphan_files
partition evolution = change spec without rewriting history (hidden partitioning)

Consistency / distributed systems (P01)

CAP: under a network Partition, choose Availability or Consistency
PACELC: Else (no partition), choose Latency or Consistency  ← the everyday tradeoff
quorum: R + W > N gives read-your-writes on that register
clocks: never trust wall clocks for ordering → use logical/Lamport/vector clocks or sequence ids

Reliability targets (P13) — example SLOs from the JD

99.9%  of critical streams process events within 60s
99.99% of accepted events are durably persisted
critical lakehouse tables queryable within 15 min of event arrival
no breaking schema change reaches prod without a compatibility check
every production dataset has an owner, SLA, lineage, and quality policy

The tradeoff dials you will always be asked to turn (P00, P15)

low latency ⟷ low cost
exactly-once ⟷ operational simplicity
flexible schema ⟷ safe contract
self-service ⟷ governance
streaming freshness ⟷ replay/batch correctness
cloud-native lock-in ⟷ open-source portability
developer freedom ⟷ platform standardization

Principal move: name the dial, quantify both ends with the workload's numbers, then choose — and write it in an ADR.

Definitions for every term above are in GLOSSARY.md.

Phase 00 — The Principal Data Infrastructure Engineer

Difficulty: ⭐⭐☆☆☆ (conceptual) → the judgment is ⭐⭐⭐⭐⭐ Estimated Time: 1 week (12–18 hours) Prerequisites: none — this is the orientation that makes the other 16 phases cohere

Why This Phase Exists

Before you touch Kafka or Flink or Iceberg, you have to understand what job you are actually being hired to do — because at the principal level the job is not "operate tools," it is "make correct tradeoffs across the entire data lifecycle when the requirements conflict, and encode those tradeoffs into platforms other people build on."

The JD says it directly: "This role is not focused on writing ordinary ETL jobs. It is focused on building the data infrastructure layer itself." And: "The hardest part is not knowing the tools. The hardest part is making correct tradeoffs when the requirements conflict."

So Phase 00 builds the mental model and the calculator for those tradeoffs. Every later phase plugs a real technology into a slot in the model you build here. If you skip this phase you will learn 16 tools and still not be able to architect — which is exactly the failure mode that keeps strong engineers stuck at senior.

Concepts

• Data pipeline vs data platform vs data product — three different things people call "data engineering," with three different success metrics and blast radii.
• The data lifecycle: creation → ingestion → validation → transport → processing → storage → query → serving → governance → replay → deletion. Every technology in this track lives at one or two of these stages; principals reason about the seams.
• The five forces: correctness, latency, throughput, cost, operability — and how every architecture decision trades them against each other.
• The tradeoff dials (low-latency↔low-cost, exactly-once↔simplicity, flexible schema↔safe contract, self-service↔governance, freshness↔replay-correctness, …) — and how to quantify both ends with the workload's numbers instead of arguing taste.
• Latency budgets — decomposing an end-to-end SLA across lifecycle stages so you know where the milliseconds (and dollars) actually go.
• Blast radius / failure domains — the unit of thinking that turns a 30-alarm storm into one root cause (developed fully in Phase 01 & 13).
• The ADR (Architecture Decision Record) — how principals make a decision durable and reviewable instead of tribal knowledge.

Labs

Lab 01 — Tradeoff & Lifecycle Modeler (flagship, implemented)

→ Lab folder: lab-01-tradeoff-modeler/

Integrated-Scenario Suggestions (carried through the whole track)

These five scenarios (the JD's "Example Hard Problems") are the spine of the capstone (Phase 16). Keep them in mind from now — each phase contributes a real component:

1. Global real-time event ingestion — billions of events/day, multi-region, Protobuf, schema validation at ingress, DLQ, Flink, S3/Iceberg, Athena, cost attribution by team.
2. Stateful fraud/risk stream processing — event-time, out-of-order, sliding+session windows, Cassandra enrichment, broadcast rules, RocksDB, savepoint upgrades, replay.
3. Lakehouse migration — Hive/Tez/Flume → Kafka/Flink/Spark/Iceberg/Athena, small-file elimination, compaction, dual-write, validation, cutover, rollback.
4. Cassandra serving layer — low-latency derived features, partition design, TTL, compaction, consistency, streaming upserts from Flink, Spark backfill, multi-region.
5. Enterprise governance platform — dataset/schema registry, field ownership, PII, lineage, contract testing, Lake Formation/Glue/IAM integration, breaking-change prevention.

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• The job is tradeoffs across a lifecycle, encoded into platforms, defended in ADRs — not pipelines.
• Every later technology fits a lifecycle slot; learn the slot first, then the tool.
• "Make it fast / cheap / correct" is not a requirement until you attach numbers; Phase 00's lab is where you learn to attach them.

Deliverables Checklist

• Lab 01 implemented; all tests pass against solution.py
• You can draw the 11-stage lifecycle from memory and name the tech at each stage
• You can take any "design X" prompt and immediately list the dials it forces
• You can write a one-page ADR for a real decision (template in WARMUP)

Warmup Guide — The Principal Data Infrastructure Engineer

Zero-to-principal primer for Phase 00: what the role actually is, the data lifecycle that organises the entire track, the five forces and the tradeoff dials you will spend your career turning, and the artifacts (latency budgets, cost models, ADRs) that turn opinion into engineering.

Table of Contents

• Chapter 1: Pipeline vs Platform vs Product
• Chapter 2: The Data Lifecycle — The Spine of the Track
• Chapter 3: The Five Forces
• Chapter 4: The Tradeoff Dials
• Chapter 5: Latency Budgets — Where the Milliseconds Go
• Chapter 6: Cost Models — Where the Dollars Go
• Chapter 7: Correctness as a First-Class Requirement
• Chapter 8: Blast Radius and Failure Domains
• Chapter 9: The ADR — Making Decisions Durable
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Pipeline vs Platform vs Product

From zero. Three things get called "data engineering," and conflating them is the single most common reason designs are over- or under-built:

• A data pipeline moves/transforms data from A to B. Success = "the job ran and the numbers look right." Blast radius = one dataset. A senior engineer owns pipelines.
• A data platform is the substrate other people build pipelines on: the ingestion SDK, the schema registry, the table-creation service, the orchestration framework, the observability that every pipeline inherits. Success = "hundreds of engineers ship correct pipelines without reinventing architecture." Blast radius = the whole org. A principal engineer owns platforms.
• A data product is a dataset/stream treated as a product: it has an owner, an SLA, a schema contract, documented semantics, a freshness guarantee, and consumers who depend on it. Success = "downstream teams trust it enough to build on it without asking."

The JD is unambiguous that this role is the middle one with strong opinions about the third: "building the data infrastructure layer itself … used by hundreds or thousands of engineers." Why does the distinction matter so much? Because the right answer changes with the noun. Exactly-once semantics in one pipeline is a code review; exactly-once as a platform guarantee every team inherits is an architecture program. Schema validation in one job is an if statement; schema governance as a platform is a registry, a compatibility checker, a CI gate, and an org-wide contract. You will be asked to build the second kind. Everything in this track is "build the platform version of X."

Chapter 2: The Data Lifecycle — The Spine of the Track

Every piece of data, in every system you will ever build, walks this path:

creation → ingestion → validation → transport → processing →
storage → query → serving → governance → replay → deletion

This is not a metaphor; it is the literal table of contents of the JD and of this track. Memorise it, because the principal skill is reasoning about the seams between stages — that's where production failures live:

The reason a principal can interview across all of Kafka and Spark and Iceberg and Cassandra is not that they memorised four manuals — it's that they hold one lifecycle and ask the same five questions (the next chapter) at every stage.

Chapter 3: The Five Forces

Every data-infrastructure decision is scored on five axes. There is no architecture that maxes all five; the art is knowing which the workload actually needs.

1. Correctness — does the output match reality? (no loss, no duplication, no skew, no silent corruption, schema-valid). The one force you can't trade away on critical data, only define precisely (exactly-once-effect, freshness SLA, quality policy).
2. Latency — how fast from event to usable result? (p50/p99; "real-time" is meaningless without a number and a percentile).
3. Throughput — how much per second, sustained and at peak? (events/s, MB/s, and the ratio of read to write).
4. Cost — $/event, $/TB stored, $/TB scanned, $/engineer-hour to operate.
5. Operability — how hard is it to run, debug, upgrade, and stay on-call for? (the force juniors ignore and principals obsess over, because it dominates total cost over a system's life).

The senior framing: "requirements" are not requirements until they are numbers on these five axes. "Make it fast and cheap and correct" is a wish. "p99 < 60s, 5M events/s peak, < $X/month, exactly-once into the lakehouse, runnable by a 4-person on-call" is a spec you can architect against — and the lab makes you compute it.

Chapter 4: The Tradeoff Dials

The five forces conflict in recurring, nameable ways. A principal names the dial, quantifies both ends with the workload's numbers, and chooses — out loud, in an ADR. The canonical dials (straight from the JD):

• Low latency ↔ low cost. Real-time Flink + Cassandra serving is fast and expensive; hourly Spark + Athena is cheap and slow. The number that decides it: what does a minute of staleness cost the business?
• Exactly-once ↔ operational simplicity. Transactional sinks and checkpoint barriers buy correctness and cost you debuggability and throughput. The number: what does one duplicate or one lost event cost? (a duplicated fraud alert vs a duplicated ad impression are not the same question).
• Flexible schema ↔ safe contract. Schemaless JSON ships fast and breaks consumers silently; strict Protobuf contracts slow producers and prevent 3 a.m. outages. The number: how many consumers, and how expensive is their breakage?
• Self-service ↔ governance. Letting teams create tables freely maximises velocity and guarantees a small-file/PII/lineage mess; gating everything is safe and becomes the bottleneck you were hired to remove. The principal answer is almost always "self-service with guardrails the platform enforces."
• Streaming freshness ↔ replay/batch correctness. The Lambda-vs-Kappa argument: maintain two code paths (fast+approximate, slow+correct) or one replayable streaming path. The number: can your streaming path be replayed deterministically? (if yes, Kappa; if no, you've bought Lambda's double maintenance).
• Cassandra serving speed ↔ lakehouse analytical flexibility. A serving store answers one query shape in single-digit ms; the lakehouse answers any query in seconds. You need both, wired safely (P11).
• Cloud-native lock-in ↔ open-source portability. Athena/Kinesis/Glue are fast to adopt and hard to leave; Trino/Kafka/Iceberg are portable and yours to operate.

Hold this: an interviewer is not testing whether you know Flink. They are testing whether, when they add "…and it must also be cheap," you can name which dial that turns and what it costs. That is the whole Round-1 signal.

Chapter 5: Latency Budgets — Where the Milliseconds Go

An end-to-end SLA (say "p99 event-to-queryable < 15 minutes") is meaningless until you decompose it across lifecycle stages, because that's how you find the stage that's eating your budget and the stage where spending more buys nothing:

ingest commit         ▏  50 ms   (acks=all to Kafka)
transport / consumer  ▏ 200 ms   (consumer poll + network)
stream processing     ▏   2 s    (Flink window + watermark delay)
write to S3/Iceberg   ▏  30 s    (batch the writes; small-file avoidance)
catalog commit/refresh▏  10 s    (snapshot commit + metadata)
query availability    ▏   --     (immediate once committed)
                      ───────────
budget consumed       ≈ 42 s     → 15-min SLA has huge headroom; spend it on cost, not speed

Two principal moves fall out of this:

1. Budget conservation — the sum of stage budgets must be ≤ the SLA; the lab enforces it (you can't allocate 110% of the budget).
2. Spend the slack. If the SLA is 15 min and you consume 42 s, the remaining 14 min is a cost lever: batch writes harder, use spot instances, compact lazily. Juniors optimise latency they don't need; principals convert unused latency into saved dollars.

Chapter 6: Cost Models — Where the Dollars Go

You cannot be the "cost efficiency" person the JD asks for without arithmetic. The ones that come up constantly (full derivations in later phases; here are the shapes):

• Athena/Trino scan cost = bytes scanned × $5/TB. This is why partitioning, columnar formats, and projection pruning are cost features, not just speed features. A query that scans 1 TB instead of 10 GB costs 100× — and the fix is layout, not hardware (P08–P10).
• S3 storage = GB-months × tier price; lifecycle policies move cold data to cheaper tiers. Plus request costs (PUT/GET per 1000) — which is why millions of small files cost money even when idle (P08).
• Compute = node-hours × instance price, where spot is up to ~90% cheaper but can be reclaimed in 2 minutes (P07). Cost = nodes × hours × $/hour × (1 − spot_discount).
• The dominant hidden cost is operability — engineer-hours on-call, debugging, and upgrading. A "cheaper" architecture that needs a dedicated team is not cheaper.

Cost attribution by producer team (a JD deliverable) is just tagging every byte and every node-hour with an owner and summing — boring, and the thing that actually changes behaviour, because teams optimise what they're billed for.

Chapter 7: Correctness as a First-Class Requirement

The JD repeats one idea more than any other: treat data pipelines like production software with explicit correctness guarantees. That means correctness is specified, tested, and monitored, not hoped for:

• Specified: "exactly-once-effect into the lakehouse," "99.99% of accepted events durably persisted," "no breaking schema change reaches prod." These are the JD's example SLOs — testable statements, not vibes.
• Tested: every lab in this track ships a property test. The flagship example (Phase 04): shuffled, duplicated delivery produces identical results to clean ordered delivery. If you can't write that test, you don't have correctness, you have luck.
• Monitored: freshness, volume, uniqueness, referential integrity, schema/semantic validity, cardinality drift (P13). The point is to find corruption before customers or executives do.

Correctness is the force you most often see traded away by accident — a processing-time aggregation here, a missing dedup there — and the failures are silent, which is why a principal builds the detection in (P13's silent-corruption chapter).

Chapter 8: Blast Radius and Failure Domains

A failure domain is the set of things that fail together (a partition, a broker, an AZ, a region, a shared metastore, a single hot key). Blast radius is how much breaks when one thing does. The first question in every incident is not "what's wrong with component X?" but "what shared resource explains all these alarms at once?"

This reframing is what lets a principal stay calm in a 30-alarm storm: 12 jobs failing that all read one Hive metastore is one incident (the metastore), not twelve. You design for blast radius (isolate tenants, bulkhead resources, cap fan-out) and you reason with it during incidents (P13). Phase 01 makes this rigorous; Phase 00 just plants the question.

Chapter 9: The ADR — Making Decisions Durable

A principal's decisions outlive their memory of them and must survive their absence. The Architecture Decision Record is the lightweight artifact that makes a decision reviewable, teachable, and reversible-on-purpose. The template you'll reuse all track:

# ADR-NNN: <short title>
Status: proposed | accepted | superseded by ADR-MMM
Date: YYYY-MM-DD

## Context
What forces are in play? What are the numbers (the five forces, the dials)?

## Decision
What we chose, stated as one sentence.

## Alternatives Considered
Each option, and the dial it turned the wrong way for *our* numbers.

## Consequences
What gets better, what gets worse, what we must now monitor, and what would
make us revisit this.

The discipline the ADR enforces is exactly Chapter 4's: name the dial, quantify both ends, choose, and write down what would change your mind. An architecture review (JD Round 5) is just a room full of people pressure-testing your ADRs.

Lab Walkthrough Guidance

Lab 01 — Tradeoff & Lifecycle Modeler, suggested order:

1. LifecycleStage enum and allocate_latency_budget(total, weights) — split a budget by weights and assert it sums to ≤ total (budget conservation; Ch. 5).
2. Cost functions: athena_scan_cost_usd(bytes) ($5/TB), s3_storage_cost_usd(...), spot_compute_cost_usd(...) (Ch. 6) — pure arithmetic, exact unit tests.
3. Sizing: kafka_partitions_needed(...), kinesis_shards_needed(...), quorum_is_strong(n, r, w) (r + w > n) — boundary tests (exactly equal is not strong).
4. classify_delivery_semantics(retries, idempotent, transactional) → at-most / at-least / exactly-once-effect (Ch. 7).
5. resolve_tradeoff(option_a, option_b, weights) — score two architectures on the five forces with workload weights; return the winner and the deciding force (explainable output, Ch. 3–4). Test the tie-break and the "weights change the winner" property.

Success Criteria

You are ready for Phase 01 when you can, from memory:

1. Define pipeline vs platform vs product and say which this role builds.
2. Recite the 11-stage lifecycle and name a technology and a seam failure for each.
3. List the five forces and explain why a "requirement" isn't one without numbers.
4. Name four tradeoff dials and, for each, the single number that decides it.
5. Decompose a 15-minute end-to-end SLA into a stage budget and identify the slack.
6. Compute an Athena scan cost and explain why layout (not hardware) is the fix.
7. Write a one-page ADR for a real decision.

Interview Q&A

Q: "Design a system to ingest 5M events/sec." Where do you start? Not with Kafka. I start by turning the prompt into numbers on the five forces: what's the read throughput vs the 5M write (the ratio drives the storage/serving split); what's the latency SLA and at what percentile; what correctness guarantee (an ad impression and a payment are different); what's the budget; and who operates it. Then I decompose the lifecycle and a latency budget, and only then do tools get chosen — each as an ADR that names the dial it turns. The 5M number alone tells me almost nothing; the ratios and the SLAs tell me everything.

Q: A team wants exactly-once everywhere. Your response? First, "exactly-once" is exactly-once-effect, re-earned at every external boundary — so "everywhere" is a much bigger bill than they think. Second, I'd ask for the cost of a duplicate and the cost of a loss per stream, because most streams don't need it: an analytics impression tolerates at-least-once with dedup at read; a balance transfer does not. Exactly-once is a dial I turn per data product, sized by what an error costs — not a platform-wide default that taxes throughput and operability for streams that never needed it.

Q: How do you make a "make it cheaper" mandate concrete? Attribute cost first — tag every byte stored, byte scanned, and node-hour to an owner, so the bill is a map. Then the levers are mechanical: scan cost falls with partitioning + columnar + projection (often 10–100×); storage falls with compaction (kills the small-file request tax) and lifecycle tiering; compute falls with spot + right-sizing + AQE. Crucially I look for unused latency budget and spend it on cost — most "expensive" pipelines are optimising a freshness no one asked for.

Q: What's the difference between a senior and a principal data engineer, in one sentence? A senior makes one pipeline correct and fast; a principal makes the platform that makes every team's pipelines correct and fast by default, and can defend why with numbers in an ADR.

References

• Martin Kleppmann, Designing Data-Intensive Applications — the whole book is this role's bible; Ch. 1 (reliability/scalability/maintainability) is this phase
• Jay Kreps, The Log: What every software engineer should know (2013) — the lifecycle's unifying abstraction
• Google SRE Book — Service Level Objectives chapter (SLO/SLI/error-budget thinking applied to data in P13)
• Michael Nygard, Documenting Architecture Decisions (2011) — the ADR pattern
• The track's own CHEATSHEET.md and GLOSSARY.md

🛸 Hitchhiker's Guide — Phase 00: The Principal Data Infra Engineer

Read this if: you can build a data pipeline but "design our data platform" still sounds like a different job. It is. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP.

0. The 30-second mental model

You are hired to build the layer other engineers build on, and to be the person who, when latency and cost and correctness fight, picks the winner with numbers and writes down why. One sentence to tattoo: you don't own a pipeline, you own the lifecycle and the tradeoffs across it — encoded into a platform.

1. The lifecycle, one breath

creation → ingestion → validation → transport → processing →
storage → query → serving → governance → replay → deletion

Every technology in this track is one slot in that line. Learn the slot, then the tool. The bugs live in the seams between slots (processing-time logic that breaks on replay; PII that flows from creation to an open query table). Principals think in seams.

2. The five forces (score everything on these)

A "requirement" with no number on these axes is a wish, not a spec.

3. The dials you'll turn forever

low latency ⟷ low cost            (what does a minute of staleness cost?)
exactly-once ⟷ simplicity         (what does one dup / one loss cost?)
flexible schema ⟷ safe contract   (how many consumers, how costly is breakage?)
self-service ⟷ governance         (answer is usually: self-service WITH guardrails)
freshness ⟷ replay-correctness    (can you replay deterministically? → Kappa, else Lambda)
serving speed ⟷ analytical reach  (you need both; wire them safely — P11)
cloud lock-in ⟷ OSS portability   (fast to adopt vs yours to leave)

The interview signal: when they add "…and cheap," can you name the dial and price it?

4. The numbers you'll actually use

5. Budget the latency, spend the slack

Decompose an end-to-end SLA across stages. If the SLA is 15 min and you consume 42 s, the other 14 min is a cost lever — batch writes harder, use spot, compact lazily. Juniors optimise latency they don't need; principals convert it to dollars.

6. War story shapes you'll relive

• "Our counts doubled overnight." → a replay hit a processing-time aggregation. Fix is event-time + idempotency, not "stop replaying" (replay is the platform's best feature).
• "The query bill 10×'d." → someone wrote a million small files / dropped partitioning. The fix is layout, not a bigger warehouse.
• "12 jobs are down!" → they all read one Hive metastore. One incident, not twelve. Ask "what's the shared resource?" first.
• "It's exactly-once, we're fine." → only inside Kafka. Crossing into Cassandra you re-earn it (upsert/dedup) or you don't have it.

7. Vocabulary that signals you've held the pager

• Blast radius / failure domain — what fails together; the first incident question.
• Exactly-once effect — the only exactly-once that exists.
• Data product — a dataset with an owner, SLA, contract, lineage.
• Backpressure — downstream slowness as upstream flow control.
• ADR — the durable record of a decision and what would reverse it.
• Kappa vs Lambda — one replayable streaming path vs two code paths.
• Self-service with guardrails — the only scalable answer to the governance dial.

8. Beginner mistakes that mark you in interviews

1. Jumping to "use Kafka" before extracting numbers and the read:write ratio.
2. Saying "real-time" or "scalable" with no percentile / no peak number.
3. Treating exactly-once as a free platform default instead of a per-product dial.
4. Optimising latency the SLA didn't ask for (and paying for it).
5. Designing without a failure-domain picture, then being surprised by blast radius.
6. Making decisions in chat instead of ADRs — so they're re-litigated every quarter.

9. How this phase pays off later

• The lifecycle is the spine: every later phase fills one slot.
• The five forces + dials are the scoring function behind every later architecture choice and every interview round.
• The latency budget + cost model (the lab) is literally Round-1 mental math.
• The ADR habit is Round-5 (architecture leadership) and your day job as a principal.

Now read the WARMUP slowly, then build the modeler. You'll use its functions as back-of-envelope checks for the rest of the track.

👨🏻 Brother Talk — Phase 00, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. You've probably been told that becoming a principal data engineer means learning more tools. It doesn't. I've watched brilliant engineers memorise Flink, Spark, Iceberg, Kafka — all of it — and still get stuck at senior, because nobody told them the secret:

The tools are the easy part. The judgment is the job.

Here's the thing that took me too long to learn. When a director says "design our event platform," they are not testing whether you know acks=all. They already assume you do. What they're watching is: do you ask what a duplicate costs before you promise exactly-once? Do you ask the read-to-write ratio before you pick a database? Do you say "a minute of staleness is worth roughly $X to us, so I'll trade freshness for cost here"? That sentence — pricing a tradeoff out loud — is the whole game. Everyone else in the room is reciting features. You're doing engineering.

So let me give you the mindset shifts, brother to brother.

Stop saying "real-time." I mean it. The word is banned for you now. Every time you want to say "real-time," say a number and a percentile instead: "p99 under 60 seconds." The moment you do this, you sound three levels more senior, because you've revealed you know that "real-time" is a budget, not a property. A junior wants real-time everywhere. A principal knows real-time is the most expensive thing on the menu and orders it only where the business pays for it.

Fall in love with the boring force: operability. Everyone optimises latency and throughput because they're sexy and they benchmark well. Nobody brags about "this is easy to debug at 3 a.m." But operability is the force that actually decides whether your platform survives, because a system runs for years and gets debugged thousands of times. The "elegant" exactly-once pipeline that takes a PhD to debug is a liability. The "boring" at-least-once-with-dedup pipeline that any on-call can reason about is an asset. Choose boring on purpose. It's a flex, not a compromise.

Replay is your superpower; protect it. The first time you have an incident — and you will — and you can say "no problem, I'll replay the last six hours from the log and the output will be byte-identical because everything's event-time and idempotent" — that's the moment people start treating you like a principal. So everything you build, ask: can I replay this and get the same answer? If the answer is no, you haven't built a pipeline, you've built a time bomb. The whole reason we obsess over event-time and idempotency in this track is to keep that superpower.

Write the ADR even when nobody asks. Here's a career hack that feels like overhead and pays off enormously: when you make a real decision, write the one-pager — context, decision, alternatives, what would change my mind. Do it even for decisions nobody requested documentation for. Six months later when someone says "why did we do it this way?" you don't defend it from memory and emotion — you point at the doc and say "here were the numbers; have they changed?" You become the person whose decisions are legible. That person gets trusted with bigger decisions. That's the ladder.

The honest truth about this role's difficulty: it's hard because it's wide, not because any one piece is deep beyond reach. Nobody is born knowing Kafka and Spark and Cassandra and Cats. The people who get here didn't have bigger brains; they had a framework (the lifecycle, the five forces, the dials) that let them learn each new tool as "oh, this is the streaming-substrate slot, here's how it answers the same five questions." That framework is this entire phase. Internalise it and the other 16 phases stop being 16 unrelated manuals and become one system you already understand.

One more thing, and it's the most important. You don't have to be the smartest person in the room to be the principal in the room. You have to be the one who turns vague fear ("will this scale? is this correct? what will it cost?") into specific, answerable questions with numbers attached. Calm comes from numbers. When the room is panicking about "will this handle Black Friday," and you say "peak is 4× average, we're sized to 6×, here's the headroom, here's what sheds load first if I'm wrong" — you're not smarter than them. You just did the arithmetic they were too anxious to do. That's leadership in this job. It's learnable. It starts with the lab in this phase.

Go build the modeler. Then come find me in Phase 01 — that's where we make the distributed systems stuff click, the stuff everyone pretends to understand and quietly doesn't.

— your brother 👨🏻

Lab 01 — Tradeoff & Lifecycle Modeler

Phase: 00 — The Principal Data Infrastructure Engineer | Difficulty: ⭐⭐☆☆☆ | Time: 3–4 hours

Round-1 of a data-platform interview is back-of-envelope arithmetic worn as architecture. This lab builds the calculator: latency budgets, cost models, sizing, quorum math, delivery-semantics classification, and a weighted tradeoff resolver whose whole point is that the same two architectures flip winners when the workload's weights change — which is why "best architecture" is a meaningless phrase.

What you build

• LifecycleStage — the 11-stage spine of the entire track, as an enum
• allocate_latency_budget / budget_headroom_ms — decompose an SLA across stages with budget conservation, and find the slack that's a cost lever
• athena_scan_cost_usd / s3_storage_cost_usd / spot_compute_cost_usd — the three cost models that actually decide architectures
• kafka_partitions_needed / kinesis_shards_needed / quorum_is_strong — the sizing and consistency arithmetic (R + W > N, and the off-by-one that isn't strong)
• classify_delivery_semantics — at-most / at-least / exactly-once-effect
• resolve_tradeoff — score two architectures on the five forces under workload weights, and return the deciding force (the one sentence for your ADR)

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference

Success criteria

• All tests pass against your implementation.
• You can explain why test_quorum_strong_boundary asserts quorum_is_strong(3,2,1) is False (3 = 3, not > 3).
• You can explain why test_weights_flip_the_winner is the whole point of the lab.
• You can take any "design X" prompt and, in 2 minutes, produce a latency budget and a scan-cost estimate on paper.

Extensions

• Add gdpr_deletion_cost(...) — model the cost of point-deletes across a partitioned lake (foreshadows P09 time-travel-vs-deletion and P14 retention).
• Add a recommend_engine(requirements) that returns Kafka-vs-Kinesis-vs-Pulsar (or Flink-vs-Spark-SS) with the deciding force — then defend it against a friend.
• Make resolve_tradeoff accept N architectures and return a ranked list with pairwise deciding forces.

Phase 01 — Distributed Systems Foundations for Data

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (30–40 hours) Prerequisites: Phase 00 (the lifecycle and the five forces)

Why This Phase Exists

The JD describes a person who can "debug at the byte, partition, checkpoint, offset, file, and query-plan level" and reason about "data correctness, ordering, replayability, idempotency, deduplication, schema compatibility … failure recovery." Every one of those words is a distributed systems word. Kafka, Flink, Spark, Cassandra, and Iceberg are not five things to learn — they are five expressions of the same handful of distributed systems ideas. Learn the ideas once, here, and the rest of the track is "how does this specific engine implement the thing I already understand."

This is the phase that separates engineers who use distributed systems from engineers who reason about them. The former are surprised by reordering, duplication, partial failure, and clock skew. The latter expect all four and design so they don't matter.

Concepts

• The log as the one abstraction under everything — append-only, ordered, offset-addressed; batch and streaming are two read modes of it (the Kreps/Kleppmann framing).
• Time: wall-clock vs logical clocks (Lamport, vector); why you must never order distributed events by wall time; happens-before vs concurrent; causality.
• Partitioning: stable hashing, per-key ordering, skew / hot partitions, and why % N is a resharding trap (consistent hashing as the fix).
• Replication: leader/follower, quorums (R + W > N), ISR, sync vs async, and the durability/availability arithmetic (tolerated failures = N − quorum).
• Consistency models: linearizable → sequential → causal → eventual; read-your-writes; CAP (under a partition: C or A) and PACELC (else: latency or consistency — the everyday tradeoff CAP omits).
• Failure: the eight fallacies of distributed computing; partial failure; the Two Generals / FLP impossibility results and what they mean practically; gray failures.
• Delivery & effect: at-most/at-least/exactly-once; idempotency, dedup, the outbox pattern; exactly-once-effect as the only achievable "exactly-once."
• Backpressure & flow control: the queueing-theory reason unbounded buffers are bugs (Little's Law, bounded queues, load shedding).

Labs

Lab 01 — Partitioner, Idempotency & Exactly-Once Kit (flagship, implemented)

→ Lab folder: lab-01-exactly-once-kit/

Extension Project A — Consistent Hashing Reshard (spec)

Implement a hash ring with virtual nodes; measure the fraction of keys that move when a partition is added vs % N (the difference is the whole reason resharding is survivable).

Extension Project B — Two Generals Demonstrator (spec)

Simulate a lossy channel and prove empirically that no fixed number of acknowledgements achieves certain agreement — then show how at-least-once + idempotency sidesteps the impossibility by changing the goal from "agree" to "same effect."

Integrated-Scenario Hooks

• Hot partition (this phase's skew math) → Kafka key choice (P02), Kinesis shards (P02), Cassandra partition keys (P11), Spark data skew (P06). It is the same problem four times.
• Exactly-once effect (this phase's dedup) → Flink sinks (P04), CDC apply (P05), Iceberg commits (P09), Cassandra upserts (P11).
• Quorum math → Kafka ISR/acks (P02), Cassandra consistency levels (P11), multi-region (P14/P15).

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• Five engines, one set of ideas: partitioning, logical time, replication, consistency, failure, idempotency, backpressure.
• Wall clocks lie; order with logical clocks or sequence ids.
• "Exactly-once" is exactly-once effect, re-earned at every boundary.
• Durability and consistency are arithmetic (R + W > N; N − quorum), not adjectives.

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass against solution.py
• You can state CAP and PACELC and give a data example of each choice
• You can explain the same "hot partition" problem in Kafka, Cassandra, and Spark terms
• Extension A or B attempted

Warmup Guide — Distributed Systems Foundations for Data

Zero-to-principal primer for Phase 01: the log, logical time, partitioning, replication, consistency, failure, idempotency, and backpressure — the small set of ideas that every engine in this track (Kafka, Flink, Spark, Cassandra, Iceberg) is a different expression of. Learn them once; recognise them everywhere after.

Table of Contents

• Chapter 1: The Log — One Abstraction Under Everything
• Chapter 2: Time, Clocks, and the Ordering of Events
• Chapter 3: Partitioning, Ordering, and Skew
• Chapter 4: Replication and the Arithmetic of Durability
• Chapter 5: Consistency Models, CAP, and PACELC
• Chapter 6: Failure — The Default State of a Distributed System
• Chapter 7: Delivery Semantics and Idempotency
• Chapter 8: Backpressure and Flow Control
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Log — One Abstraction Under Everything

From zero. A log is an append-only, totally-ordered sequence of records, each addressed by a monotonic integer offset. That is the entire abstraction — and it is the substrate of the whole field:

• A Kafka topic-partition is a log.
• A database's replication stream / WAL is a log.
• A table is a log compacted by primary key (keep the latest record per key).
• "Batch" is reading a bounded slice of a log; "streaming" is reading its unbounded tail. They are not two systems — they are two read modes of one log (Kreps's "Kappa" insight; the reason Lambda architectures with two codebases rot).

Why does a principal start here? Because the log is what makes replay possible, and replay is the most valuable property a data platform has: it is how you backfill, how you recover from a bug, how you re-derive a corrupted dataset, how you bootstrap a new consumer. Architectures that discard the log (transform-in-place, no retained source of truth) trade away their own recoverability. Hold this: every "we can't reproduce last month's numbers" incident is a log that wasn't kept or wasn't versioned.

Chapter 2: Time, Clocks, and the Ordering of Events

The single most expensive misconception in distributed data: that you can order events by their wall-clock timestamps. You cannot. Physical clocks on different machines drift, get corrected by NTP (and jump backwards), and are simply unsynchronised at the millisecond scale that matters. Ordering distributed events by wall time produces silently wrong results — duplicated counts, negative durations, "effects before causes."

The principled tools:

• Lamport clock (scalar): each node keeps a counter; bump it on every local event; on receiving a message stamped t, set local = max(local, t) + 1. Guarantee: if event a happened-before b, then L(a) < L(b). The converse is not true — L(a) < L(b) does not prove causality, which is the clock's limitation.
• Vector clock: each node keeps a vector of per-node counters. Now you can decide precisely: a happened-before b (every entry ≤ and at least one <), they are equal, or they are concurrent (neither precedes the other). Concurrency detection is what you need to find genuine write conflicts (P11) and to reason about out-of-order events.
• Sequence numbers / offsets: within a single partition, the log's offset is a total order — which is exactly why per-key ordering (next chapter) is so valuable.
• Event-time + watermarks (P04): the streaming world's pragmatic answer — trust the event's own timestamp but bound your belief about how out-of-order it can be.

The senior rule: use wall clocks for human-facing display and SLA measurement only; order computation by logical time, offsets, or event-time-with-watermarks — never by arrival.

Chapter 3: Partitioning, Ordering, and Skew

To scale beyond one machine you partition: split the keyspace across N shards by partition = stable_hash(key) % N. Three consequences a principal internalises:

1. Ordering is per-partition only. A total order across all data does not exist at scale, by design. What you engineer with is per-key ordering: route all events for account_42 to one partition (one key → one partition) and they stay ordered. Choosing the partition key is therefore choosing both your ordering guarantee and your failure-coupling.
2. The hash must be stable. Language built-in hashes (Python's hash()) are salted per process and change across restarts — using one for partitioning reshuffles every key on every restart and destroys per-key ordering. Use FNV/Murmur/CRC (the lab implements FNV-1a). This is a real, shipped-to-prod bug class.
3. Skew = hot partitions. If the key is low-cardinality or Zipfian (a few "whale" keys), some partitions get vastly more load. The skew factor = max_load / mean_load is the number you cite ("this key gives a 9× hot partition"). It is the same problem as Kafka hot partitions (P02), Kinesis hot shards (P02), Cassandra hot partitions (P11), and Spark data skew (P06). Fixes: choose a higher-cardinality key, add a salt/bucket component, or use consistent hashing.

Why % N is a resharding trap: changing N remaps almost every key to a new partition (a near-total reshuffle). Consistent hashing (a hash ring with virtual nodes) moves only ~1/N of keys when you add a node — the reason real systems (Cassandra, Dynamo) use it. Extension Project A makes you measure the difference.

Chapter 4: Replication and the Arithmetic of Durability

You replicate data to survive machine loss. The models:

• Leader/follower (primary/replica): writes go to the leader, which ships them to followers — synchronously (durable but slow, blocks on the slowest follower) or asynchronously (fast but can lose recent writes on leader failure). Kafka's ISR (in-sync replicas) + acks=all + min.insync.replicas is exactly this dial: a write is acknowledged only once it's on ≥ min.insync.replicas members of the ISR.
• Leaderless / quorum (Dynamo-style): write to W replicas, read from R, and if R + W > N the read set and write set must overlap on at least one up-to-date replica → you read your writes. This is the math behind Cassandra consistency levels (P11). It is a strict inequality: R + W = N is not strong (the lab's boundary test).
• Durability/availability arithmetic: a quorum operation tolerates N − quorum failures (RF=3, W=2 → survives 1 node down). Sizing redundancy is this subtraction, not a vibe.

The principal framing: replication gives you a dial between durability, latency, and availability, and the right setting is per data product sized by what a lost write costs.

Chapter 5: Consistency Models, CAP, and PACELC

A consistency model is a contract about what a read can return given prior writes. The ladder, strongest to weakest:

• Linearizable (strong): every operation appears to take effect atomically at a single point between its call and return; there is one real-time order. Expensive; needed for things like leader election and unique-constraint enforcement.
• Sequential / causal: operations respect program/causal order but not necessarily real-time. Causal consistency (respect happens-before) is often the sweet spot.
• Read-your-writes / monotonic reads: session guarantees — you never see your own write vanish, never go backwards in time. Often what users actually need.
• Eventual: replicas converge if writes stop; in between, reads can be stale or conflicting. Cheapest, highest availability.

CAP: when a network Partition happens, you must choose Consistency or Availability — you cannot have both while partitioned. But CAP is a trap because partitions are rare; the everyday tradeoff is PACELC: if Partition then C-or-A, Else (normal operation) then Latency-or-Consistency. The Else clause is the one you tune daily — every "should this read hit the leader or a follower?" is an L-vs-C decision. Saying "PACELC" instead of "CAP" in an interview signals you've operated these systems, not just read the Wikipedia page.

Chapter 6: Failure — The Default State of a Distributed System

In a single process, components either work or the process dies. In a distributed system, the normal state is partial failure: some nodes up, some down, some slow, some lying, the network dropping/reordering/duplicating/delaying messages. Foundational results:

• The 8 fallacies of distributed computing: the network is not reliable, latency is not zero, bandwidth is not infinite, the network is not secure, topology changes, there is not one admin, transport cost is not zero, the network is not homogeneous. Every one is a bug if you assume otherwise.
• Two Generals Problem: over a lossy channel, two parties can never be certain they've agreed — no finite number of acknowledgements suffices. Practical consequence: you cannot guarantee "delivered exactly once" over a network. So you change the goal (Ch. 7).
• FLP impossibility: in a fully asynchronous system with even one faulty process, no deterministic consensus algorithm can guarantee termination. Practical consequence: real consensus (Raft/Paxos, Kafka's KRaft) uses timeouts/randomness to make progress in practice while accepting the theoretical caveat.
• Gray failure: the worst kind — a node that is slow or partially broken but still "up" to health checks, silently poisoning the system (the stalled follower that's still in the ISR; the disk that's 100× slower but not dead). Detecting gray failure is most of real on-call (P13).

The mindset shift: assume reorder, duplication, loss, delay, and partial failure as the baseline, and design so they don't change the answer. That design is the next chapter.

Chapter 7: Delivery Semantics and Idempotency

Given that the network can lose and duplicate, what can a pipeline promise?

• At-most-once: send and forget; on failure, data is lost. Rarely acceptable.
• At-least-once: retry until acknowledged; on failure, duplicates. The practical default — the system must then tolerate duplicates.
• "Exactly-once" does not mean each message crosses the wire once (Two Generals forbids it). It means at-least-once delivery + idempotent or transactional processing = exactly-once effect. The result is as if each event were processed once.

Two ways to buy it:

1. Idempotency / dedup: make re-processing a no-op. Dedup by a stable event id (the lab's DedupStore), or use a naturally idempotent sink (upsert by key, set-union, max). Critical subtlety: you can't remember every id forever, so dedup state is bounded (evicted behind a watermark) — which means a duplicate arriving later than the window can slip through. Bounded dedup is a correctness tradeoff, not a free optimisation; you size the window against how late duplicates realistically arrive.
2. Transactions / atomic commit: commit the output and the input offset atomically, so a crash can't leave them disagreeing. Kafka's transactional producer + read_committed, or the outbox pattern at a service boundary (write the business row and an "event to publish" row in one DB transaction; a relay publishes the outbox).

The interview-grade sentence: "exactly-once is a property of the pipeline's effect on state, purchased with idempotency or atomic offset+output commits, and re-earned at every external boundary — not a network guarantee."

Chapter 8: Backpressure and Flow Control

When a downstream stage is slower than its upstream, something must give. The naive answer — an unbounded in-memory buffer — is always a bug: it converts a throughput problem into an out-of-memory crash, and it hides the slowdown until it's catastrophic. The principled answers:

• Bounded queues + backpressure: when the buffer fills, slow the producer (block, or signal it to send less). The slowness propagates upstream as flow control instead of silent latency growth. This is Reactive Streams / Akka Streams / Flink's credit-based flow control / Kafka consumer max.poll.records.
• Little's Law (L = λ × W): the average number of in-flight items equals arrival rate × time in system. It tells you the buffer size you need and what happens when W grows (latency spike) — the queueing-theory floor under all capacity planning (P13).
• Load shedding: when you can't slow the producer (it's the outside world), drop or sample on purpose, with a metric, rather than collapsing. Better to consciously drop 1% than to topple and lose 100%.

Backpressure is the central health signal in stream processing (P04): rising consumer lag and rising checkpoint duration are backpressure wearing different costumes.

Lab Walkthrough Guidance

Lab 01 — Exactly-Once Kit, suggested order:

1. fnv1a_32 then hash_partition — verify determinism (same key → same partition across calls); this is the per-key-ordering foundation.
2. partition_load + skew_factor — feel the difference between uniform keys (~1.0) and a "whale" key (large) — Ch. 3 made tangible.
3. LamportClock then VectorClock — test that causal chains order under Lamport, and that two independent local ticks are concurrent under vectors (Ch. 2).
4. DedupStore.check_and_add (first True, then False) and evict (bounded state) — Ch. 7.
5. exactly_once_apply + simulate_unreliable_delivery — then the payoff test: a reordered, duplicated delivery yields the identical final state (Ch. 7).
6. quorum_is_strong / tolerated_failures — Ch. 4's arithmetic, including the boundary R + W = N is not strong.

Success Criteria

You are ready for Phase 02 when you can, from memory:

1. Explain why batch and streaming are read modes of one log, and why that makes replay the platform's key property.
2. Give a concrete data bug caused by ordering events by wall clock, and the logical-clock fix.
3. State Kafka's ordering guarantee precisely and explain what choosing the partition key decides.
4. Compute the durability and read-your-writes properties of RF=3, R=2, W=2.
5. State CAP and PACELC and give a per-clause data example.
6. Define exactly-once-effect and both ways to buy it, including why bounded dedup is a correctness tradeoff.
7. Explain why an unbounded buffer is a bug and what backpressure does instead.

Interview Q&A

Q: A consumer was restarted and now your hourly counts are wrong. Walk me through the suspects. First, duplication: at-least-once redelivery after the restart re-counted events — fixed by dedup on a stable id (and I'd check whether dedup state was bounded such that post-restart duplicates fell outside the window). Second, ordering/partitioning: if the partition key or hash changed (e.g. someone used a salted hash()), keys remapped and per-key aggregation broke. Third, time: if the aggregation keyed on processing time, the restart's catch-up replayed old events into the wrong buckets. The permanent fix is event-time windows + idempotent dedup + a stable partitioner — the three together make the pipeline replayable, which is the actual goal.

Q: Your team wants "strong consistency and high availability." What do you say? That under a network partition they must pick one (CAP) — but I'd reframe to PACELC because partitions are rare and the daily tradeoff is latency vs consistency in normal operation. Then I'd ask what the data is: a balance check needs read-your-writes (route to leader, accept the latency); a recommendation feed is fine eventual (read any replica, win the latency). "Strong + available" isn't one setting; it's a per-data-product dial, and most products don't need the strong end.

Q: What does a vector clock give you that a Lamport clock doesn't, and when do you care? A Lamport clock gives a total order consistent with causality, but L(a) < L(b) can't tell you whether a actually caused b or they were independent. A vector clock can distinguish happens-before from concurrent — which is exactly what you need to detect a genuine write conflict in a leaderless store (two replicas updated the same key independently): concurrent writes need conflict resolution (LWW, or a CRDT merge), causal ones don't. So I care the moment I have multi-writer state — Cassandra, Dynamo, multi-region.

Q: Define exactly-once and then tell me why it's "a lie." Exactly-once delivery over a network is impossible (Two Generals). What's real is exactly-once effect: at-least-once delivery plus idempotent or transactional processing, so retries don't change the result. It's "a lie" only if someone sells it as a network property or a free platform default — it's an end-to-end property of effects on state, re-established at every boundary (and it costs throughput and operability, so you spend it where an error is expensive, not everywhere).

References

• Martin Kleppmann, Designing Data-Intensive Applications — Ch. 5 (replication), 6 (partitioning), 7 (transactions), 8 (the trouble with distributed systems), 9 (consistency & consensus). This phase is those chapters.
• Leslie Lamport, Time, Clocks, and the Ordering of Events (1978) — logical clocks
• Gilbert & Lynch, Brewer's Conjecture (2002) — the CAP proof; Abadi, PACELC (2012)
• Fischer, Lynch, Paterson, Impossibility of Distributed Consensus with One Faulty Process (FLP, 1985)
• Jay Kreps, The Log (2013) — the unifying abstraction
• Nygard, Release It! — bulkheads, circuit breakers, backpressure in practice

🛸 Hitchhiker's Guide — Phase 01: Distributed Systems Foundations

Read this if: you want the distributed-systems vocabulary and numbers fast, before the WARMUP derives them slowly. These are the ideas that make Kafka, Flink, Spark, Cassandra, and Iceberg stop being five manuals and become one.

0. The 30-second mental model

Everything is a log (append-only, offset-addressed). You scale it by partitioning (per-key order only). You survive machine death by replicating (R + W > N math). The network lies (loses, dups, reorders, delays), so you never trust wall clocks and you make re-processing a no-op (idempotency) — which is the only "exactly-once" that exists. One sentence: assume reorder + duplication + partial failure as normal, and design so they don't change the answer.

1. The five engines, one idea each

Learn the idea, recognise the engine.

2. The numbers to tattoo

strong consistency:    R + W > N      (R + W = N is NOT strong — off by one)
failure tolerance:     N − quorum     (RF=3, W=2 → survive 1)
skew factor:           max_load / mean_load   (1.0 = balanced)
consistent hashing:    add a node → ~1/N keys move  (vs % N → ~all move)
Little's Law:          L = λ × W      (in-flight = rate × latency)
ordering:              per-partition only; never total at scale

3. Time: the rule that saves you

Never order distributed events by wall clock. Clocks drift, NTP jumps backwards. Order by: offsets (within a partition), Lamport (causality ⇒ order), vector clocks (also detects concurrent), or event-time + watermarks (P04). Wall clocks are for human display and SLA stopwatches only.

• Lamport: local = max(local, received) + 1. Gives order; can't prove causality.
• Vector: per-node counters. Tells you happens-before vs concurrent = conflict detection.

4. Partitioning gotchas (the same bug, four costumes)

• Salted hash bug: using hash() (salted per process) for partitioning → keys reshuffle on restart → per-key ordering destroyed. Use FNV/Murmur/CRC.
• Hot partition: low-cardinality / Zipfian key → one shard melts. Same in Kafka, Kinesis, Cassandra, Spark. Fix: better key, salt/bucket, or consistent hashing.
• % N resharding: changing N remaps almost every key. Consistent hashing moves ~1/N.

5. CAP is a trap; say PACELC

• CAP: during a partition, choose C or A. (Rare event.)
• PACELC: if Partition → C/A; Else → Latency or Consistency. The Else clause is the tradeoff you tune every day (leader read = consistent+slow; follower read = fast+maybe stale).

6. The impossibility results (and the practical escape)

• Two Generals: can't guarantee agreement over a lossy channel → can't do exactly-once delivery. Escape: change the goal to exactly-once effect (idempotency).
• FLP: async consensus can't guarantee termination with a faulty node. Escape: real systems (Raft/KRaft) use timeouts to make progress in practice.
• Gray failure: a node that's slow, not dead, still passing health checks. The worst outages. Most of on-call is catching these.

7. Exactly-once, decoded

at-most-once   = no retries        → may lose
at-least-once  = retry till acked  → may dup   ← practical default
exactly-once   = at-least-once + (idempotency | transactions) = exactly-once EFFECT

Buy it with: dedup-by-stable-id (bounded! late dups past the window slip through — a real tradeoff), idempotent sinks (upsert/set-union/max), or the outbox pattern. Re-earned at every boundary.

8. Backpressure: unbounded buffer = bug

When downstream is slow, slow the producer (bounded queue + backpressure) — don't grow an unbounded buffer (that's an OOM with extra steps). When you can't slow the source, shed load on purpose with a metric. Rising lag and rising checkpoint time are backpressure in disguise (P04).

9. Beginner mistakes that mark you in interviews

1. Ordering events by event.timestamp (wall clock) and trusting it.
2. Saying "exactly-once" as if it's a network/platform freebie.
3. hash() (or any salted/unstable hash) for partitioning.
4. "Strong + highly available" with no mention of partitions / PACELC.
5. Quorum off-by-one: thinking R + W = N is strong.
6. Unbounded queues "for performance."
7. Assuming a node is either up or down (forgetting gray/slow failure).

10. How this phase pays off later

• Partitioning + skew → P02 (Kafka keys, Kinesis shards), P06 (Spark skew), P11 (Cassandra partitions). One mental model, four phases.
• Exactly-once effect → P04 (Flink sinks), P05 (CDC), P09 (Iceberg commits), P11 (upserts).
• Quorum math → P02 (ISR/acks), P11 (consistency levels), P14/P15 (multi-region).
• Backpressure → P04 (the central streaming health signal).
• Logical time → P04 (watermarks), P11 (conflict resolution).

Read the WARMUP slowly, build the kit, then go to P02 where the log becomes Kafka for real.

👨🏻 Brother Talk — Phase 01, Off the Record

The distributed-systems stuff everyone nods along to and quietly doesn't get. Let me save you the years it took me.

Brother, here's the thing nobody says out loud: most engineers, including senior ones, don't actually understand distributed systems. They've memorised the words. They'll say "eventually consistent" and "exactly-once" and "CAP theorem" in the right sentences, and if you push one inch — "okay, why is R + W = N not strong?" — the room goes quiet. I was that person for longer than I'd like to admit. So let me give you the handful of clicks that turned the words into understanding.

Click #1: The network is an adversary, not a pipe. Stop picturing the network as a slightly-slow wire that occasionally breaks. Picture a malicious gremlin who can drop your message, deliver it twice, deliver it out of order, deliver it ten minutes late, or — the evil one — make a machine slow instead of dead so your health checks say it's fine. Once you internalise the gremlin, every design question becomes "what does the gremlin do to this?" and suddenly idempotency and timeouts and bounded queues aren't "best practices" you memorise — they're obvious defenses against a specific enemy. The engineers who seem to "just get it" aren't smarter. They just stopped trusting the network years ago.

Click #2: Wall clocks are a lie you're emotionally attached to. This one's hard because timestamps feel so real. There's a number, it says 14:32:07.881, surely that's when it happened? No. That's when some machine's clock, which drifts and gets yanked around by NTP, thought it happened. I've personally debugged "negative event durations" (end before start) and "events from the future" — both from trusting wall clocks across machines. The day you truly stop trusting them and reach for offsets / logical clocks / event-time, you graduate. Say it with me: the timestamp is data the event carries, not the truth about when anything happened.

Click #3: "Exactly-once" is the field's most successful marketing lie. Vendors put it on slides. It sells. And it's technically impossible as delivery — the Two Generals proof is short and airtight; go read it, it'll take ten minutes and immunise you forever. What's real is exactly-once effect, and once you see that the trick is just "make doing it twice the same as doing it once" (idempotency), it stops being magic and becomes a design habit: every sink you build, you ask "what happens if this runs twice?" If the answer isn't "same result," you're not done. When an interviewer says "we need exactly-once," the senior answer is "into where, and what does a duplicate cost?" — because crossing into Cassandra or S3 you re-earn it, and most streams don't need the expensive version.

Click #4: hot partitions are one enemy wearing four masks. You'll spend your career fighting "this one shard is on fire." In Kafka it's a hot partition. In Kinesis it's a hot shard. In Cassandra it's a hot partition (again) plus a fat partition. In Spark it's data skew that makes one task run for 9 hours while 199 finish in seconds. It is the same problem every time: a key with too few distinct values or a Zipfian "whale." And the fixes rhyme: better key, add a salt/bucket, or spread differently. When you realise it's one enemy, you stop relearning it in each tool and start recognising it on sight. That recognition is what makes you look like a wizard in design reviews.

Click #5: the quorum math is just overlap. Don't memorise "R + W > N" as a spell. See it: if your write went to W replicas and your read asks R replicas, then as long as those two sets are forced to share at least one replica, your read can't miss your write. Two sets out of N must overlap exactly when R + W > N. That's it. That's the whole "consistency level" feature in Cassandra, the whole min.insync.replicas thing in Kafka. It's set overlap. Once you see it as overlap, you'll never get the off-by-one wrong, and you'll be able to derive the right consistency level live in an interview instead of recalling it.

Now, the career truth. This phase is the one that compounds the most, and it's the one most people skip because it's not a shiny tool you can put on your résumé. "Distributed systems foundations" isn't a line item; "Apache Flink" is. So everyone rushes to Flink and builds on sand. Don't. The reason a principal can walk into a Kafka problem and a Cassandra problem and a Spark problem and sound deep in all three is not that they read three manuals — it's that they understand partitioning, replication, time, and failure once, deeply, and see each tool as a variation. You're building that "once" right now. The lab feels small — a few hundred lines, hashing and dedup and quorum checks. But you are literally implementing the load-bearing ideas of every system in the next fifteen phases. Build it by hand. Make the exactly-once-under-chaos test pass. Feel the skew factor spike when you add a whale. That muscle memory is the foundation everything else screws into.

One last thing. When you're on-call at 3 a.m. and twelve alarms are screaming, the engineers who panic are the ones who see twelve problems. The ones who stay calm see the shared resource and ask "what one thing, failing, explains all of these?" That calm isn't a personality trait. It's this phase. It's knowing failure domains and blast radius in your bones, so a storm resolves into a single root cause. That calm is worth more than any tool on your résumé.

See you in Phase 02, where the log stops being a concept and becomes Kafka — with all its beautiful, sharp edges.

— your brother 👨🏻

Lab 01 — Partitioner, Idempotency & Exactly-Once Kit

Phase: 01 — Distributed Systems Foundations | Difficulty: ⭐⭐⭐☆☆ | Time: 5–6 hours

Every distributed data system is built from four primitives: partition the work, order it without trusting wall clocks, make re-processing safe, and replicate for durability without lying about consistency. This lab implements all four — and proves the one property the whole field exists to deliver: an unreliable, reordered, duplicated delivery produces the identical final state as a clean one.

What you build

• fnv1a_32 / hash_partition / partition_load / skew_factor — deterministic partitioning (the basis of per-key ordering) and the skew measurement that names a hot partition with a number
• LamportClock / VectorClock — logical time: Lamport (causality ⇒ order) and vector (distinguish happens-before from concurrent, the basis of conflict detection)
• DedupStore — bounded idempotency: first-seen is True, duplicates are no-ops, and state is evicted behind a watermark so it can't grow forever
• exactly_once_apply / simulate_unreliable_delivery — at-least-once delivery + dedup = exactly-once effect, demonstrated under reorder + duplication
• quorum_is_strong / tolerated_failures / quorum_available — replication math: R + W > N, and how many failures a quorum survives

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All tests pass — test_exactly_once_effect_under_duplicates_and_reorder and test_exactly_once_is_order_independent are the ones that certify understanding.
• You can explain why bounded dedup (evict) is a correctness tradeoff, not a free optimisation (a duplicate later than the retention window slips through).
• You can state, with numbers, the durability of RF=3 / W=2 / R=2.

Extensions

• Replace modulo partitioning with consistent hashing (a hash ring with virtual nodes) and measure how many keys move when you add a partition — the reason real systems don't use % N for resharding (foreshadows P02 rebalancing, P11 Cassandra).
• Add a transactional outbox: apply a state change and record its id atomically, then show crash-recovery doesn't double-apply (foreshadows P05 CDC, P04 exactly-once sinks).
• Implement read-repair: given replicas with vector-clocked values, return the merged value and detect a genuine conflict (foreshadows P11 last-write-wins vs CRDTs).

Phase 02 — Event Ingestion & Messaging

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (30–40 hours) Prerequisites: Phase 01 (the log, partitioning, replication, exactly-once)

Why This Phase Exists

Ingestion is the front door of the platform: if events are lost, reordered, duplicated, or unschematized here, nothing downstream can fix it. The JD's first hard problem is exactly this — "a multi-region event ingestion platform that accepts billions of events per day … Kafka or Kinesis … schema validation at ingress … dead-letter routing … reprocessing support … exactly-once or effectively-once persistence." This phase turns Phase 01's "the log" from theory into the two systems you'll actually run — Kafka and Kinesis — plus the operational machinery around them: consumer groups, retention, compaction, replay, DLQs, idempotent/transactional producers, and the migration off legacy Flume.

Concepts

• Kafka internals: brokers, topics, partitions, the segment-based log, the page cache + zero-copy path, replication & ISR, acks & min.insync.replicas, leader election, KRaft (vs ZooKeeper), controller, consumer groups & rebalancing protocols (eager vs cooperative-sticky), __consumer_offsets, retention vs compaction.
• Kinesis internals: shards (1 MB/s & 1000 rec/s in, 2 MB/s out), the partition key → shard hash, resharding (split/merge), enhanced fan-out, KCL & lease tables, vs Kafka.
• Choosing the substrate: Kafka (MSK) vs Kinesis vs Pulsar (broker/storage split, tiered storage, multi-tenancy) vs cloud queues — the decision matrix and its dials.
• Delivery & correctness on the wire: idempotent producers (PID + sequence), transactions (atomic offset+output within Kafka), read_committed, exactly-once vs effectively-once, and where the guarantee stops (the external sink).
• Ingestion safety: schema validation at ingress (handoff to P03), dead-letter queues / quarantine, poison messages, redrive, backpressure & overload, quotas.
• Replay & reprocessing: offset seek, retention windows, compacted topics as state, replay correctness (event-time + idempotency from P01).
• Hot partitions/shards: key design, salting, resharding — the P01 skew problem in its ingestion costume.
• Legacy migration: Flume (source→channel→sink) → Kafka/Kinesis; dual-write & cutover.

Labs

Lab 01 — Mini Broker (flagship, implemented)

→ Lab folder: lab-01-mini-broker/

Lab 02 — Partitioned-Log Ingestion in Scala (implemented, sbt test)

→ Lab folder: lab-02-scala-ingestion/

Extension Project A — Run real Kafka (spec)

docker compose single-broker Kafka; create a 6-partition topic; produce keyed events; demonstrate per-key ordering, consumer-group rebalancing (start/stop consumers), replay-from-offset, and a compacted topic. Wire a validating consumer with a quarantine topic.

Extension Project B — Flume → Kafka migration memo (spec)

One page + diagram: migrate a Flume spooldir → memory channel → HDFS sink ingest into Kafka/Flink, dual-writing during cutover, validating parity, and decommissioning Flume safely (the JD's explicit "migrate legacy Flume" requirement).

Integrated-Scenario Hooks

• This phase's DLQ + validation front-ends the schema registry (P03).
• Its replay feeds Flink reprocessing (P04) and Spark backfills (P06).
• Its compacted topics become CDC/KTable state (P05) and serving snapshots (P11).
• Its partitioning/shard sizing is the ingress of the capstone's global platform (P16).

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• Ingestion is unfixable downstream — get ordering, durability, and schema right here.
• Kafka and Kinesis are the same log idea with different operational models; choose by the dials, not by fashion.
• Replay + compaction + DLQ are the three operational superpowers; build them in from day 1.
• Exactly-once on the wire stops at Kafka's edge; the external sink re-earns it (P04/P09/P11).

Deliverables Checklist

• Lab 01 implemented; all 15 tests pass
• You can size partitions/shards for a stated throughput and consumer parallelism
• You can explain ISR + acks=all + min.insync.replicas durability precisely
• Extension A demo or Extension B memo

Warmup Guide — Event Ingestion & Messaging

Zero-to-principal primer for Phase 02: Kafka and Kinesis from the inside, the durability and ordering knobs that decide correctness, consumer groups and rebalancing, retention vs compaction, transactions and idempotent producers, dead-letter safety, replay, and the migration off legacy Flume.

Table of Contents

• Chapter 1: Kafka's Anatomy
• Chapter 2: The Log on Disk — Segments, Page Cache, Zero-Copy
• Chapter 3: Durability — acks, ISR, and Leader Election
• Chapter 4: Producers — Partitioning, Idempotence, Transactions
• Chapter 5: Consumer Groups and Rebalancing
• Chapter 6: Retention vs Compaction
• Chapter 7: Kinesis — The Managed Cousin
• Chapter 8: Choosing the Substrate — Kafka vs Kinesis vs Pulsar
• Chapter 9: Ingestion Safety — DLQs, Poison Messages, Backpressure
• Chapter 10: Replay, Reprocessing, and Legacy Flume Migration
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Kafka's Anatomy

Kafka is a cluster of brokers storing topics, each split into partitions — and a partition is exactly the log from P01: append-only, offset-addressed, ordered. The pieces:

• Broker: a server holding some partitions (as leader for some, follower for others).
• Partition: the unit of parallelism, ordering, and replication. More partitions = more consumer parallelism and more producer throughput, but also more open file handles, more rebalance cost, more end-to-end latency, and (until recently) more controller load. Sizing partitions is a real design decision, not a default.
• Replica / leader / follower: each partition has a replication factor (RF, usually 3) copies; one is the leader (handles all reads/writes) and the rest are followers that replicate. Clients only talk to the leader.
• Controller: one broker coordinates leadership and cluster metadata. Historically this lived in ZooKeeper; modern Kafka uses KRaft (a self-managed Raft quorum) — fewer moving parts, faster failover, the thing to mention as "current."
• __consumer_offsets: an internal compacted topic where consumer groups store their committed offsets — Kafka eating its own dog food (offsets are just a compacted log).

The mental model to keep: a topic is a sharded log; a partition is a single log; ordering is per-partition; everything else is operations around that.

Chapter 2: The Log on Disk — Segments, Page Cache, Zero-Copy

Why is Kafka fast despite writing everything to disk? Three mechanical reasons worth knowing because they explain its limits:

• Sequential I/O: appending to a log is sequential disk write, which is ~hundreds of MB/s even on spinning disks — orders of magnitude faster than random I/O. Kafka's whole performance story is "never do random I/O."
• Segments: each partition is split into segment files (+ an offset index and time index). Retention deletes whole old segments (cheap), which is why retention is coarse (segment-granular), not per-record — and why log_start_offset advances in chunks.
• Page cache + zero-copy: Kafka doesn't maintain its own cache; it relies on the OS page cache, and serves consumers via sendfile (zero-copy) — bytes go disk→NIC without passing through user space. This is why a consumer reading the tail (hot in page cache) is nearly free, and why a consumer reading old data (cold, e.g. a big replay) causes disk I/O and can hurt the brokers — a real operational gotcha during large backfills.

The lab models offsets and start_offset faithfully so this maps directly: retention advancing start_offset is segments being deleted.

Chapter 3: Durability — acks, ISR, and Leader Election

This is the chapter interviewers probe hardest because it's where correctness lives.

• ISR (in-sync replicas): the set of replicas currently caught up to the leader. A follower that falls too far behind is removed from the ISR (and re-added when it catches up).
• acks (producer): acks=0 (fire-and-forget, may lose), acks=1 (leader only — lost if the leader dies before a follower replicates), acks=all (leader waits for all in-sync replicas).
• min.insync.replicas (topic): with acks=all, a write only succeeds if at least this many replicas are in-sync. The classic safe config is RF=3, min.insync.replicas=2, acks=all: you can lose one broker and still accept writes and never lose an acknowledged write. This is P01's quorum math (R + W > N → 2 + 2 > 3) wearing Kafka's clothes.
• unclean.leader.election: if all in-sync replicas die, do you (a) promote an out-of-sync replica (available but loses data) or (b) refuse writes until an in-sync one returns (consistent but unavailable)? This toggle is the CAP choice, exposed as a config. Default it false for any data you care about.

The durability sentence to own: "RF=3, acks=all, min.insync.replicas=2 survives one broker loss with no acknowledged-write loss; the moment unclean leader election is on, you've traded that for availability."

Chapter 4: Producers — Partitioning, Idempotence, Transactions

• Partitioning: a record with a key hashes to a partition (per-key ordering); a record without a key is distributed (sticky-partitioner batching in modern clients). Key choice = ordering + skew. A low-cardinality key (e.g. country) creates hot partitions (P01's skew problem); fix with a higher-cardinality key or a key + bucket salt.
• Idempotent producer (enable.idempotence=true, now default): the producer gets a producer id (PID) and stamps each record with a sequence number per partition, so the broker can drop a retried duplicate — exactly-once append under producer retries. Note the scope: this dedupes producer retries, not application-level duplicates.
• Transactions: a producer can write to multiple partitions and commit its consumer offsets atomically (initTransactions / sendOffsetsToTransaction / commit). Consumers with isolation.level=read_committed only see committed records. This is how Kafka offers exactly-once within Kafka (consume→process→produce). The instant you write to an external sink (S3, Cassandra), you leave Kafka's transaction and re-earn the guarantee yourself (P04 two-phase-commit sinks, P09 Iceberg commits, P11 upserts).

Chapter 5: Consumer Groups and Rebalancing

A consumer group is a set of consumers cooperatively reading a topic; each partition is assigned to exactly one consumer in the group, so group parallelism is capped at the partition count (extra consumers sit idle — the lab tests this). Mechanics:

• Assignment strategies: range (contiguous blocks per topic — can imbalance across multiple topics), roundrobin (even but reshuffles more), sticky / cooperative-sticky (minimise partition movement on rebalance — the modern default).
• Rebalancing: when a member joins/leaves/dies (or partitions change), the group reassigns partitions. Eager rebalancing stops the world (everyone revokes, then re-joins) — a latency hit; cooperative rebalancing moves only the partitions that actually need to move. Frequent rebalances (from max.poll.interval.ms timeouts because a consumer is slow) are a classic production pain — the symptom is "lag sawtooths and throughput drops."
• Offset commits: a consumer commits the next offset to read (the lab's commit_through(last) = last+1). Commit after processing for at-least-once; commit before for at-most-once. Auto-commit is convenient and a duplicate/loss footgun under failure — principals usually commit manually around the processing boundary.

Chapter 6: Retention vs Compaction

Two completely different ways a topic forgets:

• Delete retention (cleanup.policy=delete): drop data older than retention.ms or beyond retention.bytes. The log is a time window of events. log_start_offset advances as old segments are deleted; offsets never renumber (the lab models this exactly).
• Log compaction (cleanup.policy=compact): keep at least the latest value per key forever; older values for a key are garbage-collected; a record with a null value (tombstone) marks a key for deletion. A compacted topic is a changelog that doubles as a table — the foundation of Kafka Streams' KTables, CDC snapshots, and Kafka's own __consumer_offsets. You can also combine (compact,delete).

When to use which: events → delete retention (you want the history window); current-state / config / CDC → compaction (you want the latest per entity, compactly). The lab's compacted_view is this fold.

Chapter 7: Kinesis — The Managed Cousin

Kinesis Data Streams is "Kafka as an AWS service," with different vocabulary and limits:

• Shard ≈ partition, but with hard quotas: 1 MB/s and 1000 records/s ingress, 2 MB/s egress per shard. Throughput is shards × those limits — so you size in shards (the lab's kinesis_shards_needed).
• Partition key → shard: an MD5 hash of the partition key maps records to a shard's hash range; ordering is per-shard. Same hot-key problem, same fixes.
• Resharding: split a hot shard or merge cold ones — but resharding is a manual, rate-limited operation (vs Kafka's fixed partitions), and consumers must handle parent→ child shard lineage. On-Demand mode auto-scales shards at a cost premium.
• Consumption: classic GetRecords is polled and shared (5 tx/s, 2 MB/s per shard across all consumers); Enhanced Fan-Out gives each consumer a dedicated 2 MB/s push pipe (lower latency, more cost). The KCL (Kinesis Client Library) manages leases & checkpoints in a DynamoDB lease table — the moral equivalent of consumer groups + __consumer_offsets.
• Retention: 24h default, extendable to 365 days; replay = re-read by sequence number / timestamp.
• Firehose is a different product: managed delivery to S3/Redshift/OpenSearch with buffering — no replay, no custom processing; great for "just land it," wrong for stream processing.

Chapter 8: Choosing the Substrate — Kafka vs Kinesis vs Pulsar

The decision matrix (and the dial each row turns — P00):

The principal answer is never "use X." It's: "all-AWS team optimising for low ops and we're already in the ecosystem → Kinesis/MSK; need maximum throughput control, the Connect/Streams ecosystem, or cloud portability → Kafka; need elastic storage decoupled from compute and first-class multi-tenancy/geo → Pulsar." Then write the ADR.

Chapter 9: Ingestion Safety — DLQs, Poison Messages, Backpressure

The front door must stay open under bad input and overload:

• Poison message: a record that can't be processed (malformed, fails schema, triggers a bug). Without handling, it blocks its partition forever — the consumer retries, fails, retries, and every record behind it starves. One bad message takes down a partition.
• Dead-letter queue / quarantine (the lab's ingest): route the failing record to a DLQ topic with the reason attached, and move on. The pipeline keeps flowing; the bad data is preserved for diagnosis; the DLQ rate becomes an alert (a step change = upstream broke something — you find out in minutes, not at model-metric time). After fixing the cause, redrive the DLQ back into the main flow.
• Schema validation at ingress (P03): the most common DLQ trigger; validate against the contract at the door so bad data never enters the lake.
• Backpressure & quotas (P01 Ch. 8): when consumers can't keep up, lag grows — that's the signal. Kafka offers client quotas (bytes/s per principal) to stop one noisy producer from starving others; on overload you slow producers or shed load on purpose, never grow an unbounded buffer.

Chapter 10: Replay, Reprocessing, and Legacy Flume Migration

• Replay is just seek to an older offset and re-consume — possible because the log is retained (or compacted). It's how you backfill a new consumer, recover from a processing bug, or re-derive a corrupted dataset. Replay is only correct if processing is event-time + idempotent (P01): otherwise the replay double-counts or lands in the wrong windows. The test of a well-built pipeline is literally "can you replay it and get the same answer."
• Flume migration (a JD requirement): Flume is the legacy Hadoop-era agent (source → channel → sink, e.g. spooldir/tail → memory/file channel → HDFS sink). It's push-based, has weak ordering/replay, and couples producers to HDFS. The migration pattern: (1) stand up Kafka/Kinesis alongside; (2) dual-write — keep Flume running while a new producer also writes to Kafka; (3) build the new consumer (Flink/Spark) reading Kafka and validate parity against the Flume-fed HDFS output; (4) cut over consumers to the Kafka path; (5) decommission Flume once parity holds for a soak period. The principles are the same as any migration (P15): dual-write, validate, cut over, keep rollback.

Lab Walkthrough Guidance

Lab 01 — Mini Broker, suggested order:

1. fnv1a_32 + partition_for_key, then Partition.append/end_offset/read with absolute offsets — verify offsets are monotonic and stable.
2. retain_count / retain_time — confirm start_offset advances but end_offset doesn't (Ch. 2/6).
3. compacted_view — latest-per-key, tombstone deletes (Ch. 6).
4. assign_partitions (range, then roundrobin) and re-run with a new member = rebalance (Ch. 5); confirm extra consumers go idle.
5. ConsumerGroup position/poll/commit_through/seek/lag — then the replay test: seek(0,0) makes lag jump back (Ch. 10).
6. ingest with a DLQ — poison messages quarantined with a reason; good records after a poison one still land (Ch. 9).

Success Criteria

You are ready for Phase 03 when you can, from memory:

1. Draw Kafka's anatomy (broker/topic/partition/replica/leader/ISR/controller).
2. Explain acks=all + min.insync.replicas durability as quorum math, and what unclean leader election trades.
3. Explain idempotent producers and Kafka transactions, and exactly where the guarantee stops.
4. Describe consumer-group rebalancing (eager vs cooperative) and why frequent rebalances hurt.
5. Contrast delete-retention and compaction and pick the right one for events vs state.
6. Size Kinesis shards (and Kafka partitions) for a stated throughput.
7. Choose Kafka vs Kinesis vs Pulsar for a stated context, with the deciding dial.
8. Explain the DLQ pattern and the Flume→Kafka migration steps.

Interview Q&A

Q: A single bad message is stalling a partition and lag is climbing. Walk me through it. That's a poison message: the consumer fails on it, retries, and starves everything behind it in that partition. Immediate mitigation: route the failing record to a DLQ with the error attached and advance past it, so the pipeline flows and the bad data is preserved. Permanent fix: validate at ingress against the schema contract (so it never enters), alert on DLQ rate (a step change is an upstream regression), and add a redrive to reprocess the DLQ once fixed. The anti-pattern is infinite in-place retry, which converts one bad record into a partition outage.

Q: Producers are creating hot partitions. Diagnose and fix. The partition key is too low-cardinality or skewed (Zipfian "whale" keys), so the hash piles many keys onto few partitions — P01's skew factor, now in Kafka. Confirm by per-partition byte/record rate. Fixes, in order of preference: choose a higher-cardinality key that still preserves the ordering you need; if a single real key is hot (one giant account), add a key + bucket salt and reconcile downstream; only then consider more partitions (doesn't help a single hot key). The thing not to do is raise hardware — skew is a layout problem.

Q: When would you choose a compacted topic? When consumers need the current state per entity, not the event history: a config topic, a CDC snapshot of a table, a feature/profile keyed by user, or offsets themselves. Compaction keeps the latest value per key (tombstones delete), so a new consumer can bootstrap the full current state by reading the topic once — a changelog that doubles as a table. For genuine event streams (clicks, payments) you want delete-retention because the history is the data.

Q: "We're on Kinesis, we get exactly-once." True? No more than Kafka does. Kinesis gives ordered, at-least-once delivery per shard with replay; exactly-once is still an effect you engineer — KCL checkpoints can replay a batch after a worker failure, so your processing must be idempotent (dedup by sequence number, or idempotent upserts at the sink). The guarantee, as always, is re-earned at the external boundary, not granted by the transport.

References

• Kafka: The Definitive Guide (Shapira, Palino, Sivaram, Petty) — the operational bible
• Kafka docs — Design (delivery, ISR, transactions) and KRaft
• Kreps, The Log (2013); Kleppmann, DDIA Ch. 11 (stream processing)
• Amazon Kinesis Data Streams developer guide (shards, resharding, KCL, enhanced fan-out)
• Apache Pulsar concepts (segments, tiered storage, multi-tenancy, geo-replication)
• Confluent: Exactly-Once Semantics in Apache Kafka (the transactions design)

🛸 Hitchhiker's Guide — Phase 02: Event Ingestion & Messaging

Read this if: you want Kafka and Kinesis in your head fast — the anatomy, the durability knobs, the numbers, the gotchas — before the WARMUP derives them. Skim, then read the WARMUP, then build the broker.

0. The 30-second mental model

Kafka/Kinesis = a partitioned, replicated, retained log (P01's log, made real). You produce keyed records (key → partition → per-key order), they're replicated for durability (ISR + acks), consumed by groups that track offsets (so they can replay), and they age out by retention or fold to a table by compaction. The front door must survive bad input (DLQ) and overload (backpressure). One sentence: get ordering, durability, and schema right here — nothing downstream can fix ingestion.

1. Kafka anatomy in one diagram

Producer ──key→hash→ Partition (leader) ──replicate→ Followers (ISR)
                         │                               ▲
                  append (offset N)               acks=all waits for
                         │                          min.insync.replicas
Consumer GROUP ──assigned partitions──> poll(from committed offset) ──commit(next)
                         │
        retention(delete)  OR  compaction(latest-per-key + tombstones)
Controller (KRaft quorum) coordinates leaders & metadata

2. The durability knobs (memorise this combo)

RF=3  +  acks=all  +  min.insync.replicas=2
   → survive 1 broker loss, never lose an ACKed write   (P01: 2 + 2 > 3)
unclean.leader.election = false   → consistency over availability (turn ON only if you
                                    can tolerate data loss)
acks=1  → fast, loses data if leader dies pre-replication
acks=0  → fire & forget, may lose

3. The numbers

4. Producer truths

• Key = ordering + skew. Low-cardinality key → hot partition (P01 skew, again). Fix: higher-cardinality key or key+bucket salt.
• Idempotent producer (default now): PID + per-partition sequence → broker drops retried dups. Exactly-once append, not app-level dedup.
• Transactions: atomic multi-partition write + offset commit → exactly-once inside Kafka (read_committed). Stops at the external sink.

5. Consumer-group truths

• Each partition → exactly one consumer in the group.
• Rebalance = reassign on join/leave/death. Eager = stop-the-world; cooperative-sticky = move only what must move (use it).
• Frequent rebalances = a slow consumer blowing max.poll.interval.ms. Symptom: lag sawtooth + throughput drop.
• Commit after processing = at-least-once (dedup downstream). Auto-commit = footgun.

6. Retention vs compaction (pick by intent)

events (clicks, payments)   → cleanup.policy=delete   (history IS the data)
state (config, CDC, profiles)→ cleanup.policy=compact  (latest-per-key + tombstone delete)

A compacted topic = a changelog that doubles as a table (KTable, __consumer_offsets).

7. Kafka vs Kinesis vs Pulsar (the one-liners)

• Kinesis: lowest ops, AWS-native, shard-quota'd, AWS lock-in. "We're all-in on AWS, optimize ops."
• Kafka/MSK: max control, richest ecosystem (Connect/Streams), portable. "We need throughput control / portability / the ecosystem."
• Pulsar: broker/storage split → elastic storage, built-in multi-tenancy + geo. "We need elastic decoupled storage and tenants."

8. Ingestion safety in three moves

1. Validate at ingress (P03 schema) → bad data never enters the lake.
2. DLQ with reason → poison messages quarantined, pipeline flows, DLQ rate = alert.
3. Backpressure / quotas → slow producers or shed load on purpose; never unbounded buffer.

9. Beginner mistakes that mark you

1. acks=1 (or default-without-checking) for data you can't lose.
2. Infinite in-place retry on a poison message → partition outage.
3. Over-partitioning "for scale" → rebalance + latency + handle costs.
4. Low-cardinality partition key → hot partition; then blaming Kafka.
5. "Kinesis/Kafka gives exactly-once" → only inside; sink re-earns it.
6. Replaying a processing-time pipeline → double counts (need event-time + idempotency).
7. Auto-commit + crash → silent duplicates or loss.

10. War stories

• "Lag is sawtoothing." → rebalance storm; a slow consumer keeps getting kicked. Tune max.poll.records/max.poll.interval.ms or speed the handler; switch to cooperative-sticky.
• "Replay melted the brokers." → big backfill read cold (off page cache) → disk I/O storm. Throttle the replay / use a read replica / tiered storage.
• "One country's events vanished into one partition." → country as the key. Re-key.
• "We can't reprocess last month." → retention was 7 days, no lake copy. The log wasn't kept; this is why you tee raw events to S3 (P08) on day one.

11. How this phase pays off later

• DLQ + validate → schema registry & contracts (P03).
• Replay + compaction → Flink reprocessing (P04), CDC/KTable (P05), serving snapshots (P11).
• Partition/shard sizing → the capstone's global ingestion platform (P16).
• acks/ISR quorum → Cassandra consistency (P11), multi-region (P14/P15).

Read the WARMUP, build the broker, then P03 puts a schema contract on the front door.

👨🏻 Brother Talk — Phase 02, Off the Record

Kafka and Kinesis, the stuff the docs won't tell you because the docs are written by people who want you to succeed and aren't on your pager.

Brother, ingestion is where I've seen the most careers quietly dented, and it's because of one brutal property: you cannot fix ingestion downstream. If you lose an event at the front door, it's gone — no Flink job, no Spark backfill, no clever SQL resurrects it. If you let a garbage schema through, it poisons every table that reads from it. So the stakes here are higher than they look, and the work is less glamorous than Flink or the lakehouse. Do it right anyway. The people who skip the boring durability config are the people who later write the postmortem titled "How We Lost Six Hours of Payments."

Let me give you the hard-won stuff.

Config the durability knobs on day one, in writing. RF=3, acks=all, min.insync.replicas=2, unclean.leader.election=false. Memorise that line like your phone number. The default acks in some clients has changed over the years, and "we used the default" is not a sentence you want in an incident review. And unclean.leader.election — that innocent-looking boolean — is literally the CAP theorem as a config flag. If it's true on a topic that matters, you've signed up to silently lose acknowledged data when the ISR empties. I've seen it default-on bite a team hard. Check it. Write down why it's set the way it is. That's a 10-minute ADR that saves a 6-hour outage.

The poison message will come for you. Not might. Will. Some upstream team will ship a malformed payload, or a null where you expected a value, or an emoji in a field that your deserializer chokes on, and if your consumer does the naive thing — try, fail, retry forever — that one record wedges the whole partition and everything behind it starves. I've watched lag climb into the millions because of one bad record nobody could see. The fix is a mindset: every consumer assumes its input is hostile. Validate, and on failure, shove it in the DLQ with the reason attached and keep moving. Then — and this is the part people forget — alert on the DLQ rate, because a sudden spike in quarantined records is the earliest, cheapest signal that an upstream producer broke their contract. You find out in minutes from a clean alert instead of in days from a confused executive asking why a dashboard looks wrong.

Respect the partition key like it's load-bearing, because it is. Choosing the key feels like a throwaway decision and it's actually one of the most consequential things you'll do. It decides your ordering guarantee (all of one key's events stay ordered) and your skew (pick a low-cardinality key like country or event_type and you've built yourself a hot partition that no amount of hardware fixes). I once inherited a pipeline keyed by event_type with about eight types — eight hot partitions doing all the work while the others idled. The fix wasn't more brokers; it was re-keying. When someone proposes a partition key, immediately ask: "how many distinct values, and is the distribution skewed?" That one question marks you as someone who's been burned.

"Exactly-once" will be on a vendor slide in a meeting you're in. Be ready. Someone will say "we don't need to worry about duplicates, Kafka/Kinesis gives us exactly-once." The junior move is to nod. The senior move is to ask, gently, "exactly-once into where?" Because the guarantee is real inside Kafka (transactions, read_committed) and evaporates the moment you write to S3 or Cassandra or a feature store — which is, you know, the entire point of the pipeline. You re-earn it at the sink with idempotency. Saying this calmly in a meeting, without making anyone feel dumb, is exactly the kind of thing that gets you promoted, because you just prevented a class of bug the whole room was about to wave away.

Keep a raw copy. Always. The single most valuable insurance policy in data engineering is teeing your raw events to cheap storage (S3) the instant they arrive, before any processing, with long retention. Kafka retention is finite (often 7 days), and the day you have a bug in your processing logic that's been running for two weeks, the only thing that saves you is being able to replay from a raw copy that goes back far enough. "We can't reproduce last month's data" is the most demoralizing sentence in this job, and it's almost always caused by someone trusting the stream's retention as their source of truth. The stream is transport. S3 is memory. Don't confuse them.

Now the career note. Ingestion is unsexy, and that is exactly why owning it is a power move. Everyone wants to build the cool Flink job; nobody wants to own the schema-validation gateway and the DLQ redrive tooling and the partition-key standards. So if you become the person who owns the front door — the one who can say "no event is lost, every bad payload is quarantined and visible, and we can replay anything" — you become infrastructure in the org's mind. You're not "the person who built feature X," you're "the person the platform depends on." That's a much harder position to be laid off from, and a much easier one to be promoted from. Boring, load-bearing, and owned by you. That's the play.

Build the broker by hand. When you make the replay test work — seek back to zero and watch the lag jump and the events flow again — you'll feel why replay is the platform's superpower. That feeling is worth more than reading about it ten times.

Next stop, P03: we put a real contract on this front door, because an unvalidated topic is just an outage with a delay timer.

— your brother 👨🏻

Lab 01 — Mini Broker (a faithful partitioned-log)

Phase: 02 — Event Ingestion & Messaging | Difficulty: ⭐⭐⭐⭐☆ | Time: 6–8 hours

You don't understand Kafka until you've built one. This lab implements the Kafka mental model in ~200 lines of pure Python: a partitioned, offset-addressed log with keyed partitioning, absolute monotonic offsets that survive retention, consumer-group offset tracking + replay, partition assignment + rebalancing, log compaction (the changelog→table duality), consumer lag, and a dead-letter-queue path that quarantines poison messages without blocking the pipeline.

What you build

• partition_for_key — stable hashing → per-key ordering (P01 carried forward)
• Partition — append-only log with absolute offsets; retain_count / retain_time advance start_offset (records age out) but offsets never renumber
• compacted_view — fold a keyed log into the latest-value-per-key table; tombstones (value None) delete — the basis of KTables and CDC snapshots
• assign_partitions — range and roundrobin assignors; re-running with a different membership is a rebalance
• ConsumerGroup — committed offsets, poll, commit_through, seek (replay), lag
• ingest — validate-at-ingress; valid → main log, invalid → DLQ with the reason attached so the pipeline keeps flowing

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 15 tests pass.
• You can explain why test_retention_by_count_advances_start_offset asserts end_offset == 10 even after dropping 6 records (offsets never renumber).
• You can explain why seek(0, 0) makes lag jump back up — and why that's a feature.
• You can state what the consumer of a compacted topic gets vs a regular topic.

Extensions

• Add replication: each partition has RF replicas + an ISR; produce(acks="all") only returns once min.insync.replicas have it; kill a replica and watch the ISR shrink (P01 quorum math made concrete).
• Add idempotent producer semantics (producer id + sequence number) so a retried produce doesn't duplicate (P01 exactly-once carried forward).
• Add a DLQ replay/redrive: after fixing the validator, re-ingest the DLQ into main and assert the poison records now land.
• Add sticky assignment and measure how many partitions move on rebalance vs range (the Kafka CooperativeStickyAssignor motivation).

Lab 02 — Partitioned-Log Ingestion in Scala

Phase: 02 — Event Ingestion & Messaging | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

The Scala twin of Lab 01's Python mini-broker — the ingestion substrate as a JVM library. Stable partitioning (per-key ordering), monotonic offsets, consumer-group offset tracking

• replay, retention, log compaction, and a DLQ ingest gateway. Verified with ScalaTest.

What you build

• Partitioner — 32-bit FNV-1a stable hash → partition (a key always maps to the same partition → per-key ordering survives restarts, unlike a salted runtime hash)
• PartitionedLog — append (monotonic offsets), read-from-offset, retain (retention), compact (keep latest per key)
• ConsumerGroup — poll / commit / seek (replay)
• IngestGateway — validate → append, or route to the DLQ (poison-message safety)

Run

sbt test

Expected: 7 tests, all green in IngestionSpec — including per-key ordering, replay via seek, retention, compaction, and DLQ routing.

Key concepts (Scala expression)

Success criteria

• sbt test → all 7 specs pass.
• You can explain why a stable hash (not the JVM's hashCode across processes) is required for per-key ordering.
• You can explain compaction vs retention and when each applies.

Extensions

• Add consumer rebalancing: N consumers share P partitions; reassign on join/leave.
• Add idempotent producer semantics (dedup by a producer-supplied sequence number).
• Add exactly-once commit: commit the consumer offset and the downstream write atomically (the 2-phase-commit idea — P04).

Phase 03 — Serialization, Schema & Data Contracts

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (28–36 hours) Prerequisites: Phase 02 (the topics these schemas govern)

Why This Phase Exists

The JD dedicates an entire core responsibility to this: "Own schema governance across event streams, lakehouse tables, and analytical datasets … manage Protobuf, Avro, and JSON schema evolution … build compatibility checks … integrate schema registry workflows into CI/CD … prevent breaking changes in event contracts." And the deliverables include a "data contract system integrated into CI/CD" and a "schema governance platform."

Here's the truth that makes this phase matter: the format on the wire is a contract, and an unversioned contract is an outage waiting for a deploy. When a producer and a consumer disagree about the bytes, you get silent corruption, deserialization storms, or a poisoned table — and the producer and consumer are always on different versions, because they're owned by different teams who deploy at different times. Schema governance is how you make that disagreement impossible to ship.

Concepts

• Why encoding matters: text (JSON/CSV) vs binary (Avro/Protobuf); size, speed, type-safety, and self-description tradeoffs; why analytics wants columnar (Parquet/ORC, P08) but ingestion wants row-oriented schema-carrying formats.
• Protobuf deeply: field numbers as wire identity, varint/zig-zag/length-delimited wire types, optional/repeated/oneof/map, unknown-field preservation, packed encoding, and the evolution rules (never reuse/repurpose a number; add, don't mutate).
• Avro deeply: schema-travels-with-(or-near)-the-data, reader vs writer schema resolution, defaults, unions/nullability, and why Avro + a registry is the classic Kafka pairing.
• JSON Schema: ubiquity and self-description vs cost and weak typing; when "just JSON" is fine and when it's a liability.
• Compatibility: backward (new reader ↔ old data), forward (old reader ↔ new data), full, transitive variants; which one to pick based on deploy order.
• Schema registry design: subjects, versions, IDs, the magic-byte wire envelope, and the registry as the source of truth.
• Data contracts: schema + semantics + SLA + ownership + PII classification; contract testing; producer/consumer compatibility matrices; safe deprecation.
• CI/CD integration: the registry compatibility check as a pre-merge gate (the lab).

Labs

Lab 01 — Schema Registry & Data-Contract Engine (flagship, implemented)

→ Lab folder: lab-01-schema-registry/

Lab 02 — Schema Compatibility & Contracts in Scala (implemented, sbt test)

→ Lab folder: lab-02-scala-contracts/

Extension Project A — Real Protobuf round-trip (spec)

Define a .proto, generate code, serialize/deserialize, then evolve it (add a field, change a number — observe the break) and verify unknown-field preservation across versions.

Extension Project B — Contract test in CI (spec)

Wire buf breaking (Protobuf) or the Confluent registry's compatibility check into a GitHub Action that fails a PR introducing an incompatible change. Produce the producer/consumer compatibility matrix as a build artifact.

Integrated-Scenario Hooks

• This phase's validation is the ingress check the P02 DLQ routes failures from.
• Its PII classification field-tags feed the governance platform (P14).
• Its compatibility matrix is the centerpiece of the Hive/Flume → modern migration (P15).
• Its Validated is the real Cats abstraction you implement in Scala (P12).

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• The wire format is a contract; version it or ship outages on a delay timer.
• Backward vs forward is decided by who deploys first, not by taste.
• Protobuf identity is the number; Avro leans on names + defaults + reader/writer resolution.
• Enforce contracts in CI, accumulate all violations, and tag PII at the schema.

Deliverables Checklist

• Lab 01 implemented; all 17 tests pass
• You can state the add/remove/type-change rules for backward, forward, and full
• You can explain Protobuf number reuse danger and unknown-field preservation
• Extension A round-trip or Extension B CI gate

Warmup Guide — Serialization, Schema & Data Contracts

Zero-to-principal primer for Phase 03: how bytes carry meaning (Protobuf, Avro, JSON), why field numbers and reader/writer resolution exist, the precise compatibility rules, the schema registry, and how a data contract becomes a CI gate that makes breaking changes un-shippable.

Table of Contents

• Chapter 1: Encoding — Turning Structure into Bytes
• Chapter 2: Protobuf in Working Detail
• Chapter 3: Avro and Reader/Writer Schema Resolution
• Chapter 4: JSON, JSON Schema, and When Text Is Fine
• Chapter 5: Compatibility — Backward, Forward, Full
• Chapter 6: The Schema Registry
• Chapter 7: Data Contracts — Beyond the Schema
• Chapter 8: Validation, Accumulation, and Contract Testing in CI
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Encoding — Turning Structure into Bytes

A record in memory is a graph of typed values; the wire is a flat sequence of bytes. Encoding is the rule that maps between them, and the choice has consequences:

• Text formats (JSON, CSV): self-describing (field names travel with every record), human-readable, ubiquitous — but large (field names repeated on every record), slow to parse, and weakly typed (is "42" a number? is the date a string?). Great for config and debugging; expensive at billions of events/day.
• Binary, schema-on-write (Avro, Protobuf): compact, fast, strongly typed — but you need the schema to read the bytes. This is the right default for high-volume ingestion.
• Columnar (Parquet, ORC — P08): a storage layout for analytics (read few columns over many rows), not an ingestion wire format. Different job.

The principal framing: ingestion wants row-oriented schema-carrying binary (Avro/Protobuf); analytics wants columnar (Parquet); JSON is the convenient default you grow out of. Picking the wrong one shows up as cost (JSON at scale) or pain (CSV's no-types nightmare).

Chapter 2: Protobuf in Working Detail

Protocol Buffers is the JD's preferred event payload, and its design is its evolution story. A message is a set of fields, each with a number (the tag), a type, and a name:

message Order {
  string order_id = 1;       // the "1" is the wire identity, not the name
  double amount   = 2;
  optional string currency = 3;
  repeated string tags = 4;
}

The wire format: each field is encoded as (field_number << 3 | wire_type) then the value. Wire types: varint (int32/64, bool, enum — 7 bits/byte, smaller numbers = fewer bytes), zig-zag (sint, so negatives don't cost 10 bytes), 64-bit/32-bit fixed, and length-delimited (strings, bytes, embedded messages, packed repeated). Consequences a principal knows:

• The number is everything. Renaming a field is free (names aren't on the wire); changing or reusing a number re-interprets old bytes as the wrong field — silent corruption. The cardinal rule: never reuse or repurpose a field number; reserve retired ones.
• Adding a field is safe if you give it a new number; old readers don't know it and skip it.
• Unknown-field preservation: a Protobuf reader keeps fields it doesn't recognise and can re-serialize them intact — so an intermediary on an old schema doesn't drop data added by a newer producer. This is huge for multi-hop pipelines.
• optional vs implicit (proto3): proto3 scalars have no "was it set?" by default (a missing int reads as 0, indistinguishable from an explicit 0); optional brings back presence tracking. repeated defaults to empty (never "missing"). oneof models mutually exclusive fields. map is sugar for repeated key/value.
• Enums: add symbols safely (with a known default at 0); removing a symbol or changing a number is breaking. Always reserve a 0 = UNKNOWN.

The lab models the number-as-identity rule (protobuf_violations): a number whose type changed is flagged because it breaks every byte ever written with it.

Chapter 3: Avro and Reader/Writer Schema Resolution

Avro takes a different tack: the data is encoded against a writer schema, and the reader supplies its own reader schema; Avro resolves the two at read time:

• Fields present in the writer but not the reader → ignored. Fields in the reader but not the writer → filled from the reader's default (so a reader field must have a default to be added safely). This resolution is exactly why Avro compatibility is defaults-driven.
• Avro is fully self-describing if the schema travels with the file (object-container files embed it) — but in Kafka the schema is too big to ship per-record, so you ship a tiny schema ID and resolve it via a registry (Ch. 6).
• Unions (["null", "string"]) model nullability; the order matters (the first is the default branch).

Avro vs Protobuf, the honest comparison: Avro's reader/writer resolution + defaults are elegant for data files and the Kafka+registry world; Protobuf's number-based identity + codegen + unknown-field preservation are excellent for RPC and multi-language services and multi-hop event pipelines. Both are correct choices; the JD names Protobuf as primary, so know it cold and know Avro well.

Chapter 4: JSON, JSON Schema, and When Text Is Fine

JSON's superpower is that everything speaks it and a human can read it. JSON Schema adds a validation contract (types, required, ranges, patterns, enums). It's the right call when: volume is modest, consumers are heterogeneous/external, debuggability matters more than bytes, or you're prototyping. It's a liability when: you're at billions of events/day (the repeated keys and parse cost dominate), you need strong cross-language typing, or you need compact binary. Principals don't sneer at JSON — they cost it out and switch to Avro/Protobuf when the numbers say so, keeping a JSON Schema contract either way.

Chapter 5: Compatibility — Backward, Forward, Full

The single most important operational idea in this phase, because producers and consumers are always on different versions. Definitions (precise, the way the lab implements them):

• Backward compatible: a reader on the new schema can read data written with the old schema. → Consumers can upgrade first. You may add fields with defaults and remove fields; you may not add a required (no-default) field (old data lacks it) or change a field's type.
• Forward compatible: a reader on the old schema can read data written with the new schema. → Producers can upgrade first. You may add fields (old readers ignore them) and remove optional fields; you may not remove a required field (old reader needs it) or change a type.
• Full: both. → Only add or remove optional fields, never change types. The safest, and what you want for a contract many independent teams share.
• Transitive variants check against all prior versions, not just the latest — stricter, prevents "compatible with N but not N−2" drift.

The decision rule: pick the mode from your deploy order. If consumers roll out before producers, you need backward; if producers roll first, forward; if you can't coordinate (the usual case at scale), full. Note the pleasing symmetry: remove is backward-safe/forward-dangerous; add-required is forward-safe/backward-dangerous. The lab's tests pin exactly this.

Chapter 6: The Schema Registry

A registry is a service that makes schemas first-class:

• Subjects & versions: schemas are registered under a subject (often <topic>-value); each registration is a new version with a global ID.
• The wire envelope: instead of shipping the schema per message, the producer ships a magic byte + the 4-byte schema ID, then the payload; the consumer fetches & caches the schema by ID. Tiny overhead, full resolution.
• The compatibility gate: when you register a new version, the registry checks it against the subject's compatibility mode and rejects a breaking change. This is the lab's register() — and wired into CI, it's how "no breaking schema change reaches production without compatibility checks" (a JD SLO) becomes literally true.
• Confluent Schema Registry, AWS Glue Schema Registry, and Buf Schema Registry are the market implementations; the concept is what you must own.

Chapter 7: Data Contracts — Beyond the Schema

A schema says what shape; a data contract says everything a consumer needs to depend on the producer safely:

• Schema (types, fields) + compatibility policy.
• Semantics: what the fields mean — units, ranges, invariants (an amount is in minor currency units and is non-negative; a status is one of these enum values). These are the semantic rules the lab's Validated enforces beyond mere types.
• SLA / freshness: how often, how late, how complete (handoff to P13's SLOs).
• Ownership: a named team, on a pager, accountable for the contract.
• PII classification: each field tagged by sensitivity (none/pii/sensitive) to drive masking, access, and retention downstream (handoff to P14).
• Lifecycle / deprecation: how a field or version is announced, deprecated, and removed over versions — never yanked.

The shift: a data product (P00) is a dataset with a contract. Contracts turn "that table broke and nobody knew who owned it" into "the contract test failed in the producer's CI before merge."

Chapter 8: Validation, Accumulation, and Contract Testing in CI

Two distinct enforcement points:

• Schema-evolution checks (build time): does the new schema break the old one? → the registry/buf breaking gate (Ch. 5–6). Catches contract regressions before merge.
• Record validation (runtime, at ingress — P02): does this record satisfy the schema + semantics? Failures → DLQ. Here the accumulation property matters: a fail-fast validator (Either/exceptions) reports the first error and stops, so the producer fixes one thing, re-runs, finds the next — a slow loop. An accumulating validator (Cats Validated, the lab's) reports all violations at once, so the producer fixes everything in one pass. For data contracts, accumulation is the correct shape, and it's why P12 builds Validated for real.

Contract testing closes the loop: the consumer publishes the contract it depends on; the producer's CI runs the consumer's expectations against the producer's output (Pact-style), so a breaking change fails the producer's build — shifting the failure left from prod to PR.

Lab Walkthrough Guidance

Lab 01 — Schema Registry & Contract Engine, suggested order:

1. backward_violations — add-no-default breaks; remove is fine; type change breaks.
2. forward_violations — remove-required breaks; add is fine; type change breaks (note the symmetry with backward).
3. compatibility_violations — dispatch on mode; full = both.
4. protobuf_violations — by number; type change on a shared number is wire-breaking.
5. Validated.combine + _type_ok (bool is not int!) + validate_record — accumulate all errors; unknown fields allowed.
6. SchemaRegistry.register — the gate: compatible → version bump; breaking → raise and don't store.

Success Criteria

You are ready for Phase 04 when you can, from memory:

1. Contrast Protobuf, Avro, and JSON and say which you'd pick for ingestion vs analytics vs external APIs.
2. State the Protobuf number rule and what unknown-field preservation buys you.
3. Explain Avro reader/writer resolution and why defaults drive its compatibility.
4. Give the add/remove/type-change rules for backward, forward, and full — and pick a mode from a deploy order.
5. Describe the registry's wire envelope and the compatibility gate.
6. List the parts of a data contract beyond the schema (semantics, SLA, ownership, PII).
7. Explain why a contract validator should accumulate errors.

Interview Q&A

Q: You must add a mandatory tenant_id to an event consumed by 30 teams. How? Adding a required field is backward-incompatible — old data lacks it, so new consumers can't read history and the registry (rightly) rejects it under backward/full. The safe path: add it as optional with a default (or a sentinel like UNKNOWN), deploy producers to start populating it, let consumers adopt it at their pace, monitor coverage, and only later — if ever — tighten validation to "must be present going forward" via a new major version or a runtime contract rule rather than a schema-required flag. The principal point: "mandatory" is a rollout, not a flag flip, when 30 teams and historical data are involved.

Q: A producer renamed a Protobuf field and consumers broke. Why, if names aren't on the wire? They didn't just rename — almost certainly they reused or shifted a number, or regenerated code such that a number now maps to a different field/type. On the wire Protobuf is keyed by number; a rename alone is invisible and safe, but if currency (number 3) became region (number 3) with a different type, every old byte for "3" now deserializes as the wrong thing. The fix is the cardinal rule: never reuse a number — reserve the old one and give the new field a fresh number.

Q: Backward vs forward — how do you choose? By deploy order. If I roll consumers out before producers (so new readers must handle old data), I need backward. If producers roll first (old readers must tolerate new data), forward. At scale I usually can't coordinate the order across many teams, so I default to full (only add/remove optional fields, never change types) and use transitive mode to stop drift against older versions. The mnemonic: remove is backward-safe but forward-risky; add-required is forward-safe but backward-risky.

Q: Why should a data-contract validator accumulate errors instead of failing fast? Because the consumer of the result is a producer fixing their data, and the humane, fast loop is "here are all 6 things wrong, fix them and resubmit" — not "fix one, resubmit, find the next, repeat 6 times." Fail-fast (Either/exceptions) is right for dependent steps where step 2 is meaningless if step 1 failed; validation steps are independent, so the applicative Validated that accumulates is the correct abstraction (I build it in Scala in P12).

References

• Kleppmann, DDIA Ch. 4 (Encoding and Evolution) — the canonical treatment
• Protocol Buffers language guide and encoding
• Apache Avro specification — schema resolution rules
• Confluent Schema Registry — compatibility types
• Buf — breaking change detection (Protobuf CI)
• Data Contracts — Andrew Jones, and the data-mesh literature on contracts & ownership

🛸 Hitchhiker's Guide — Phase 03: Serialization, Schema & Data Contracts

Read this if: you want Protobuf/Avro/JSON and the compatibility rules in your head fast. Skim, then read the WARMUP, then build the registry.

0. The 30-second mental model

The bytes on the wire are a contract between a producer and a consumer who are always on different versions. Schema governance makes a breaking change un-shippable: you encode in Protobuf/Avro (compact, typed), register the schema in a registry, and a compatibility gate rejects changes that would break the other side. One sentence: version the contract, pick the compatibility mode from your deploy order, and enforce it in CI.

1. Format cheat sheet

2. Protobuf in 6 bullets

• The number is the wire identity; the name is not. Rename freely; never reuse or change a number's type → silent corruption.
• Add fields with new numbers → old readers skip them (safe).
• Unknown-field preservation: old readers keep & re-emit fields they don't know → safe multi-hop.
• proto3 scalars have no presence (missing == 0); use optional for presence; repeated defaults empty.
• Enums: reserve 0 = UNKNOWN; add symbols safely; don't remove/renumber.
• Wire types: varint (small ints cheap), zig-zag (sint for negatives), length-delimited (strings/messages/packed).

3. Avro in 4 bullets

• Data is written with a writer schema, read with a reader schema; Avro resolves them.
• Reader field not in writer → needs a default (so adds are safe only with defaults).
• Writer field not in reader → ignored.
• In Kafka you ship a schema ID (magic byte + 4 bytes), not the whole schema.

4. Compatibility — the symmetry to memorise

backward  = new READER reads OLD data   → consumers upgrade first
            ✅ add-with-default, remove        ❌ add-required, change type
forward   = old READER reads NEW data   → producers upgrade first
            ✅ add, remove-optional            ❌ remove-required, change type
full      = both → only add/remove OPTIONAL fields; never change types

Mnemonic: remove is backward-safe / forward-risky; add-required is forward-safe / backward-risky. Pick the mode from who deploys first (can't coordinate → full).

5. Registry = the CI gate

producer ships:  [magic byte][4-byte schema ID][payload]
consumer:        fetch+cache schema by ID → resolve
register(new):   check vs latest under mode → REJECT if breaking  ← the gate

Wired into CI, this makes "no breaking schema change reaches prod" literally true.

6. A data contract is more than a schema

schema (types) + compatibility policy
+ semantics (units, ranges, invariants)   ← the Validated semantic rules
+ SLA / freshness                          → P13 SLOs
+ ownership (a team on a pager)
+ PII classification (per field)           → P14 masking/access/retention
+ deprecation lifecycle (announce, deprecate, remove — never yank)

7. Validate by ACCUMULATING

Fail-fast (Either/exceptions) → "fix one error, resubmit, find the next." Accumulating (Validated) → "here are all 6, fix them at once." For contracts, accumulate. (You build Validated for real in Scala in P12.)

8. Beginner mistakes that mark you

1. Reusing/renumbering a Protobuf field → silent corruption.
2. Adding a required field and calling it backward-compatible.
3. Thinking "rename broke it" when it was a number change.
4. Shipping JSON at billions/day without costing it (key repetition + parse).
5. No registry / no CI gate → breaking changes discovered in prod.
6. A schema with no owner, no SLA, no PII tags → not a contract, just a shape.
7. Fail-fast validation that makes producers fix errors one at a time.

9. War stories

• "Half the events deserialize as garbage after a deploy." → a field number got reused. Reserve retired numbers; add buf breaking to CI.
• "New consumer can't read last year's data." → someone added a required field (backward break) and backfilled nothing. Use optional-with-default.
• "A PII column leaked into an open table." → no PII tags on the schema; governance had nothing to enforce on (P14).

10. How this phase pays off later

• Validation = the P02 DLQ's reason-for-rejection.
• Validated = real Cats code in P12.
• PII tags = the input to P14 governance.
• Compatibility matrix = the heart of the P15 migration ADR.

Read the WARMUP, build the registry gate, then P04: Flink turns these validated events into stateful, exactly-once results.

👨🏻 Brother Talk — Phase 03, Off the Record

Schemas and contracts — the least glamorous phase, the one that quietly prevents more 2 a.m. pages than any other. Let me tell you why to care.

Brother, I'm going to be honest: schema governance sounds like the most boring possible topic. "Compatibility modes." "Field numbers." Your eyes want to glaze. Mine did too. And then I lived through the alternative, and now I evangelize this stuff like a convert, because unversioned schemas are the single most common cause of self-inflicted data outages I've ever seen, and they're 100% preventable.

Here's the scene that converted me. A producer team — good engineers, well-meaning — renamed a field and shipped it on a Tuesday afternoon. They tested it. It worked for them. What they didn't know is that under the hood a field number got shuffled, and forty downstream consumers — analytics jobs, ML features, a fraud model, a finance report — started silently deserializing garbage. Not crashing. Silently wrong. The fraud model degraded. The finance numbers drifted. It took days to trace it back, because the failure was nowhere near the cause. That entire week of pain, across a dozen teams, would have been a single red X on a pull request if a compatibility check had been wired into CI. One red X versus a week of cross-team archaeology. That's the whole pitch for this phase.

So let me give you the stuff that matters.

"It worked when I tested it" is the most dangerous sentence in data engineering. A producer tests against the current consumer, in the current deploy, with fresh data. But the real world has consumers on six different versions, historical data written months ago, and replay jobs reading last year's bytes. The schema contract is the only thing that reasons about all of those at once. When you internalize that producer and consumer are never on the same version — that they literally cannot be, at any real org — the whole idea of compatibility modes stops being academic and becomes obviously necessary.

Make the breaking change impossible to merge, not merely discouraged. Here's the thing about humans: if a rule is a wiki page, it gets violated. If a rule is a failing CI check, it gets followed. The entire value of a schema registry isn't the storage — it's that register() rejects incompatible changes, and when you wire that into the producer's CI, breaking the contract becomes literally un-shippable. You're not asking people to be careful. You're making carelessness impossible. That's the principal move: don't write a guideline, build a guardrail.

Field numbers are sacred. Treat retiring one like retiring a jersey. In Protobuf, the number is the identity. When you remove a field, reserve its number so nobody ever reuses it. I cannot tell you how many corruptions trace to "oh, number 7 was free, I'll use it" when number 7 was actually a retired field still present in old data and replay streams. Reserve it. Comment it. Never reuse it. This one discipline prevents a whole category of silent corruption.

Tag PII at the schema, today, even if nobody's asked. This is the unglamorous favor you do your future self. When you define an event, mark which fields are PII right there in the contract. Why now, when there's no governance system yet to consume the tags? Because P14's governance platform — masking, access control, GDPR deletion — is infinitely easier to build when the PII is already labeled at the source than when you have to go back and classify ten thousand fields across a thousand schemas under a compliance deadline. The tag costs you ten seconds at definition time and saves a future team a quarter of misery. Do it.

Accumulate your errors, because the human on the other end is a person fixing data. When you validate a record and it has six problems, tell them all six. The fail-fast pattern — report one error, stop — is fine for a compiler but cruel for a data producer, who then fixes one thing, resubmits, finds the next, and slowly loses their mind. The Validated pattern in this lab (and for real in Scala in P12) exists because someone realized validation is about helping a human fix everything at once. Small thing. Huge difference in how much people hate or love your platform.

Now the career angle, because it's real. The person who owns schema governance becomes the person who prevents the org's most embarrassing outages, and that's a reputation worth having. It's not flashy — nobody throws a party because a breaking change got caught in CI instead of in prod — but the senior people notice. They notice that since you built the gate, the "mystery data corruption" incidents dropped to near zero. They notice that onboarding a new consumer is now safe because contracts are explicit. Quiet competence in the load-bearing, unglamorous places is exactly what gets you trusted with the architecture decisions. The flashy stuff gets you applause; the boring guardrails get you authority.

Build the registry gate. Make a breaking change fail the test. Feel how good it is to know — not hope — that the bad change can't ship. Then come to P04, where we take these clean, validated, contract-checked events and do the genuinely hard thing: stateful, exactly-once, event-time stream processing with Flink.

— your brother 👨🏻

Lab 01 — Schema Registry & Data-Contract Engine

Phase: 03 — Serialization, Schema & Data Contracts | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

"A topic without a schema is an outage on a delay timer." This lab builds the machinery that keeps a thousand producers from breaking a thousand consumers: compatibility checking (backward / forward / full), Protobuf field-number rules, an error-accumulating Validated validator, and a registry that rejects breaking changes on register() — i.e. the CI gate that the JD requires.

What you build

• Schema / Field — fields with Protobuf-style numbers and Avro-style names/defaults/required (so you learn both evolution models in one model)
• backward_violations / forward_violations / compatibility_violations — the exact rules: add-with-default is safe; add-required breaks backward; remove-required breaks forward; type change breaks both
• protobuf_violations — wire identity is the number; a number whose type changed corrupts existing bytes
• Validated — applicative, error-accumulating validation (collect every contract violation in one pass, not fail-fast) — the Cats Validated you build for real in P12
• SchemaRegistry — versioned schemas + a compatibility mode; register is the pre-merge CI check

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 17 tests pass.
• You can explain why "remove a field" is backward-compatible but forward-incompatible.
• You can explain why Validated accumulates instead of failing on the first error, and why that's the right shape for a data contract.
• You can state which compatibility mode you'd set for an event topic where consumers deploy before producers (forward) vs after (backward).

Extensions

• Add transitive compatibility (check the new schema against all prior versions, not just the latest) — the stricter BACKWARD_TRANSITIVE mode.
• Add enum evolution rules (adding a symbol is forward-safe with a default; removing one isn't) and nullable vs optional distinctions.
• Generate a producer/consumer compatibility matrix (which writer versions which reader versions can read) — the artifact you put in a migration ADR (P15).
• Wire it as a real pre-commit hook: fail the build if a .proto/.avsc change is incompatible with the registered latest.

Lab 02 — Schema Compatibility & Data Contracts in Scala

Phase: 03 — Serialization, Schema & Data Contracts | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

The Scala twin of Lab 01's Python registry — the schema-governance logic a JVM platform team ships as a library. Compatibility checking (backward/forward/full + Protobuf field-number rules) plus Cats Validated record validation that accumulates every violation in one pass. Verified with ScalaTest via sbt test.

What you build

• Schema / Field — a schema model with Protobuf field numbers and the hasDefault rule that makes add/remove safety mechanical
• Compatibility — backwardViolations, forwardViolations, compatibilityViolations (none/backward/forward/full), and protobufViolations (reused tag + changed type = silent byte corruption)
• Validation.validateRecord — Cats ValidatedNel validation accumulating missing-required, type-mismatch, and semantic-rule failures (a producer fixes them all at once)

Run

sbt test

First run downloads Scala 2.13 + Cats + ScalaTest, then compiles and runs. Expected: 11 tests, all green across CompatibilitySpec and ValidationSpec.

Key concepts (Scala expression)

Success criteria

• sbt test → all 11 specs pass.
• You can explain why a renamed field with the same number+type is Protobuf-safe but a retyped reused number is catastrophic.
• You can explain why validateRecord uses Validated (accumulate) not Either (fail-fast).

Extensions

• Add enum evolution rules (adding a symbol is backward-safe; removing isn't).
• Add a CI gate function: assertCompatible(old, new, mode) that throws with all violations — the pre-merge check (P03 WARMUP Ch. 7).
• Replace typ: String with a sealed FieldType ADT for exhaustive, typo-proof matching (P12's make-illegal-states-unrepresentable).

Phase 04 — Apache Flink

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Prerequisites: Phase 01 (time, exactly-once), Phase 02 (the source), Phase 03 (validated events)

Why This Phase Exists

Flink is the JD's reference engine for "low-latency stream processing … stateful operators … checkpointing … savepoints … exactly-once … RocksDB state backend … rescaling." It is also the subject of an entire interview round (Round 2: "a Flink job with rising checkpoint duration, increasing RocksDB state size, delayed watermarks, and growing Kafka lag — diagnose it"). If you can reason about Flink's execution and state model, you can reason about every stream processor; the others (P05) are variations.

This is the hardest phase in the streaming half, because correct stateful streaming is genuinely hard: you must get event time, out-of-order arrival, state, fault tolerance, and exactly-once output all correct simultaneously. The lab makes you implement the core, so the words "checkpoint" and "exactly-once sink" stop being slogans and become code you've debugged.

Concepts

• Execution model: job graph → operators → task slots & parallelism; the streaming dataflow; keyBy → keyed streams; chaining; the JobManager/TaskManager split.
• Time: event time vs processing time vs ingestion time; watermarks (how generated, per-partition watermarks and the min-across-inputs rule, idle sources); allowed lateness; side outputs.
• Windows: tumbling, sliding, session (merging), global; window assigners, triggers, evictors; the state cost of windows.
• State: keyed state vs operator state; ValueState/ListState/MapState; state backends (heap vs RocksDB, incremental checkpoints); state TTL; why state size drives everything.
• Fault tolerance: the Chandy-Lamport distributed snapshot; barriers & alignment (and unaligned checkpoints); checkpoints (engine-owned) vs savepoints (user-owned, portable); exactly-once vs at-least-once checkpointing.
• Exactly-once sinks: idempotent sinks vs two-phase commit (pre_commit on barrier, commit on completion); transactional Kafka/file sinks; the Iceberg/Delta sink (P09).
• Rescaling: key groups as the redistribution unit; max parallelism; restoring a savepoint at a new parallelism.
• Backpressure (P01): credit-based flow control; the symptoms — rising checkpoint duration, alignment time, and lag — and their causes.
• Flink SQL / Table API: declarative streaming; dynamic tables; the streaming-batch unification.

Labs

Lab 01 — Flink-Grade Stream Processor (flagship, implemented)

→ Lab folder: lab-01-flink-engine/

Lab 02 — Event-Time Stream Processing in Scala (implemented, sbt test)

→ Lab folder: lab-02-scala-windows/

Extension Project A — Run real Flink (spec)

docker Flink cluster; a DataStream job consuming the P02 Kafka topic; keyed windows; a RocksDB state backend; take a savepoint, change the job, restore from the savepoint; observe the Flink UI's backpressure, watermark, and checkpoint metrics.

Extension Project B — Stateful fraud detector (spec; → capstone)

The JD's Problem 2: event-time sliding + session windows, broadcast rules state, Cassandra enrichment (P11), RocksDB, S3 checkpoints, savepoint-based rule upgrades, replay for validation. Sketch the topology and the state/keying plan.

Integrated-Scenario Hooks

• This engine consumes P02's validated topic and writes P09's Iceberg tables via the 2PC sink.
• Its checkpoints live in S3 (P08); its state backend is RocksDB (an LSM, P08/P11).
• Its enrichment reads Cassandra (P11); its rules arrive as broadcast state.
• It is the processing core of capstone Problems 1 and 2 (P16).

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• Watermarks turn "when is a window done?" into an explicit, tunable answer.
• A checkpoint is a consistent global snapshot; restore resumes with exact state — that's exactly-once state.
• Exactly-once output needs idempotent or two-phase-commit sinks; the engine alone can't.
• State size is the cost center; RocksDB, TTL, and key design exist to manage it.
• Rescaling works because key groups (not keys) are the redistribution unit.

Deliverables Checklist

• Lab 01 implemented; all 15 tests pass
• You can explain checkpoint barriers, alignment, and what stalls them
• You can explain the two-phase-commit sink protocol end to end
• You can diagnose the Round-2 scenario (rising checkpoint duration + state + lag)
• Extension A savepoint round-trip or Extension B topology sketch

Warmup Guide — Apache Flink

Zero-to-principal primer for Phase 04: Flink's execution and state model, event time and watermarks, the distributed-snapshot checkpoint algorithm, savepoints and rescaling, exactly-once sinks, and how to read the failure signatures interviewers love (rising checkpoint duration + growing state + delayed watermarks + lag).

Table of Contents

• Chapter 1: Flink's Execution Model
• Chapter 2: Event Time and Watermarks, Precisely
• Chapter 3: Windows and Triggers
• Chapter 4: State and State Backends
• Chapter 5: Checkpoints — The Distributed Snapshot
• Chapter 6: Savepoints, Upgrades, and Rescaling
• Chapter 7: Exactly-Once Sinks
• Chapter 8: Backpressure and the Failure Signatures
• Chapter 9: Flink SQL and the Table API
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Flink's Execution Model

Flink runs a streaming dataflow: a directed graph of operators (sources → transforms → sinks) processing records one at a time (not micro-batches — that's Spark, P05). The runtime pieces:

• JobManager (the coordinator): builds the execution graph, schedules tasks, triggers checkpoints, coordinates recovery. TaskManagers (the workers): run operator subtasks in task slots (a slot = a unit of resource isolation; one slot can run a pipeline of chained operators).
• Parallelism: each operator has N parallel subtasks; records are distributed across them. keyBy(k) partitions the stream by key (hash) so all records for a key go to the same subtask — keyed streams are where keyed state lives, and the per-key ordering of P01 applies.
• Operator chaining: Flink fuses adjacent operators (e.g. map→filter) into one task to avoid serialization/handoff — a real performance lever and a thing to know when reading the job graph.
• Sources/sinks: connectors (Kafka, Kinesis, files, Iceberg). Modern sources implement a unified interface with split enumeration and per-split watermarks (Ch. 2).

Chapter 2: Event Time and Watermarks, Precisely

P01 established why event time (wall clocks lie). Flink's implementation:

• Each event carries a timestamp (extracted at the source). A WatermarkStrategy generates watermarks — typically boundedOutOfOrderness(d): W = max_ts_seen − d. The watermark is a record-like marker flowing with the stream that asserts "no event with ts ≤ W is still coming."
• Per-partition watermarks + the min rule: each source partition tracks its own watermark; an operator with multiple inputs takes the minimum across them (the slowest input gates progress). This is why a single idle partition can stall watermarks org-wide — Flink offers withIdleness(...) to let idle partitions be ignored. A classic prod incident: "watermarks stopped advancing" → one partition went silent.
• Allowed lateness: after a window fires at the watermark, keep it for allowedLateness to absorb stragglers (re-emitting corrections); beyond that, route to a side output (OutputTag) — late data is information, never silently dropped.
• The triangle (P01): larger d = more complete first results but higher latency and more retained state; you measure the real out-of-order distribution and choose d against it.

Chapter 3: Windows and Triggers

Windows turn an unbounded stream into bounded computations:

• Tumbling (fixed, non-overlapping), sliding/hopping (fixed size, overlapping — one event in size/slide windows, multiplying state), session (gap-based, dynamic boundaries that merge when a late event bridges two sessions — the trickiest), and global (one window, you supply the trigger).
• Assigner → trigger → (evictor) → function: the assigner places events in windows; the trigger decides when to fire (default: on watermark past window end; can fire early on processing time or counts); the function aggregates (reduce/aggregate are incremental and cheap; process buffers all elements and is expensive).
• State cost: open windows × keys × accumulator size. This is the number behind "our RocksDB state is growing" — usually too many open windows (huge allowed lateness, sliding windows, or unbounded key cardinality). The lab keeps an explicit lifecycle (open→closeable→expired) for exactly this reason.

Chapter 4: State and State Backends

State is what makes streaming powerful and dangerous:

• Keyed state (scoped to the current key): ValueState, ListState, MapState, ReducingState, AggregatingState. Operator state (scoped to a subtask): used by sources/sinks (e.g. Kafka offsets). Broadcast state: a low-throughput stream (rules, config) replicated to every subtask — the fraud-rules pattern.
• State backends: HashMapStateBackend (state on the JVM heap — fast, but bounded by memory, full snapshots) vs EmbeddedRocksDBStateBackend (state in an embedded LSM tree on local disk — supports state far larger than memory, incremental checkpoints, at the cost of serialization + disk I/O per access). The choice is "does my state fit in heap?" — large keyed state ⇒ RocksDB.
• RocksDB specifics that show up on-call: it's an LSM (P08/P11) so writes go to memtables then SST files with background compaction; state grows when compaction can't keep up or when TTL isn't set; read amplification from many SST levels. State TTL expires old entries so state doesn't grow unbounded (the bounded-dedup idea of P01, as a feature).
• The lab's snapshot()/restore() is a state backend's job: serialize all keyed state to a durable structure and rebuild it.

Chapter 5: Checkpoints — The Distributed Snapshot

How do you snapshot a distributed, running dataflow consistently, without stopping it? The Chandy-Lamport asynchronous barrier snapshotting algorithm — Flink's crown jewel:

1. The JobManager injects a checkpoint barrier (numbered N) into the source streams.
2. Barriers flow with the data. When an operator receives barrier N on all its inputs (alignment), it snapshots its state (asynchronously, to durable storage — S3) and forwards barrier N downstream.
3. When all operators (including sinks) confirm checkpoint N, the JobManager marks it complete. On failure, the whole job restarts from the last complete checkpoint — every operator restores its state and sources rewind to the offsets in that checkpoint.

Key subtleties:

• Alignment is where exactly-once is bought and where it can hurt: an operator must wait for the barrier on every input, buffering the faster inputs. Under skew or backpressure, alignment time balloons → rising checkpoint duration (the Round-2 sym ptom). Unaligned checkpoints trade some of this by snapshotting in-flight data, helping under backpressure.
• Exactly-once vs at-least-once checkpointing: at-least-once skips alignment (lower latency) but can double-count on recovery. Exactly-once aligns.
• Because sources rewind to checkpointed offsets and operators restore exact state, the effect is exactly-once within the Flink job — output exactly-once still needs Ch. 7.
• The lab's checkpoint-recovery test (restore-and-continue == one run) is this property in miniature.

Chapter 6: Savepoints, Upgrades, and Rescaling

• Checkpoint vs savepoint: same mechanism, different ownership. Checkpoints are automatic, engine-owned, optimized for fast recovery, and may be cleaned up. Savepoints are manual, user-owned, portable, self-contained snapshots you take on purpose to upgrade code, change parallelism, migrate clusters, or A/B a new version. The deploy pattern: take a savepoint → stop the job → start the new job from the savepoint.
• Stateful upgrades require operator UIDs (so Flink can map saved state to operators across code changes) and state-schema compatibility (evolving the state's types — P03 ideas applied to state). Forget a UID and your state won't restore.
• Rescaling: changing parallelism means redistributing keyed state. Flink solves this with key groups: each key maps to one of maxParallelism key groups (fixed at job creation); key groups are assigned to subtasks in contiguous ranges; rescaling moves whole key-group ranges between subtasks. The lab implements key_group and task_for_key_group exactly. The catch interviewers probe: maxParallelism is fixed forever at first run — set it generously, because you can't exceed it later.

Chapter 7: Exactly-Once Sinks

Checkpoints give exactly-once state; the output to an external system needs more, because a checkpoint restore replays the records since the last checkpoint — so the sink must not re-emit them. Two patterns:

• Idempotent sink: writes are naturally repeatable — upsert by key, set-union, overwrite-by-id. Replaying produces the same final state. Simplest; works when the sink supports it (Cassandra upsert, a keyed KV store).
• Two-phase commit (Flink's TwoPhaseCommitSinkFunction): on each checkpoint barrier, pre-commit the buffered output as a pending transaction (durable, invisible); on the checkpoint-complete notification, commit it (visible). On recovery, re-commit pending transactions (they were part of the checkpoint) — idempotently. This is how transactional Kafka sinks, the file sink ("pending" → "finished" file rename), and the Iceberg/Delta sink (P09: stage data files, commit the snapshot) achieve exactly-once. The lab implements this protocol, including the crucial property: buffered-but-not-yet- pre-committed data is dropped on crash — and that's correct, because the source offset for it wasn't checkpointed either, so it'll be re-read.

Chapter 8: Backpressure and the Failure Signatures

Flink uses credit-based flow control: a downstream subtask grants the upstream credits (buffer space); when it's slow, credits dry up and the upstream slows — backpressure propagates cleanly to the source (which slows its Kafka consumption → lag). The Round-2 signature and how the symptoms connect:

slow operator / sink  →  backpressure  →  buffers fill  →  checkpoint barriers move slowly
                                                         →  alignment time ↑  →  checkpoint DURATION ↑
                                                         →  source slows  →  Kafka LAG ↑
late/out-of-order data + huge allowed lateness/sliding   →  many open windows  →  STATE size ↑
no state TTL / unbounded key cardinality                 →  RocksDB grows  →  STATE size ↑, GC ↑

The diagnostic discipline: find the backpressured operator (Flink UI colors it), then ask why it's slow — a slow sink (external system, the usual culprit), data skew on a hot key (P01), an expensive process function buffering elements, or under-parallelism. The fix matches the cause: scale the sink / async I/O, re-key or salt the hot key, switch to incremental aggregation, add state TTL, or raise parallelism (via savepoint rescale, Ch. 6). Saying that chain out loud is passing Round 2.

Chapter 9: Flink SQL and the Table API

Not everything needs the DataStream API. Flink SQL treats streams as dynamic tables (a changelog) and lets you write windowed aggregations, joins, and pattern matching (MATCH_ RECOGNIZE) declaratively — with the same event-time/watermark/state semantics underneath. The streaming-batch unification: the same SQL can run over a bounded source (batch) or unbounded (streaming). Use SQL for the 80% of standard transforms (it's less code and optimizable); drop to DataStream for custom state, timers, and complex logic. Know that the state/exactly-once story is identical — SQL doesn't escape watermarks or checkpoints, it sits on them.

Lab Walkthrough Guidance

Lab 01 — Flink-Grade Stream Processor, suggested order:

1. assign_window + KeyedWindowProcessor.process/flush (watermark, lateness, side output, dedup) — get the order-independence and lateness tests green first.
2. snapshot()/restore() — then the headline test: process half, snapshot, restore into a fresh operator, process the rest → identical to one run (checkpoint recovery).
3. key_group / task_for_key_group — deterministic, contiguous, covers all tasks (rescaling).
4. TwoPhaseCommitSink — invisible until commit; crash before commit → recover exactly-once; double commit idempotent; uncheckpointed buffer dropped.

Success Criteria

You are ready for Phase 05 when you can, from memory:

1. Draw Flink's execution model (JobManager/TaskManager/slots/parallelism/keyBy/chaining).
2. Explain watermark generation, the min-across-inputs rule, and the idle-partition stall.
3. Explain the barrier-snapshot algorithm and why alignment buys exactly-once and can stall.
4. Contrast checkpoints and savepoints and describe a stateful upgrade (UIDs!).
5. Explain key groups and why maxParallelism is fixed at job creation.
6. Explain idempotent vs two-phase-commit sinks and where each fits.
7. Walk the Round-2 failure chain (backpressure → checkpoint duration + state + lag) and give the cause-matched fix.

Interview Q&A

Q (Round 2): A Flink job has rising checkpoint duration, growing RocksDB state, delayed watermarks, and growing Kafka lag. Diagnose. These are one story, not four bugs. Something downstream is slow (most often the sink, or a hot-key skew, or an expensive process function). That backpressures upstream: buffers fill, so checkpoint barriers move slowly and alignment time climbs → checkpoint duration ↑; the source is throttled → Kafka lag ↑. Meanwhile delayed watermarks point at event-time/idle-partition issues or the same backpressure delaying barrier/marker flow, and growing RocksDB state points at too many open windows (huge allowed lateness / sliding windows / unbounded keys) or missing state TTL. I'd find the backpressured operator in the UI, confirm whether it's the sink or a skewed key, then fix the cause: async/scaled sink, re-key/salt the hot key, incremental aggregation, add state TTL, or rescale via savepoint — and consider unaligned checkpoints to relieve the alignment stall while I fix the root cause.

Q: Checkpoints vs savepoints — when do you use each? Checkpoints are the engine's automatic safety net for failure recovery — frequent, fast, engine-owned, possibly cleaned up. Savepoints are my tool for planned change — upgrade the job's code, change parallelism, migrate clusters, fork for A/B — taken on purpose, self-contained, portable. The upgrade ritual: savepoint → stop → start new version from savepoint, which only works if operators have stable UIDs and the state schema is compatible.

Q: How does Flink rescale stateful jobs, and what's the gotcha? Keyed state is partitioned by key group (hash(key) % maxParallelism), and key groups are assigned to subtasks in contiguous ranges, so changing parallelism just moves whole key-group ranges — no per-key reshuffle needed. The gotcha is that maxParallelism is fixed at job creation and caps your future parallelism; set it generously up front (changing it later requires a state migration), because you can scale up to it but never beyond it.

Q: You need exactly-once into S3/Iceberg. How? Checkpoints give exactly-once state inside Flink, but the S3 write needs a transactional sink: two-phase commit. On each checkpoint barrier the sink pre-commits — for files, it writes data to a temp/pending location; for Iceberg, it stages data files. On checkpoint-complete it commits — rename pending files to final, or commit the Iceberg snapshot atomically. On recovery it re-commits pending transactions idempotently. Crucially, data buffered since the last checkpoint is dropped on a crash, which is correct because the source offset wasn't advanced past it either — so it's re-read and re-processed, netting exactly-once.

References

• Stream Processing with Apache Flink (Hueske & Kalavri) — the book
• Akidau et al., The Dataflow Model (VLDB 2015) & Streaming Systems — windows/watermarks
• Carbone et al., Lightweight Asynchronous Snapshots for Distributed Dataflows (the Flink checkpoint paper); Chandy & Lamport (1985) — the original snapshot algorithm
• Flink docs — concepts: time, state, fault tolerance
• Flink docs — TwoPhaseCommitSinkFunction / exactly-once sinks
• SeniorMLEngineer track Phase 02 WARMUP — the windows/watermarks primer this phase deepens

🛸 Hitchhiker's Guide — Phase 04: Apache Flink

Read this if: you want Flink's model — execution, time, state, checkpoints, exactly-once — in your head fast, and to be ready for the "diagnose this Flink job" interview round. Skim, read the WARMUP, build the engine.

0. The 30-second mental model

Flink processes records one at a time over a graph of operators, keeps keyed state, orders by event time using watermarks, snapshots the whole running job consistently with barrier checkpoints (so a crash resumes with exact state), and gets exactly-once output via idempotent or two-phase-commit sinks. One sentence: Flink = stateful event-time streaming with consistent snapshots; correctness is checkpoints + the right sink.

1. Execution model in one diagram

JobManager (coordinates, triggers checkpoints)
  └─ TaskManagers → task slots → operator subtasks
Source → keyBy(k) → window/process (KEYED STATE) → sink
         hash-partition      checkpoint barriers flow with data
parallelism = N subtasks/operator; chaining fuses adjacent ops

2. Watermarks (the thing everything hangs on)

W = max_event_time_seen − bounded_delay
window fires when  W ≥ window.end
multi-input op watermark = MIN across inputs  ← one idle partition stalls everything
                                                (fix: withIdleness)
late-but-allowed → re-emit correction; too-late → side output (never drop)

Trade: bigger delay = more complete first results, more latency, more state.

3. State & backends

State grows from: too many open windows (sliding / huge allowed lateness), unbounded key cardinality, no state TTL. State size is the cost center.

4. Checkpoints = consistent distributed snapshot

JobManager injects BARRIER N into sources
→ each operator, on receiving N on ALL inputs (ALIGNMENT), snapshots state async → S3
→ all confirm → checkpoint N COMPLETE
crash → restart from last complete checkpoint: restore state + rewind source offsets

Alignment buys exactly-once and is where it stalls under backpressure → unaligned checkpoints help.

5. Checkpoint vs savepoint

checkpoint = automatic, engine-owned, fast recovery, may be GC'd
savepoint  = manual, user-owned, portable → upgrades, rescale, migrate
upgrade ritual: savepoint → stop → start new version from savepoint (need stable operator UIDs)

6. Rescaling via key groups

key → key_group = hash(key) % maxParallelism   (fixed at job creation!)
key_group → task = kg * parallelism // maxParallelism   (contiguous ranges)
rescale = move whole key-group ranges between subtasks
GOTCHA: maxParallelism caps you forever — set it generously

7. Exactly-once OUTPUT (state alone isn't enough)

idempotent sink     → upsert/set-union/overwrite-by-id (Cassandra, KV)   ← simplest
two-phase commit    → pre_commit on barrier (pending, invisible)
                      commit on checkpoint-complete (visible, idempotent)
                      recover → re-commit pending  (files rename / Iceberg snapshot commit)
buffered-not-precommitted data is DROPPED on crash → correct (offset not advanced either)

8. The Round-2 failure chain (memorize this)

slow sink / hot-key skew / expensive process()
   → backpressure → buffers fill
       → checkpoint barriers slow + alignment ↑ → CHECKPOINT DURATION ↑
       → source throttled → KAFKA LAG ↑
huge allowed-lateness / sliding / no TTL / unbounded keys → many open windows → STATE ↑

Fix the CAUSE: async/scale the sink, re-key/salt the hot key, incremental aggregation, state TTL, rescale; unaligned checkpoints relieve alignment while you fix root cause.

9. Beginner mistakes that mark you

1. Ordering/aggregating by processing time → wrong on replay (P01 again).
2. No withIdleness → one quiet partition freezes all watermarks.
3. process() that buffers all elements instead of incremental aggregate.
4. No state TTL + unbounded keys → RocksDB grows forever.
5. Forgetting operator UIDs → savepoint won't restore after a code change.
6. Setting maxParallelism too low (or defaulting it) → can't scale up later.
7. Assuming checkpoints give exactly-once output (they give exactly-once state).

10. How this phase pays off later

• 2PC sink → P09 Iceberg/Delta exactly-once writes.
• Checkpoints in S3 / RocksDB LSM → P08 storage, P11 LSM internals.
• Broadcast state + Cassandra enrichment → P11 + capstone fraud detector (P16).
• The failure chain → P13 observability & incident response.

Read the WARMUP, build the engine, then P05: the other streaming engines (Kafka Streams, Spark SS, Akka/FS2) as variations — plus CDC and streaming joins.

👨🏻 Brother Talk — Phase 04, Off the Record

Flink is the phase that separates the people who talk about streaming from the people who've actually run it at 3 a.m. Let me get you to the second group.

Brother, this is the big one. Flink is where a lot of data engineers hit a wall, and it's not because the API is hard — it's because stateful, fault-tolerant, exactly-once, event-time streaming requires you to hold five hard things in your head at once, and if any one of them slips, you get silent wrongness. So let me give you the mental scaffolding that kept me sane.

First: respect that streaming is harder than batch, and stop fighting it. In batch, you have all the data, you process it, you're done — if something's wrong you just re-run. In streaming, the data never stops, it arrives out of order, some of it is late, your job will crash mid-flight, and you still have to be correct and not lose or duplicate anything. That's genuinely a harder problem, and Flink's apparent complexity is the irreducible complexity of that problem, not accidental complexity someone could have designed away. Once I accepted that — "Flink is complicated because the problem is complicated" — I stopped resenting the watermarks and checkpoints and started appreciating them as the minimum machinery that makes the impossible possible.

Second: watermarks are a confidence statement, and the idle-partition bug will get you. A watermark says "I'm now confident nothing older than this is coming." Beautiful. But here's the trap that's bitten me and everyone I know: Flink takes the minimum watermark across all input partitions, so if one partition goes quiet — a low-traffic region, a sensor that stopped, a test topic with no data — its watermark freezes, the minimum freezes, and your entire job's windows stop firing. Output just... stops. And you stare at it for an hour before you realize one idle partition is holding the whole thing hostage. Learn withIdleness now, and when someone says "the Flink job stopped emitting but isn't crashing," your first thought should be "which partition went quiet?" That instinct alone is worth this whole phase.

Third: state is where the bodies are buried. Every production Flink horror story I know is ultimately a state story. State grew unbounded because nobody set a TTL. State exploded because someone used sliding windows with a huge allowed-lateness and suddenly there were millions of open windows. State got hot because one key (the "whale" from P01 — it's always P01) carried 80% of the traffic onto one subtask. RocksDB compaction couldn't keep up and the job slowed to a crawl. When you design a Flink job, the question that matters most is not "what's the logic" — it's "how big does the state get, and what bounds it?" If you can't answer that, you haven't designed it yet, you've prototyped it.

Fourth: "exactly-once" has a specific, narrow meaning, and the slide deck lies about it (again). Flink genuinely gives you exactly-once state — the checkpoint algorithm is real and beautiful, restore resumes with exact state. But the output to your database or S3? That's exactly-once only if your sink is idempotent or does two-phase commit, because on recovery Flink replays everything since the last checkpoint. I've seen teams assume "we're on Flink, so exactly-once" and then wonder why their database has duplicates after every job restart. The answer is always: the sink wasn't transactional or idempotent. Build the two-phase-commit sink in this lab and feel the protocol — pre-commit on the barrier, commit on completion, recover re-commits — and you'll never make that assumption again.

Fifth, and this is the interview gold: the Round-2 question — "checkpoint duration is rising, state is growing, watermarks are delayed, lag is climbing" — looks like four problems and it's one. It's almost always backpressure from a slow sink, cascading into all four symptoms. The junior tries to fix four things. The principal says "these are one story" and traces the backpressure to its source. When you can stand at a whiteboard and draw the chain — slow sink → backpressure → buffers fill → barriers slow → checkpoint duration up + source throttled → lag up — calmly, while everyone else is spooked by the four red graphs, you have just demonstrated exactly the "escalation point for the hardest failures" the JD is hiring for. That's not memorization; it's understanding the causal model. Build the lab and the model becomes yours.

Now, the encouragement, because Flink can feel demoralizing. You will be confused the first time you try to reason about barrier alignment. Everyone is. The concepts are genuinely deep — the checkpoint algorithm is a published academic paper (Chandy-Lamport, then the Flink team's async version). You're not slow; you're learning load-bearing distributed-systems theory that most "senior" engineers wave their hands at. The payoff is enormous: Flink expertise is rare and valuable precisely because it's hard, and the person who can actually debug a stateful streaming job — not just write one in the happy path, but debug it when state is exploding and checkpoints are timing out — is the person who gets the principal title and the pager that comes with it.

Build the engine. Make the checkpoint-recovery test pass — process half a stream, snapshot, restore into a fresh operator, finish, and watch it match the uninterrupted run exactly. That moment, when you see exact state survive a simulated crash, is when Flink stops being scary and starts being yours.

Next stop, P05: the other streaming engines, so you can say why Flink and not Kafka Streams or Spark Structured Streaming — and handle CDC and streaming joins.

— your brother 👨🏻

Lab 01 — Flink-Grade Stream Processor

Phase: 04 — Apache Flink | Difficulty: ⭐⭐⭐⭐⭐ | Time: 8–10 hours

This is the flagship of the whole streaming half of the track. You build the core of Flink's event-time machinery by hand — windows, watermarks, allowed lateness, keyed state — and then the two things that make Flink Flink: checkpoint/restore (so a crash resumes with exact state) and a two-phase-commit sink (so output is exactly-once). Plus key-group rescaling, so you can change parallelism safely.

What you build

• KeyedWindowProcessor — tumbling event-time sum/count per key with watermark-driven emission, allowed-lateness re-emission, and a late-event side output (never dropped)
• snapshot() / restore() — capture ALL operator state into a portable structure and rebuild from it: checkpoint recovery. The test proves restore-and-continue == one uninterrupted run.
• key_group / task_for_key_group — Flink's mechanism: keys → fixed key groups → contiguous task ranges, so rescaling redistributes whole key-group ranges
• TwoPhaseCommitSink — write → pre_commit(barrier) → commit(complete); nothing is visible until its checkpoint completes, a crash mid-way recovers exactly-once, and a double commit is idempotent

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 15 tests pass — test_checkpoint_restore_continues_identically and test_crash_between_precommit_and_commit_recovers_exactly_once are the ones that certify you understand Flink's correctness model.
• You can explain why output is invisible until commit, and what happens to buffered data on a crash (dropped — and why that's still exactly-once).
• You can explain why key groups (not keys) are the rescaling unit and why that fixes maxParallelism at job creation.

Extensions

• Add sliding and session windows (session needs merge-on-late-bridge — genuinely hard; do it to feel why state is the cost center).
• Make seen_ids bounded by a TTL behind the watermark (real dedup state can't grow forever — P01's bounded-dedup tradeoff, now in a real operator).
• Add broadcast state: a second "rules" stream that every key sees (the fraud-rules pattern from the JD's Problem 2).
• Add a RocksDB-style spilling state backend behind an interface and measure the heap-vs-disk tradeoff — the abstraction real Flink exposes.

Lab 02 — Event-Time Stream Processing in Scala

Phase: 04 — Apache Flink | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–6 hours

Stream processing on the JVM is the in-demand Scala data-engineering skill — Flink, Kafka Streams, and Spark are Scala/JVM systems, and their operators are written in this style. This lab implements Flink's event-time core in idiomatic Scala (case classes, immutable accumulators, a keyed processor), verified with ScalaTest via sbt test. It mirrors Lab 01's Python flink-engine so you can compare the language to the semantics.

What you build

• Windowing.assignWindow — tumbling [start, end) arithmetic (boundary → next window)
• StreamProcessor — per-key event-time aggregation with watermark-driven emission, allowed-lateness corrected re-emission, late-event side output, and dedup for exactly-once effect
• Windowing.finalResults — fold emissions+updates to the consumer's latest-per-window view

Run

sbt test

First run downloads Scala 2.13 + ScalaTest, then compiles and runs. Expected: 6 tests, all green, including the property that certifies the design — order independence under bounded shuffle (shuffled delivery == in-order delivery).

Key concepts (same as Lab 01, in Scala)

Success criteria

• sbt test → all 6 specs pass.
• You can read the Scala and map every line to the Flink concept (WARMUP).
• You can explain why the shuffle test is the one that proves correctness.

Extensions

• Make the processor purely functional: thread state through an immutable fold (no var/ mutable), returning (newState, emitted) — the FS2/Cats way (P12).
• Add sliding and session windows (session needs merge-on-late-bridge).
• Bound the seen dedup set by a TTL behind the watermark (P01's bounded-dedup tradeoff).
• Wire it to real Flink: drop this logic into a KeyedProcessFunction[String, Event, ...] with ValueState/timers (the production form — see WARMUP Ch. 4).

Phase 05 — Streaming Alternatives & CDC

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 04 (Flink — the reference engine these are compared against)

Why This Phase Exists

Flink is not the only stream processor, and a principal must be able to say why Flink and not Kafka Streams here, or Spark Structured Streaming there, or Akka Streams for that service. The JD lists them all — "Kafka Streams, Flink DataStream API, Flink SQL, Spark Structured Streaming, Akka Streams" — and names CDC platforms, streaming joins, enrichment, and sessionization as required patterns. This phase rounds out the streaming half: the engine landscape, the FP/JVM streaming libraries (Akka Streams, FS2 — foreshadowing P12), and the join/CDC patterns that every engine implements and that you build here once, engine-agnostically.

Concepts

• Kafka Streams: a library, not a cluster — embeds in your app; KStream vs KTable (the stream/table duality, P02 compaction made API); local state in RocksDB + changelog topics for fault tolerance; interactive queries; when "no cluster to run" wins.
• Spark Structured Streaming: micro-batch (and continuous) on the Spark SQL engine; checkpointing to a durable store; the same DataFrame API as batch (unification); higher latency than Flink but trivial if you're already on Spark.
• Akka Streams: Reactive-Streams back-pressured graphs on the actor runtime; typed, composable, in-process — the right tool for streaming inside a service (P12).
• FS2 (Functional Streams for Scala): pure, back-pressured streaming on Cats Effect — resource-safe, referentially transparent (P12).
• The engine decision matrix: latency, state size, exactly-once needs, ops model, language, ecosystem — Flink vs Kafka Streams vs Spark SS vs Akka/FS2.
• CDC: turning a DB's binlog/WAL into events (Debezium); snapshot + streaming bootstrap; applying CDC to a table idempotently (the lab); the OLTP→lakehouse bridge.
• Join patterns: stream↔table (temporal / as-of), stream↔stream (interval/windowed), enrichment from an external store (P11); the leakage and state-size traps.
• Sessionization and other stateful patterns (dedup, top-N, running aggregates).

Labs

Lab 01 — Streaming Joins & CDC Apply (flagship, implemented)

→ Lab folder: lab-01-joins-cdc/

Extension Project A — Engine bake-off memo (spec)

One page: pick a real use case (clickstream sessionization; CDC replication; in-service enrichment) and choose Flink vs Kafka Streams vs Spark SS vs Akka/FS2 — with the deciding dial (P00) and an ADR.

Extension Project B — Debezium CDC pipeline (spec)

docker Postgres + Debezium + Kafka; capture row changes; materialize a table with your apply_cdc; demonstrate snapshot→streaming handover and idempotent re-apply.

Integrated-Scenario Hooks

• This phase's CDC apply feeds the lakehouse upsert tables (P09) and Cassandra serving (P11).
• Its as-of join is the enrichment step of the capstone fraud detector (P16, JD Problem 2).
• Its Akka/FS2 preview is built for real in Scala in P12.

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• The engines differ in latency, ops model, and language — not in the patterns they must implement.
• Kafka Streams = a library (no cluster); Flink = lowest latency + richest state; Spark SS = free if you're on Spark; Akka/FS2 = in-service streaming.
• CDC materialization is idempotent-by-LSN; the stream/table duality is everywhere.
• Joins are where leakage (as-of) and unbounded state (intervals) bite — bound both.

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass
• You can choose an engine for a stated use case with the deciding dial
• You can explain KStream vs KTable and the changelog-topic fault-tolerance model
• Extension A memo or Extension B CDC pipeline

Warmup Guide — Streaming Alternatives & CDC

Zero-to-principal primer for Phase 05: the stream-processing engine landscape (Kafka Streams, Spark Structured Streaming, Akka Streams, FS2) and how to choose between them, Change Data Capture, and the join/sessionization patterns every engine must implement.

Table of Contents

• Chapter 1: The Engine Landscape
• Chapter 2: Kafka Streams — Streaming as a Library
• Chapter 3: Spark Structured Streaming — Micro-Batch on the SQL Engine
• Chapter 4: Akka Streams and FS2 — Functional, In-Service Streaming
• Chapter 5: Choosing an Engine
• Chapter 6: Change Data Capture
• Chapter 7: Join Patterns
• Chapter 8: Sessionization and Stateful Patterns
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Engine Landscape

P04 taught Flink as the reference. The others trade Flink's power for different operational shapes:

• Flink: dedicated cluster, record-at-a-time, lowest latency, richest state & exactly-once. Most powerful, most to operate.
• Kafka Streams: a library you embed in your service — no cluster — using Kafka itself for state durability. Lowest ops if you're already on Kafka.
• Spark Structured Streaming: micro-batch on the Spark engine; same DataFrame API as batch; higher latency, but "free" if your org already runs Spark.
• Akka Streams / FS2: in-process, back-pressured streaming inside a single service — for app-level pipelines, not cluster-scale analytics.

The principal point: these are not "better/worse," they're different operational commitments. The skill is matching the engine to the use case (Ch. 5).

Chapter 2: Kafka Streams — Streaming as a Library

Kafka Streams flips the model: instead of submitting a job to a cluster, you write a normal JVM application that links the Streams library and scales by running more instances (in the same consumer group). The model:

• KStream = an unbounded stream of records (events). KTable = a changelog folded into the latest-value-per-key table (P02 compaction, as a first-class type). The two interconvert (toTable, toStream) — the stream/table duality made API. The lab's apply_cdc is a KStream→KTable fold.
• Local state + changelog topics: stateful operations keep state in a local RocksDB store; for fault tolerance, every state change is also written to a changelog topic in Kafka, so a failed instance's state is rebuilt by replaying the changelog. Exactly-once is available via Kafka transactions (processing.guarantee=exactly_once_v2).
• Interactive queries: because state is local, you can query it directly (a lightweight serving path).
• When it wins: you're already on Kafka, you want no separate cluster to operate, and your scale fits "run more app instances." When it doesn't: cross-cluster sources, the very largest state, or sub-millisecond latency needs (Flink territory).

Chapter 3: Spark Structured Streaming — Micro-Batch on the SQL Engine

Spark Structured Streaming models a stream as an unbounded table that grows; each micro-batch processes the new rows with the same DataFrame/SQL API as batch Spark (P06). Properties:

• Micro-batch means latency is bounded by the batch interval (typically hundreds of ms to seconds) — higher than Flink's record-at-a-time, but plenty for most analytics. A continuous mode exists for lower latency with weaker guarantees.
• Checkpointing to a durable store (HDFS/S3) plus idempotent/transactional sinks gives exactly-once (the offsets processed and the output are tracked together).
• Watermarks & windows exist (similar semantics to P04) but with micro-batch granularity.
• When it wins: you already run Spark, your latency budget is seconds not milliseconds, and you want one engine and one API for batch and streaming (the unification). When it doesn't: hard low-latency or very large keyed-state, fine-grained timers — Flink.

Chapter 4: Akka Streams and FS2 — Functional, In-Service Streaming

These are not cluster engines; they're in-process streaming libraries for building back-pressured pipelines inside a service (an ingestion gateway, an enrichment microservice, a connector):

• Akka Streams: a Reactive Streams implementation on the Akka actor runtime. You build a typed graph of Source → Flow → Sink with built-in backpressure (demand flows upstream — P01 Ch. 8 made concrete). Great for bounded-resource, composable in-service streaming with strong typing.
• FS2 (Functional Streams for Scala): pure, referentially-transparent streams on Cats Effect (P12), with resource safety (a stream that opens a file/connection releases it even on failure or cancellation) and backpressure by construction. The functional-purist's choice.
• These matter for this role because the JD explicitly lists Akka/Cats/ZIO/FS2 and asks you to build internal SDKs (P12). You'll often build the platform's in-service components (a validating ingestion sidecar, a CDC connector) with exactly these libraries, while the cluster-scale processing runs on Flink/Spark.

Chapter 5: Choosing an Engine

The decision matrix (and the dial each turns — P00):

The principal answer is always contextual: "low-latency stateful analytics at scale → Flink; we're all-Kafka and want no cluster → Kafka Streams; we already run Spark and need seconds-latency with one API → Spark SS; it's a single service's internal pipeline → Akka/ FS2." Then the ADR.

Chapter 6: Change Data Capture

CDC turns a database's commit log (MySQL binlog, Postgres WAL) into a stream of row-level change events — the bridge from OLTP systems to the lakehouse and to event-driven consumers, without the dual-write problem (you don't write to both the DB and Kafka; you capture what the DB already committed).

• Debezium is the dominant connector: it reads the binlog/WAL and emits change events (op = c/u/d, before/after images, source LSN) to Kafka.
• Snapshot + streaming: on first run, CDC takes a consistent snapshot of the table (the current rows), then switches to streaming the log from the snapshot's position — so consumers get the full state and all subsequent changes, with no gap. (Extension B.)
• Applying CDC (the lab): fold the change stream into a materialized table. The correctness keys: apply changes in LSN order, make it idempotent by version (a re-delivered or out-of-order change is a no-op — P01 again), and keep tombstone version memory so a stale insert can't resurrect a deleted row. This is exactly how a CDC sink keeps a lakehouse upsert table (P09) or a Cassandra table (P11) correct.
• Ordering & keys: CDC events for a row must stay ordered → key the Kafka topic by the primary key (P02 per-key ordering). Schema changes in the source flow through as schema evolution (P03).

Chapter 7: Join Patterns

Joins are where streaming gets subtle, because at least one side is unbounded:

• Stream ↔ Table (temporal / "as-of" join): enrich each event with a dimension value — but the value valid at the event's time, not the latest. Using the latest is temporal leakage: you'd attribute an event to a dimension state that didn't exist yet (a user's current tier applied to last month's event). The lab's temporal_join does the point-in-time-correct lookup; this is the streaming twin of the feature-store point-in-time-join leakage bug.
• Stream ↔ Stream (interval / windowed join): join two streams where the records fall within a bounded time interval of each other (click→purchase within 30 min). The bound is what makes it feasible — each side's state need only be retained for the interval width (Flink's interval join). Unbounded "join everything to everything" is unimplementable at scale; the bound is the design.
• Enrichment from an external store: look up a key in Cassandra/DynamoDB (P11) per event — use async I/O (don't block the operator) and a cache; mind the consistency (the store may lag the stream).

Chapter 8: Sessionization and Stateful Patterns

• Sessionization (the lab): group a key's events into sessions separated by an inactivity gap (a 30-min silence ends the session). Boundaries are dynamic (unlike fixed windows), and a late event can bridge two sessions into one (the merge logic Flink session windows implement). The natural shape for "web session," "trip," "viewing session" features.
• Other stateful patterns you'll implement on these engines: dedup (P01's seen-ids, with bounded TTL state), top-N / leaderboards (keyed sorted state), running aggregates (incremental, not buffer-all), pattern detection (CEP / MATCH_RECOGNIZE — the fraud-rule sequences). All share the lesson: bound the state (TTL, window, gap), or it grows forever (P04 Ch. 4).

Lab Walkthrough Guidance

Lab 01 — Streaming Joins & CDC Apply, suggested order:

1. apply_cdc — version-gated apply; delete with tombstone memory; then the idempotent-under-shuffle and stale-insert-after-delete tests (the correctness core).
2. temporal_join — sort dim versions per key, bisect for the latest valid_from ≤ ts; None before the first version (no leakage).
3. interval_join — per-key, the [ts+lower, ts+upper] window; validate lower ≤ upper.
4. sessionize — per-key, split when ts − last > gap (gap exclusive — the boundary test).

Success Criteria

You are ready for Phase 06 when you can, from memory:

1. Contrast Flink, Kafka Streams, Spark SS, and Akka/FS2 and pick one for a stated use case.
2. Explain KStream vs KTable and Kafka Streams' changelog-topic fault tolerance.
3. Explain CDC, the snapshot+streaming bootstrap, and the idempotent-by-LSN apply.
4. Explain the as-of join and the temporal-leakage bug it prevents.
5. Explain why stream-stream joins must be bounded and how sessionization differs from fixed windows.

Interview Q&A

Q: When would you NOT use Flink? When its operational weight isn't justified. If we're already all-in on Kafka and the use case fits "run more app instances," Kafka Streams removes a whole cluster to operate. If we already run Spark and our latency budget is seconds with batch+stream sharing one API, Spark Structured Streaming is the lower-friction choice. If it's a single service's internal back-pressured pipeline, Akka Streams or FS2 keeps it in-process. Flink earns its operational cost when you need low-latency, large keyed-state, fine-grained timers, and the strongest exactly-once — otherwise a lighter engine is the principal choice.

Q: How does CDC stay correct under retries and reordering? Each change carries a monotonic source LSN/version, and the apply is version-gated: a change is applied only if its version exceeds the last applied version for that key, so duplicates and out-of-order delivery are no-ops — exactly-once effect on the materialized table. Deletes keep tombstone version memory so a stale insert that arrives after a delete (with a lower version) can't resurrect the row. Order is preserved per row by keying the CDC topic on the primary key. The snapshot+streaming handover ensures no gap between the initial state and the change stream.

Q: A teammate enriched events with users.current_tier. What's wrong? Temporal leakage. They joined each event to the user's current tier, but the correct value is the tier that was valid at the event's timestamp — an as-of join. Using the current value means a user who upgraded last week has all their old events mislabeled as "pro," which corrupts any time-based analysis or training data (it's the same point-in-time bug as a feature store). The fix is a temporal join keyed by event time against the dimension's validity intervals.

References

• Kafka Streams developer guide — KStream/ KTable, changelog topics, exactly-once
• Spark Structured Streaming programming guide
• Akka Streams · FS2 (built for real in P12)
• Debezium documentation — CDC connectors, snapshots
• Kleppmann, DDIA Ch. 11 (change data capture, stream-table joins)

🛸 Hitchhiker's Guide — Phase 05: Streaming Alternatives & CDC

Read this if: you know Flink (P04) and need the rest of the streaming world fast — the other engines, when to pick each, CDC, and the join patterns. Skim, read WARMUP, build the lab.

0. The 30-second mental model

Flink isn't the only stream processor — it's the heaviest. Kafka Streams is a library (no cluster), Spark Structured Streaming is micro-batch on the Spark engine, Akka Streams / FS2 are in-service back-pressured pipelines. They differ in ops and latency, not in the patterns: CDC (change stream → table), as-of joins, interval joins, sessionization. One sentence: pick the engine by operational fit; the patterns are the same everywhere.

1. Engine picker (one line each)

Flink           → low-latency, large state, strongest exactly-once; a cluster to run
Kafka Streams   → a LIBRARY in your app, state in RocksDB + changelog topics; no cluster
Spark Str. Str. → micro-batch (seconds), same API as batch Spark; free if you're on Spark
Akka Streams    → in-service, typed, back-pressured graphs (Reactive Streams)
FS2             → in-service, pure/resource-safe streams on Cats Effect  (P12)

2. KStream vs KTable (the duality, again)

KStream = events (append)        KTable = latest-value-per-key (compacted log → table)
toTable() / toStream() interconvert     state durability = changelog topic in Kafka

This is P02 compaction as a typed API; your apply_cdc is a KStream→KTable fold.

3. CDC in 5 bullets

• DB commit log (binlog/WAL) → change events (Debezium): op=c/u/d, before/after, LSN.
• No dual-write: you capture what the DB already committed.
• Snapshot + streaming: full table snapshot, then stream from that position (no gap).
• Apply idempotently by version/LSN: dup/out-of-order = no-op (P01).
• Tombstone version memory: keep the version after delete so a stale insert can't resurrect the row. Key the topic by PK for per-row ordering.

4. Join patterns (and their traps)

stream↔table (AS-OF)   : value valid AT EVENT TIME, not latest → else TEMPORAL LEAKAGE
stream↔stream (INTERVAL): r.ts ∈ [l.ts+lower, l.ts+upper] → bound = bounded state
enrichment (external)   : async I/O + cache; the store may lag the stream

5. Sessionization

new session when (ts - last) > gap   (gap exclusive)
boundaries are DYNAMIC; a late event can BRIDGE two sessions into one (merge)

The shape for web sessions, trips, viewing sessions.

6. Beginner mistakes that mark you

1. "Always Flink" — can't justify it; ignoring Kafka Streams / Spark SS fit.
2. Enriching with the current dimension value → temporal leakage.
3. Unbounded stream-stream joins → unbounded state.
4. Blocking external lookups in an operator (no async I/O) → backpressure.
5. CDC apply without LSN gating → duplicates/resurrected rows.
6. Forgetting CDC needs PK-keyed topics for per-row ordering.

7. War stories

• "Old events all show users as 'pro'." → enriched with current tier; needed as-of join.
• "The CDC table has rows we deleted." → no tombstone version memory; a stale insert resurrected them.
• "Our enrichment service fell over under load." → synchronous external lookups; switch to async I/O + cache.
• "Kafka Streams app lost its state on redeploy." → no/short changelog retention; the changelog is the durability.

8. How this phase pays off later

• CDC apply → P09 lakehouse upserts, P11 Cassandra/DynamoDB sinks.
• As-of join → capstone fraud detector enrichment (P16).
• Akka/FS2 → built for real in Scala (P12).
• Engine picker → ADRs in P15.

Read WARMUP, build the lab, then P06: we cross from streaming into batch — Spark internals, the other half of the platform.

👨🏻 Brother Talk — Phase 05, Off the Record

The engine wars, CDC, and joins — where I'll save you from two expensive mistakes: over-engineering with Flink, and the leakage bug that quietly poisons your data.

Brother, let me start with a confession that'll save you grief: for a long time I reached for Flink for everything, because it was the powerful, impressive choice. And I paid for it — in operational toil, in 3 a.m. pages for a cluster that was overkill for the job, in onboarding pain for teammates who just wanted to enrich a stream. The lesson took too long: the most powerful tool is not the best tool; the best tool is the one that fits the job with the least operational weight. A principal who reaches for Flink reflexively is signaling the same immaturity as a junior who reaches for microservices reflexively.

So here's the engine maturity ladder, the way I think about it now. If you're already on Kafka and the job fits "run more app instances," Kafka Streams lets you delete an entire cluster from your life — your streaming logic lives in a normal app you deploy normally. If you already run Spark and your latency budget is seconds, Spark Structured Streaming means one engine, one API, one set of skills for batch and stream. If it's a single service's internal pipeline, Akka Streams or FS2 keeps it in-process where it belongs. And then, when you genuinely need millisecond latency with huge keyed state and the strongest exactly-once — then Flink earns its keep. Being able to say "we don't need Flink here, and here's the lighter thing that fits" is a more senior move than knowing every Flink knob.

Now, the bug. I'm going to plant this so deep you'll never commit it: temporal leakage in joins. It's the most innocent-looking, most damaging mistake in stream enrichment. You have events, you have a dimension table (users, products, accounts), you join them — and the obvious thing, the thing every tutorial shows, is to look up the dimension's current value. And it's wrong. Because the event happened in the past, and the dimension's value then may differ from now. A user who's "pro" today was "free" when they generated last month's events. If you label those old events "pro," every time-based analysis is corrupted, every model trained on that data learns a lie, and — the cruel part — nothing crashes. The dashboard looks fine. The model trains fine. It's just subtly, silently wrong, and you find out months later when a number doesn't reconcile. The fix is the as-of join: look up the value that was valid at the event's timestamp. Build it in this lab, feel why the point-in-time lookup matters, and carry that paranoia forever. (It's the same bug as feature-store leakage — the data world keeps re-discovering it under different names.)

Third thing: CDC is magic, and like all magic it has rules. Change Data Capture is one of the most powerful patterns you'll wield — it turns any database into an event stream without the dual-write problem, because you're capturing what the DB already committed rather than asking the app to write twice. But the apply side has to be correct or you get corruption that's miserable to debug. The rules: apply in LSN order, gate by version so duplicates and reordering are no-ops, and — the one people forget — keep tombstone memory so that after you delete a row, a stale older insert can't wander in and resurrect it. I've seen "deleted" customers reappear in a downstream table because someone's CDC apply forgot that last rule. It looks like a ghost. It's an ordering bug. Build the idempotent apply in the lab and you'll never ship the ghost.

A word on the functional libraries — Akka Streams and FS2. They'll feel out of place next to Flink and Spark, and I want you to not skip them, because they're a preview of the most intellectually rewarding phase in this whole track (P12). When you build the platform's in-service pieces — a validating ingestion sidecar, a CDC connector, an enrichment service — you'll often reach for exactly these, and they bring something the cluster engines don't: resource safety by construction. A stream that opens a connection cannot forget to close it, even on failure, because the type system and the runtime guarantee it. That's a different kind of correctness than checkpoints — it's correctness you get at compile time — and once you taste it, sloppy resource handling in other languages starts to feel barbaric. Hold that feeling; P12 turns it into a superpower.

Career note: the person who can look at a problem and say "this doesn't need a cluster" saves the org real money and real toil, and that gets noticed by the people who pay the cloud bill and carry the pager. Restraint reads as seniority. The flashy move is building the impressive distributed thing; the principal move is building the right-sized thing and being able to defend why it's right-sized with the workload's numbers. Practice saying no to Flink as confidently as you say yes.

Build the lab. Make CDC idempotent under a shuffled, duplicated stream. Make the as-of join refuse to leak. Then come to P06, where we cross the bridge from streaming to batch and take apart Spark — the engine that, when it misbehaves, generates the most expensive bills and the longest job runtimes in the business.

— your brother 👨🏻

Lab 01 — Streaming Joins & CDC Apply

Phase: 05 — Streaming Alternatives & CDC | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Flink, Kafka Streams, and Spark Structured Streaming differ in API and runtime, but they all have to get the same four patterns correct. This lab implements them engine-agnostically: CDC materialization (change stream → table, idempotent by LSN), the as-of join (point-in-time correct stream↔dimension enrichment — the leakage killer), the interval join (stream↔stream within a time window), and sessionization.

What you build

• apply_cdc — fold a Debezium-style change stream into a materialized table; apply only newer versions so duplicates and out-of-order delivery are no-ops (exactly-once effect on the table); deletes leave tombstone version memory so a stale insert can't resurrect a deleted key
• temporal_join — enrich each event with the dimension value valid at the event's time (not the latest) — using a future value is temporal leakage
• interval_join — pair two streams within [ts+lower, ts+upper]; bounded intervals bound the state
• sessionize — gap-based session windows per key (the merge logic of Flink session windows, distilled)

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 14 tests pass — test_cdc_idempotent_under_duplicate_and_shuffle and test_cdc_stale_insert_after_delete_rejected certify the CDC correctness model.
• You can explain why the as-of join must use the event-time-valid dimension value, and what bug you get if you use the latest.
• You can explain why a stream-stream join needs a bounded interval to be feasible.

Extensions

• Add late-data handling to the as-of join (a dimension update that arrives after the event — do you re-emit? this is P04's allowed-lateness on the dimension side).
• Implement the CDC snapshot + streaming bootstrap (initial full-table snapshot then switch to the change stream — Debezium's actual lifecycle).
• Add a windowed stream-stream join with watermark-driven state cleanup (combine this lab with P04's watermark machinery).

Phase 06 — Apache Spark Internals

Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Prerequisites: Phase 01 (partitioning, skew), Phase 08 helps but isn't required yet

Why This Phase Exists

Spark is the workhorse of batch (and a serious streaming engine, P05), and it is the subject of interview Round 3: "a Spark job on EMR runs for 9 hours, spills heavily, produces millions of small Parquet files, and causes Athena queries to scan excessive data." The JD demands you "understand Spark execution internals: DAGs, stages, tasks, shuffles, broadcast joins, adaptive query execution, partitioning, memory management, spill behavior, executor sizing, speculative execution, and failure recovery" and "debug slow Spark jobs, shuffle explosions, data skew, executor OOMs, excessive spill, bad partitioning, small-file problems."

This phase makes those words mechanical. The performance of a Spark job is almost entirely determined by shuffle, skew, partitioning, memory, and file layout — and once you can reason about those five, a "9-hour job" becomes a checklist, not a mystery.

Concepts

• The execution hierarchy: application → jobs (per action) → stages (split at shuffle) → tasks (one per partition) → executors/cores; the DAG scheduler.
• Narrow vs wide dependencies: narrow (map/filter — pipelined) vs wide (groupBy/join/ repartition — shuffle); why the shuffle is the cost center.
• The shuffle: map-side write (sorted/spilled to disk) → reduce-side fetch; shuffle service; why shuffles dominate runtime, network, and disk; shuffle explosions.
• Data skew: a few hot keys → fat partitions → straggler tasks; detection (median multiple) and fixes (salting, AQE skew-join, broadcast).
• Joins: broadcast-hash vs sort-merge vs shuffle-hash; the broadcast threshold; AQE runtime join switching and skew-join splitting; dynamic partition pruning.
• Adaptive Query Execution (AQE): coalescing shuffle partitions, switching join strategies, optimizing skew — using runtime statistics.
• Memory & spill: unified memory (spark.memory.fraction), execution vs storage, Tungsten off-heap, spill to disk, executor OOM, executor sizing (cores × memory).
• File layout: the small-file problem, target file sizes, repartition/coalesce before write, output partitioning (and over-partitioning), Parquet (P08).
• Reliability: task retries, speculative execution, stage recomputation, idempotent writes (staging + atomic commit / table formats), safe backfills and reprocessing.
• Spark SQL & Catalyst: logical → optimized → physical plan; predicate pushdown; cost-based optimization (P10); reading explain().

Labs

Lab 01 — Spark Execution Simulator (flagship, implemented)

→ Lab folder: lab-01-spark-simulator/

Lab 02 — Spark Execution Model in Scala (implemented, sbt test)

→ Lab folder: lab-02-scala-spark-model/

Extension Project A — Tune a real Spark job (spec)

Generate a skewed dataset; run a groupBy/join locally (PySpark); read the Spark UI; fix it with AQE, repartitioning, broadcast, and salting; measure the runtime and small-file count before/after.

Extension Project B — Safe backfill framework (spec; → P13)

Design (and sketch in code) a backfill that is idempotent (staging + atomic commit or a table-format write), partitioned by date, resumable, and reproducible — the JD's "reproducible and auditable backfills."

Integrated-Scenario Hooks

• This phase's small-file report is the cause of the Athena scan-cost explosion (P10) and the motivation for compaction (P09).
• Its idempotent write pattern is realized by table-format commits (P09).
• Its skew is P01's skew, fourth costume; its shuffle sizing reappears for EMR cost (P07).

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• Spark performance = shuffle + skew + partitioning + memory + file layout. Master those five.
• A stage boundary is a shuffle; tasks = partitions; the shuffle is where time and money go.
• Spill is a partition-count problem as much as a memory one; AQE coalesces small partitions.
• Small files are the silent killer that turns into Athena's bill (P10).
• Idempotent writes (staging + atomic commit / table formats) make backfills safe.

Deliverables Checklist

• Lab 01 implemented; all 16 tests pass
• You can narrate the Round-3 9-hour-job diagnosis end to end
• You can explain AQE's three tricks (coalesce, join switch, skew-join)
• Extension A real-tune or Extension B backfill design

Warmup Guide — Apache Spark Internals

Zero-to-principal primer for Phase 06: how Spark turns your code into jobs/stages/tasks, why the shuffle dominates everything, how skew/spill/small-files actually arise, what AQE does, and how to read and fix a 9-hour job.

Table of Contents

• Chapter 1: The Execution Hierarchy
• Chapter 2: Narrow vs Wide — Why Shuffles Make Stages
• Chapter 3: The Shuffle in Detail
• Chapter 4: Data Skew
• Chapter 5: Join Strategies
• Chapter 6: Adaptive Query Execution
• Chapter 7: Memory, Spill, and Executor Sizing
• Chapter 8: File Layout and the Small-File Problem
• Chapter 9: Reliability, Idempotent Writes, and Backfills
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Execution Hierarchy

Spark turns a program into a hierarchy you must be able to name to debug:

• Application → one driver + executors. Job → triggered by each action (count, write, collect). Stage → a set of tasks that can run without a shuffle; stages are split at shuffle boundaries (Ch. 2). Task → the unit of execution, one per partition, run on an executor core.
• So the parallelism of a stage = its partition count, and the wall-clock time of a stage ≈ the slowest task (the straggler — Ch. 4). Two numbers explain most performance: how many partitions, and how evenly the data is spread across them.
• The DAG scheduler builds the stage graph; the task scheduler places tasks on executors honoring locality. The lab's plan_stages is the DAG scheduler's split-at-shuffle in miniature.

Chapter 2: Narrow vs Wide — Why Shuffles Make Stages

A transformation's dependency determines whether it needs a shuffle:

• Narrow: each output partition depends on one input partition (map, filter, union). These pipeline within a stage — no data movement across the network.
• Wide: each output partition depends on many input partitions (groupByKey, reduceByKey, join, repartition, distinct, sort). These require a shuffle: redistribute data by key across the cluster. A shuffle materializes intermediate data (written to disk) and ends a stage — the next stage reads the shuffle output.

So #stages = #shuffles + 1, and every wide op is a place where data crosses the network and hits disk. This is why the principal's first instinct on a slow job is "how many shuffles, and can I remove one?" (e.g. replace a shuffle-join with a broadcast-join, Ch. 5).

Chapter 3: The Shuffle in Detail

The shuffle is the most expensive operation in Spark and the source of most pathologies:

• Map side: each task partitions its output by the target partition (hash of key), sorts/buffers it, and spills to disk when buffers fill, producing shuffle files.
• Reduce side: each task in the next stage fetches its partition's data from every map task (an all-to-all transfer) — M × R connections in the worst case. The external shuffle service serves these so executors can die without losing shuffle data.
• Shuffle explosion: too many partitions (tiny tasks, scheduler overhead, many small files) or too few (huge tasks, spill, OOM). spark.sql.shuffle.partitions defaults to 200 — almost always wrong for your data; size it by bytes (the lab's shuffle_partitions: ceil(total / 128MB)).
• The cost levers: fewer shuffles (broadcast, pre-partitioning/bucketing), right-sized partitions (128–256 MB), and AQE (Ch. 6).

Chapter 4: Data Skew

Skew is P01's hot-partition problem, now killing a batch job: a few hot keys (a "whale" account, a NULL join key, a Zipfian distribution) send disproportionate data to a few reduce partitions, so a handful of tasks run for hours while the rest finish in seconds — and the stage can't complete until the straggler does.

• Detect: per-partition (per-task) input sizes in the Spark UI; a few partitions far above the median (the lab's detect_skew, median-multiple). Also: one task at 100% while others idle; massive spill on a few tasks.
• Fix:
  • Broadcast the other side if it's small (no shuffle at all — Ch. 5).
  • Salting: append a random bucket to the hot key (key#0..key#n), aggregate in two stages (per-bucket then combine) — spreads the whale across partitions (P01's salt).
  • AQE skew-join (Ch. 6): Spark automatically splits skewed partitions into sub-partitions at runtime.
  • Filter/handle the NULL-key case explicitly (a shockingly common cause).

Chapter 5: Join Strategies

Spark picks a join physical plan; knowing them lets you force the right one:

• Broadcast-hash join: ship the small side to every executor and hash-join locally — no shuffle. The fastest when one side fits spark.sql.autoBroadcastJoinThreshold (default 10 MB; raise it deliberately for medium dims). The lab's should_broadcast.
• Sort-merge join: shuffle both sides by key, sort, merge. The default for two large tables; robust but pays two shuffles + sorts.
• Shuffle-hash join: shuffle both, build a hash table on one side; used in narrower cases.
• Dynamic partition pruning: at runtime, use the dimension's join keys to prune the fact-table partitions scanned — a big win for star schemas.

The principal move: confirm the small side is actually small (stats can be stale → ANALYZE TABLE), broadcast it, and you've turned a two-shuffle sort-merge into a no-shuffle broadcast — often the single biggest speedup.

Chapter 6: Adaptive Query Execution

AQE re-optimizes the plan at runtime using actual shuffle statistics (instead of trusting stale compile-time estimates). Its three tricks, all interview-relevant:

1. Coalesce shuffle partitions: after a shuffle, merge small adjacent partitions up to a target size — so you can set shuffle.partitions high without ending up with thousands of tiny tasks (the lab's coalesce_partitions). Note it never splits a big partition — that's the skew path.
2. Switch join strategy: a planned sort-merge join becomes a broadcast join once runtime stats reveal one side is actually small.
3. Skew-join handling: detect skewed partitions and split them into sub-partitions that run in parallel — automating Ch. 4's salting.

Turn AQE on (spark.sql.adaptive.enabled=true, default in modern Spark) and a lot of manual tuning evaporates — but you still must understand what it's doing to debug when it doesn't help (e.g. it can't fix skew it can't see, or a UDF that blocks pushdown).

Chapter 7: Memory, Spill, and Executor Sizing

• Unified memory: an executor's heap is split (after reserved memory) into a region governed by spark.memory.fraction (default 0.6), shared between execution (shuffle, join, sort, aggregation buffers) and storage (cached data), which borrow from each other. The rest is user memory.
• Spill: when a task's working set (e.g. a partition being sorted/aggregated) exceeds the available execution memory, Spark spills to disk — correct but slow (the lab's estimate_spill). The fix is frequently more partitions (smaller per-partition working set), not just more memory — a key insight juniors miss.
• Executor OOM: too few partitions (giant tasks), a huge collect/broadcast, skew piling data on one executor, or too many cores per executor (each core's task competes for the same heap). Sizing: a common rule is ~4–5 cores/executor and memory sized so target-size partitions fit with headroom; more, smaller executors often beat few giant ones.
• Off-heap / Tungsten: Spark's binary memory format + code generation reduce GC and object overhead — why Spark SQL is faster than RDD-of-objects.

Chapter 8: File Layout and the Small-File Problem

How Spark writes determines downstream cost (P08–P10):

• A write produces one file per output partition per task, so 200 shuffle partitions × N tasks can emit millions of tiny files — the small-file problem. Tiny files mean huge metadata, slow listing, poor scan performance, and (on S3) request-cost and throttling pain. The lab's small_file_report quantifies the fix.
• Target file size: 128 MB – 1 GB. Too small = the tax above; too large = poor read parallelism and memory pressure.
• Fixes: repartition(n) / coalesce(n) to target the right file count before write (repartition shuffles to balance; coalesce avoids a shuffle but can't increase partitions); partition the output by a sensible column (date) without over-partitioning (which recreates the small-file problem per partition); run compaction (P09) on table formats.

Chapter 9: Reliability, Idempotent Writes, and Backfills

• Fault tolerance: tasks retry on failure; stages recompute from shuffle data (or recompute the lineage if shuffle data is lost). Speculative execution re-launches slow ("straggler") tasks on other executors and takes whichever finishes first — helps with bad nodes, wastes resources on genuine skew (where the fix is Ch. 4, not speculation).
• Idempotent writes: a naive job that fails mid-write can leave partial output; a retry then duplicates. The fixes: write to a staging path then atomically rename/commit (the FileOutputCommitter v2 caveats matter on S3 — no atomic rename!), or use a table format (P09) whose commit is atomic (the safe modern answer). Overwrite a partition atomically (INSERT OVERWRITE dynamic partition) for partition-scoped backfills.
• Safe backfills (a JD deliverable): partitioned by date, resumable (reprocess only failed partitions), reproducible (deterministic logic — event-time, P01 — so a rerun yields identical output), and auditable (record what was reprocessed). The test of a good backfill is the same as a good replay: rerun it and get the same answer.

Lab Walkthrough Guidance

Lab 01 — Spark Execution Simulator, suggested order:

1. plan_stages/num_stages — new stage at each wide op; #stages = #wide + 1.
2. detect_skew — median-multiple; the fat partition pops out.
3. coalesce_partitions — greedy merge to target; the big partition stands alone.
4. should_broadcast + shuffle_partitions — the join + sizing decisions.
5. estimate_spill — partition vs execution-memory share.
6. small_file_report — the compaction pitch (reduction ratio).

Success Criteria

You are ready for Phase 07 when you can, from memory:

1. Map application → job → stage → task and say what creates a stage.
2. Explain the shuffle (map write/spill → reduce fetch) and M×R cost.
3. Diagnose skew (detection + four fixes) and explain salting.
4. Choose broadcast vs sort-merge and state the threshold.
5. Explain AQE's three tricks.
6. Explain spill and why more partitions can fix it; size executors sanely.
7. Explain the small-file problem and how to fix it on write.
8. Describe an idempotent, reproducible backfill.

Interview Q&A

Q (Round 3): A Spark job runs 9 hours, spills heavily, writes millions of small Parquet files, and Athena then scans too much. Walk me through it. Four linked symptoms. The 9 hours + heavy spill point at shuffle + skew: I'd check the Spark UI for a stage where one task runs far longer than the rest (skew) and for partitions exceeding execution memory (spill). Fix skew by broadcasting the small side if applicable, salting the hot key, or enabling AQE skew-join; fix spill by raising the partition count (smaller working sets) rather than just memory. The millions of small files come from too many output partitions × tasks — I'd repartition/coalesce to target 128 MB–1 GB files before write (or write to a table format and compact). That same small-file problem is why Athena scans too much and costs too much (P10): few large, well-partitioned, columnar files let Athena prune and skip. So the through-line is: tame the shuffle/skew to fix runtime, fix file sizing to fix both the write and the downstream scan cost.

Q: When does adding executor memory NOT fix spill? When the real problem is too few partitions. Spill happens when a single task's working set exceeds its execution-memory share; if you have 200 partitions over a 1 TB shuffle, each task handles ~5 GB and will spill regardless of reasonable heap sizes. Doubling memory delays it; doubling the partition count halves each task's working set and often eliminates it. So my first move on spill is "are partitions sized to ~128–256 MB?" before "give it more RAM" — more memory is the expensive fix for a free one.

Q: Broadcast vs sort-merge join — how do you decide, and what's the trap? If one side fits the broadcast threshold (default 10 MB, often worth raising for medium dimensions), broadcast it — no shuffle, dramatically faster. Otherwise sort-merge (shuffle both, sort, merge). The trap is stale statistics: Spark might not broadcast because it thinks a side is bigger than it is (or AQE hasn't kicked in), so I ANALYZE TABLE to refresh stats or hint the broadcast explicitly. The bigger trap is broadcasting something that's not actually small and OOMing the executors — confirm the size, don't guess.

Q: How do you make a backfill safe? Idempotent, reproducible, resumable, auditable. Idempotent: write to a staging path with an atomic commit, or use a table format (Iceberg/Delta) whose commit is atomic and overwrite by partition — so a retry can't duplicate. Reproducible: deterministic logic keyed on event time (P01), so a rerun produces identical output. Resumable: partition by date and reprocess only the failed partitions. Auditable: record which partitions were rewritten and from what input version. The acceptance test is the replay test — run it twice, diff the output, expect zero.

References

• Spark: The Definitive Guide (Chambers & Zaharia); Learning Spark, 2e
• High Performance Spark (Karau & Warren) — skew, shuffle, memory in depth
• Spark SQL performance tuning & AQE
• Spark tuning guide — memory, serialization
• Zaharia et al., Resilient Distributed Datasets (NSDI 2012) — the RDD/lineage paper
• PMC track Phase 07 WARMUP — partitioning/shuffle foundations

🛸 Hitchhiker's Guide — Phase 06: Apache Spark Internals

Read this if: you want to debug a slow/expensive Spark job fast and ace the "9-hour-job" interview round. Skim, read WARMUP, build the simulator.

0. The 30-second mental model

Spark splits your code into stages at every shuffle; each stage runs one task per partition; a stage finishes when its slowest task does. So performance = shuffle + skew + partitioning + memory + file layout. One sentence: fewer shuffles, evenly-sized partitions (128–256 MB), broadcast the small side, let AQE coalesce, and don't write a million tiny files.

1. The hierarchy

app → job (per action) → STAGE (split at shuffle) → TASK (1 per partition) → core/executor
#stages = #shuffles + 1        stage time ≈ slowest task (the straggler)

2. Narrow vs wide

narrow (map/filter)         → pipeline in-stage, no network
wide (groupBy/join/repartition/distinct/sort) → SHUFFLE → new stage, hits disk+network
first instinct on a slow job: "how many shuffles, can I remove one (broadcast)?"

3. The five performance levers

4. The numbers

spark.sql.shuffle.partitions default = 200   ← almost always wrong; size by bytes
target partition / file size = 128 MB – 1 GB
autoBroadcastJoinThreshold default = 10 MB   ← raise deliberately for medium dims
spark.memory.fraction = 0.6 (exec+storage share of heap)
spill ≈ max(0, partition_bytes − heap×0.6)   ← more partitions shrinks this

5. Join picker

small side ≤ threshold → BROADCAST hash join (no shuffle)   ← biggest single speedup
two big tables         → SORT-MERGE (shuffle both, sort, merge)
stale stats hiding a small side → ANALYZE TABLE / broadcast hint
star schema            → dynamic partition pruning

6. AQE = three runtime tricks

1. coalesce small shuffle partitions (never splits a big one)
2. switch sort-merge → broadcast when a side is actually small
3. skew-join: split skewed partitions into parallel sub-partitions
turn it on (default modern Spark); still understand it to debug when it can't help

7. Spill: the counterintuitive one

spill happens when a TASK's working set > its execution-memory share
fix = MORE PARTITIONS (smaller working set) at least as often as more RAM

8. Small files = the silent bill

write emits 1 file / output-partition / task → millions of tiny files
→ huge metadata, slow listing, bad scans, S3 request cost (P08/P10)
fix: repartition/coalesce to target BEFORE write; compact table formats (P09)

9. Beginner mistakes that mark you

1. Leaving shuffle.partitions=200 for any data size.
2. "Add more memory" for spill instead of more partitions.
3. Sort-merge joining when one side fits a broadcast.
4. Writing without controlling file count → small-file explosion.
5. Ignoring the NULL-join-key skew bomb.
6. Non-idempotent writes → duplicate data on retry/backfill.
7. Trusting stale table stats (no ANALYZE) so AQE/broadcast misfire.

10. War stories

• "One task at 100%, 199 idle." → skew. Salt the hot key or AQE skew-join.
• "OOM on a join." → broadcasting a not-actually-small side, or too few partitions.
• "Athena bill 20×'d after the new pipeline." → the Spark job writes millions of small files; compact + partition (P09/P10).
• "Backfill double-counted." → non-idempotent write; staging+atomic commit or table format.

11. How this phase pays off later

• Small-file report → P08 (Parquet sizing), P09 (compaction), P10 (Athena cost).
• Idempotent writes → P09 table-format commits.
• Shuffle/executor sizing → P07 EMR cost & YARN.
• Skew → P01, P02, P11 — same enemy, fifth costume.

Read WARMUP, build the simulator, then P07: where Spark runs — Hive/Tez/MapReduce lineage and EMR operations.

👨🏻 Brother Talk — Phase 06, Off the Record

Spark is where the big cloud bills and the all-night job-watching live. Let me give you the debugging instincts that took me years and a lot of wasted compute to earn.

Brother, here's a number that'll focus your attention: a misbehaving Spark job is often the single most expensive thing in a data platform. A streaming bug loses some events; a Spark bug burns a 200-node cluster for nine hours and produces a million files that then make every downstream query cost ten times more. So getting good at Spark isn't just a skill — it's directly money. The people who can look at a 9-hour job and make it a 40-minute job are paid well precisely because that swing is worth real dollars every single day.

So let me give you the debugging instinct, the one that separates the people who flail at Spark from the people who diagnose it. When a Spark job is slow, do not start changing configs. Open the Spark UI and look at the stages. Find the stage that's taking the time. Look at its tasks. And ask one question: is the time spread evenly across tasks, or is one task taking forever while the rest finished? That single observation forks your entire diagnosis:

• If one task is the straggler → it's skew. A hot key is dumping most of the data onto one partition. No amount of memory or nodes fixes skew — you fix the distribution (salt the key, broadcast the other side, enable AQE skew-join, or kill the NULL-key bomb).
• If all tasks are slow and spilling → it's partition sizing / memory. Your partitions are too big, each task's working set exceeds memory, everything spills to disk. Fix: more partitions, smaller working sets.
• If the job is fast but the output is a zillion files → it's a write/layout problem that becomes someone else's query bill. Fix: coalesce/repartition before write.

That fork — even vs uneven task times — is 80% of Spark debugging, and almost nobody teaches it. They teach you knobs. The knobs are useless until you've read the UI and know which problem you have.

Now, the counterintuitive truth I want to drill into you, because it's the most common expensive mistake: "the job is spilling, give it more memory" is usually wrong. Spill happens when a single task's chunk of data is bigger than the memory that task gets. If you have a terabyte split into 200 partitions, each task is wrestling 5 GB and will spill no matter how fat you make the executors. The fix isn't a bigger executor — it's more partitions, so each task handles 256 MB instead of 5 GB. More memory is the expensive band-aid; more partitions is the free cure. I watched teams 3× their cluster cost fighting spill that a one-line repartition would have fixed. Don't be that team.

The small-file thing deserves its own rant, because it's the gift that keeps on taking. When Spark writes, it makes roughly one file per partition per task, and if you're not paying attention you get millions of tiny files. And here's the cruelty: the Spark job that creates them might run fine. The pain lands downstream — on the Athena queries (P10) that now scan and pay for all that fragmentation, on the metastore that chokes listing them, on the S3 bill for all those requests. So the engineer who created the mess often isn't the one who feels it, which is why it keeps happening. Be the person who controls file count on write. It's a coalesce away, and it saves the org a downstream tax they'd otherwise pay forever.

A word on AQE: turn it on, love it, but don't worship it. Adaptive Query Execution will fix a lot of your partition-sizing and even some skew automatically, and it's genuinely great. But it can only fix what it can see in the runtime stats, and it can't fix a UDF that blocks predicate pushdown, or a fundamentally bad join, or skew it doesn't detect. So understand what it does — coalesce, join-switch, skew-split — so that when it doesn't help, you know why and what to do by hand. AQE is an autopilot; you still need to know how to fly.

The reliability stuff — idempotent writes, safe backfills — feels boring next to performance, but it's where the trust is won. The first time you have to backfill three months of data because of an upstream bug, and you can do it knowing that a retry won't duplicate and a rerun produces byte-identical output, you'll understand why we obsess over it. The teams that fear backfills are the teams whose writes aren't idempotent. The teams that backfill casually, like it's nothing, built on table formats (P09) with atomic commits. Be the second team. "Can you safely reprocess the last quarter?" should make you shrug, not sweat.

Career-wise: Spark expertise is one of the most legible forms of value in data engineering, because the wins are measurable. "I took the nightly job from 9 hours to 45 minutes and cut the cluster cost 70%" is a sentence that gets you promoted, and it's the kind of win this phase teaches you to produce. Performance work has a scoreboard, and a scoreboard is a beautiful thing for a career.

Build the simulator. Internalize the even-vs-uneven fork. Then come to P07, where we look at where Spark actually runs — Hive and Tez and the Hadoop lineage, and EMR operations, where spot instances and cluster sizing turn all this into the cloud bill.

— your brother 👨🏻

Lab 01 — Spark Execution Simulator

Phase: 06 — Apache Spark Internals | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Round 3 of the interview hands you "a Spark job on EMR runs for 9 hours, spills heavily, produces millions of small Parquet files, and makes Athena scan too much." This lab builds the calculator and diagnostics behind every sentence of that answer: stage planning at shuffle boundaries, skew detection, AQE coalescing, broadcast decisions, shuffle sizing, spill estimation, and a small-file report.

What you build

• plan_stages / num_stages — split a logical plan into stages at shuffle (wide) boundaries (the thing that makes Spark Spark)
• detect_skew — partitions far above the median → the "one task runs forever" bug
• coalesce_partitions — AQE's merge of small post-shuffle partitions (and why a fat partition stands alone — coalesce never splits)
• should_broadcast — broadcast-hash-join decision (skip the shuffle for a small side)
• shuffle_partitions — size parallelism by bytes (the fix for the wrong default 200)
• estimate_spill — when a partition exceeds execution memory → spill (a partition-count problem, not just a memory one)
• small_file_report — the compaction pitch ("1M files → 8k") that fixes Athena cost (P10)

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 16 tests pass.
• You can narrate the Round-3 answer end to end using these functions.
• You can explain why spill is fixed by more partitions as often as by more memory.
• You can explain why a >target partition is left alone by coalesce but split by skew-join.

Extensions

• Add salting: split a hot key into key#0..key#n, re-aggregate, and show the skew factor (P01) drop — the manual version of AQE skew-join.
• Model AQE dynamic join switching (a sort-merge join that becomes a broadcast join once runtime stats show one side is small).
• Add an idempotent write planner: stage-dir → atomic commit (foreshadows P09 table-format commits) so a retried job doesn't double-write.

Lab 02 — Spark Execution Model in Scala

Phase: 06 — Apache Spark Internals | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Spark is a Scala/JVM system, so its tuning logic is naturally Scala. This lab is the Scala twin of Lab 01's Python simulator — stage planning at shuffle boundaries, skew detection, AQE coalescing, broadcast/shuffle sizing, spill, and the small-file report — as pure functions verified with ScalaTest (sbt test).

Why a model, not a real SparkSession? Spark 3.5 targets JDK 8/11/17/21; this environment runs JDK 26, where Spark's heavy sun.misc.Unsafe/JDK-internal reflection won't start. The execution model captures the reasoning the interview tests (Round 3) without the runtime — and runs anywhere. The "run it on real Spark" path is the extension.

What you build (SparkModel)

• planStages / numStages — split a plan into stages at wide (shuffle) ops
• detectSkew — partitions > factor × median (the straggler detector)
• coalescePartitions — AQE greedy merge to target (big partition stands alone)
• shouldBroadcast / shufflePartitions — join + parallelism sizing
• estimateSpill — partition vs execution-memory share
• smallFileReport — the compaction pitch (reduction ratio)

Run

sbt test

Expected: 6 tests, all green in SparkModelSpec.

Success criteria

• sbt test → all 6 specs pass.
• You can narrate the Round-3 "9-hour job" diagnosis using these functions (WARMUP).
• You can explain why spill is a partition-count problem and why coalesce never splits a fat partition (that's the skew-join path).

Extensions

• Run it on real Spark (JDK 17/21): add "org.apache.spark" %% "spark-sql" % "3.5.1", build a skewed DataFrame, groupBy/join, read explain(), and confirm AQE coalesces and switches the join — then compare to this model's predictions.
• Add salting (split a hot key k#0..k#n, re-aggregate) and show the skew factor drop.
• Model AQE dynamic join switching (sort-merge → broadcast once runtime stats arrive).

Phase 07 — Hive, Tez, MapReduce & EMR

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 06 (Spark — runs on the same YARN/EMR substrate)

Why This Phase Exists

The JD lists Hive, Tez, MapReduce legacy systems, EMR, EMR Serverless, EMR on EKS, YARN and demands you "understand Hive and Tez execution … metastore behavior … legacy warehouse migration" and "EMR operations: cluster lifecycle, managed scaling, spot interruptions, bootstrap actions, instance fleets, YARN tuning, Spark-on-YARN, EMR Serverless, EMR on EKS, IAM, logging, and cost controls." It also lists migrating off Hive/Tez (Round 5). So you must understand the Hadoop lineage you're modernizing and operate the EMR clusters your Spark/Flink jobs run on — including the cost arithmetic that makes you the "cost efficiency" person.

Concepts

• The Hadoop lineage: HDFS (P08) + MapReduce (map → shuffle/sort → reduce, every stage to disk) → Tez (DAG execution, pipelined, no disk between steps) → why Spark won (in-memory DAGs). Knowing this is knowing what you're migrating from.
• Hive: the metastore (table → files + partitions + stats), the query compiler (HiveQL → MapReduce/Tez DAG), partitions & partition pruning, file formats (ORC native), statistics, bucketing, and the SerDe model. The ancestor of the lakehouse catalog (P10).
• Tez: DAG vertices/edges, container reuse, dynamic parallelism — the bridge between MapReduce and Spark-style execution.
• YARN: ResourceManager/NodeManager, containers (memory + vcores), the ApplicationMaster, schedulers (capacity/fair), and resource binding (the lab) — the cluster OS under Spark/ Hive/Tez on EMR.
• EMR operations: cluster lifecycle (provision → bootstrap → run → terminate), instance fleets/groups, managed scaling, core vs task nodes, spot interruptions & safe placement, bootstrap actions, logging to S3, IAM roles (EMRFS), and cost controls.
• EMR flavours: EMR-on-EC2 (you manage the cluster/YARN), EMR Serverless (no cluster, pay per vCPU/GB-hour), EMR on EKS (Spark on your Kubernetes) — and when each wins.
• Cost: spot economics, right-sizing, the small-file/partition-pruning levers (P06/P10), cost attribution by team (a JD deliverable).

Labs

Lab 01 — EMR/YARN Cost & Scheduling Model (flagship, implemented)

→ Lab folder: lab-01-emr-cost-model/

Extension Project A — Hive → Spark/Iceberg migration plan (spec; → P15)

Map a legacy Hive/Tez warehouse to a modern lakehouse: preserve history, convert ORC layouts, keep the metastore working through the transition, eliminate small files, and decommission MapReduce/Tez — the JD's Problem 3 and Round 5.

Extension Project B — EMR cost review (spec)

Take a (synthetic) month of EMR usage; find waste (idle clusters, on-demand where spot is safe, over-provisioned nodes, full scans from missing partition filters) and produce a prioritized savings list with dollar estimates and cost attribution by team.

Integrated-Scenario Hooks

• This phase's partition pruning is the same lever as Athena cost (P10) and Iceberg partitioning (P09).
• Its spot/core-task model is how the capstone's batch tier (P16) runs cheaply.
• Its migration plan is the spine of P15's Round-5 design review.

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• MapReduce → Tez → Spark is a story of less disk between steps; know it to migrate well.
• Hive's metastore + partitions are the ancestor of the lakehouse catalog (P10).
• YARN packs containers by the binding resource (memory or vcores) — size tasks to fit.
• Spot is cheap but interruptible: core nodes on-demand, task nodes on spot.
• EMR Serverless wins for spiky/short jobs; EC2/EKS for steady or K8s-native ones.

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass
• You can explain the MapReduce → Tez → Spark progression and why each step was faster
• You can choose EMR-on-EC2 vs Serverless vs EKS with the deciding dial
• Extension A migration plan or Extension B cost review

Warmup Guide — Hive, Tez, MapReduce & EMR

Zero-to-principal primer for Phase 07: the Hadoop lineage you're modernizing (MapReduce → Tez → Spark), Hive's metastore and partition model, YARN as the cluster OS, and EMR operations + cost — the substrate your batch jobs run on and the bills you must control.

Table of Contents

• Chapter 1: The Hadoop Lineage — MapReduce → Tez → Spark
• Chapter 2: Hive — Metastore, Partitions, Formats
• Chapter 3: Tez Execution
• Chapter 4: YARN — The Cluster OS
• Chapter 5: EMR Operations
• Chapter 6: EMR Flavours — EC2 vs Serverless vs EKS
• Chapter 7: Spot, Cost, and Cost Attribution
• Chapter 8: Migrating Off the Legacy Stack
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Hadoop Lineage — MapReduce → Tez → Spark

You will spend part of this role migrating off the Hadoop stack, so you must understand it:

• MapReduce (2004): the original model. A job is map → shuffle/sort → reduce, and crucially every stage boundary materializes to HDFS (disk). Multi-step jobs chain many MR jobs, each writing/reading disk — robust but slow. It's the conceptual root of everything (P06's stages are MR's map/reduce generalized).
• Tez (2013): replaced MR under Hive/Pig with a DAG engine — vertices and edges, pipelined, no forced disk write between every step, container reuse, dynamic parallelism. A big speedup for Hive without rewriting queries.
• Spark (2014): in-memory DAGs with lineage-based recovery (P06) — caches intermediate data in memory, generalizes beyond map/reduce, unifies batch + SQL + streaming + ML.

The through-line: each step removed disk I/O between stages. When you justify a Hive/Tez → Spark/Iceberg migration (Round 5), this is the performance story.

Chapter 2: Hive — Metastore, Partitions, Formats

Hive is "SQL over files," and its lasting contribution is the metastore:

• Hive Metastore (HMS): a relational DB mapping logical tables → physical locations (HDFS/S3 paths), partitions, schemas, SerDes, and statistics. Every engine — Spark, Trino, Presto, Flink — can use the HMS (or the Glue Data Catalog, its AWS-managed equivalent, P10) as the source of truth for "what tables exist and where." The lakehouse catalog (P09/P10) is the metastore's descendant.
• Partitions: a Hive table is partitioned by columns (e.g. dt=2026-06-14/region=us), which become directory paths. A query with a predicate on a partition column prunes to the matching directories (the lab's partitions_scanned) — the single biggest cost/perf lever. The classic failure: too many partitions (high-cardinality partition column → millions of tiny directories → metastore meltdown) or the small-file problem within partitions (P06/P08).
• Statistics: ANALYZE TABLE populates row counts and column stats the optimizer needs (P10 CBO); stale stats = bad plans.
• Formats: Hive's native columnar format is ORC (P08), with built-in indexes and bloom filters; Parquet is equally supported. The SerDe abstracts row encoding.
• External vs managed tables: managed = Hive owns the data (drop = delete); external = Hive only catalogs it (drop = metadata only). Get this wrong and you delete the lake.

Chapter 3: Tez Execution

Tez is the DAG engine that made Hive fast before Spark took over. Worth knowing because legacy warehouses still run on it and the JD lists "Tez container failures" as a debugging target:

• A Tez job is a DAG of vertices (each like a map or reduce stage) connected by edges (data movement: one-to-one, broadcast, scatter-gather/shuffle). It pipelines vertices and reuses containers across them (avoiding MR's per-stage JVM startup).
• Dynamic parallelism: Tez can adjust the number of reducer tasks at runtime based on data size (a precursor to Spark's AQE, P06).
• Common failures: container OOM (vertex memory misconfigured), shuffle/skew (same as P06), and metastore bottlenecks under many concurrent queries.

Chapter 4: YARN — The Cluster OS

YARN (Yet Another Resource Negotiator) is the resource manager under MapReduce/Tez/Hive/Spark on EMR-on-EC2:

• ResourceManager (cluster-wide scheduler) + NodeManagers (per-node agents) + per-job ApplicationMaster (negotiates resources, tracks the job). Spark-on-YARN runs the Spark driver in (or alongside) the AM and executors in YARN containers.
• A container = a slice of a node's memory + vcores. How many containers fit on a node = min(node_mem // container_mem, node_vcores // container_vcores) — and the binding resource (whichever runs out first) is what you tune (the lab's yarn_plan). The classic surprise: "this node has 64 cores but only fits 8 containers" because memory bound first.
• Schedulers: Capacity (queues with guaranteed shares) and Fair (equalize over time) — how you isolate tenants on a shared cluster.
• Tuning: container sizing, yarn.nodemanager.resource.memory-mb, overhead for off-heap, and matching Spark executor memory to container memory (with overhead headroom, P06 Ch. 7).

Chapter 5: EMR Operations

EMR is AWS's managed Hadoop/Spark/Flink/Hive. What you operate:

• Cluster lifecycle: provision → bootstrap actions (install deps, configure) → run steps → terminate (transient clusters) or stay (long-running). Transient per-job clusters are a cost pattern; long-running shared clusters are a multi-tenancy pattern.
• Instance fleets / groups: define node roles — master (1, runs the RM/AM), core (run NodeManagers and HDFS DataNodes — losing one risks data + the job), task (compute only, no HDFS — safe to lose). Fleets let you mix instance types and spot/on-demand with target capacities.
• Managed scaling: EMR auto-adds/removes nodes based on YARN demand within min/max bounds — elastic cost without manual resizing.
• Storage: EMRFS lets Hadoop read/write S3 as if it were HDFS (the lake lives in S3, P08); local HDFS is for shuffle/temp. IAM roles (instance profile + EMRFS role) govern S3 access (P14).
• Logging: push YARN/Spark logs to S3 so they survive cluster termination — essential for post-mortems (P13).

Chapter 6: EMR Flavours — EC2 vs Serverless vs EKS

The decision (lab's cost models): steady heavy batch → EC2; bursty/unpredictable → Serverless (no idle cost); already running everything on Kubernetes → EKS (one platform, one scheduler). The hidden factor is idle + bootstrap cost: a cluster that boots for 5 minutes and sits idle between hourly jobs is pure waste — Serverless eliminates it.

Chapter 7: Spot, Cost, and Cost Attribution

The "cost efficiency" mandate is arithmetic:

• Spot instances: up to ~70–90% cheaper than on-demand, but reclaimable with a 2-minute warning. The probability that at least one of your spot nodes is interrupted is 1 − (1−p)^n (the lab) — which grows fast with node count, so at 50 nodes you should expect interruptions and design for them.
• Safe spot placement: put core nodes (HDFS + AM — losing them kills the job/data) on on-demand, and task nodes (stateless compute) on spot (the lab's safe_spot_plan). Checkpoint long jobs so a reclaimed task node just reschedules its work.
• Other levers (cross-phase): partition pruning (this lab) + columnar formats (P08) + compaction/no-small-files (P06/P09) cut scan and storage cost; right-sized containers cut compute; managed scaling + transient clusters cut idle.
• Cost attribution (a JD deliverable): tag every cluster/job/bucket with an owner and a cost-allocation tag, so the bill is a map by team — the thing that actually changes behavior (teams optimize what they're charged for).

Chapter 8: Migrating Off the Legacy Stack

The JD's Problem 3 / Round 5: migrate Hive/Tez/Flume → Kafka/Flink/Spark/Iceberg/Athena. The principal pattern (full treatment in P15):

1. Preserve history: keep the metastore working; the lakehouse can read the same S3 data via the catalog during transition.
2. Convert layouts: ORC/Parquet conversion as needed; fix broken partitions; eliminate small files (compaction).
3. Introduce the table format (Iceberg/Delta, P09): register existing data, then add ACID/time-travel/streaming writes.
4. Dual-run & validate: run old (Hive/Tez) and new (Spark/Iceberg) in parallel; diff outputs (P13 data-diffing) until parity holds.
5. Cut over queries (Athena/Trino, P10) and writers (Flink/Spark); decommission MapReduce/Tez/Flume only after a soak period with rollback available.

The non-negotiables: keep history reproducible, validate parity, and always have a rollback — the same migration discipline as schema evolution (P03) and backfills (P06).

Lab Walkthrough Guidance

Lab 01 — EMR/YARN Cost Model, suggested order:

1. yarn_plan — min(mem//task, vcores//task); report the binding resource (the lesson).
2. cluster_cost_usd — on-demand + spot mix.
3. emr_serverless_cost_usd — per vCPU/GB-hour; scales linearly with time.
4. spot_interruption_probability — 1−(1−p)^n; grows with node count.
5. safe_spot_plan — core on-demand, task spot.
6. partitions_scanned — equality-filter pruning; no filter = full scan.

Success Criteria

You are ready for Phase 08 when you can, from memory:

1. Explain MapReduce → Tez → Spark as a disk-I/O-reduction story.
2. Explain the Hive metastore, partitions, pruning, and managed-vs-external tables.
3. Explain YARN containers and the binding-resource computation.
4. Explain EMR node roles (master/core/task) and why spot goes on task nodes.
5. Choose EMR-on-EC2 vs Serverless vs EKS for a workload, with the deciding factor.
6. Compute spot interruption probability and a safe core/task split.
7. Outline a Hive/Tez → lakehouse migration with validation and rollback.

Interview Q&A

Q: You're asked to cut the EMR bill 40%. Where do you look first? Attribution first — tag clusters/jobs by team so the bill is a map; you can't cut what you can't see. Then the mechanical levers in order of usual impact: (1) idle/bootstrap waste — move spiky jobs to EMR Serverless or transient clusters so you stop paying for warm idle clusters; (2) spot — put task nodes on spot (core nodes stay on-demand) for ~70% off the compute that's safe to interrupt; (3) scan cost — partition pruning + columnar + no small files (P06/P08/P10) often cuts the Athena/Spark scan bill 10×; (4) right-sizing — containers and node types matched to the binding resource. Each is a number I can estimate, so I'd present a prioritized list with dollar figures, not a vibe.

Q: A core node on spot got reclaimed and the job died. What went wrong? Core nodes run HDFS DataNodes and (often) the ApplicationMaster — losing one can lose shuffle/temp data and kill the application, so they must be on-demand. Spot belongs on task nodes, which are stateless compute: a reclaimed task node just reschedules its work elsewhere. The fix is the core/task split, plus checkpointing long jobs so a reclamation is a reschedule, not a restart. At any real node count, spot interruptions are a certainty (1−(1−p)^n), so the architecture has to assume them.

Q: When is EMR Serverless the right call over a cluster? When the workload is spiky, short, or intermittent, because a managed cluster charges you for boot time and idle minutes between jobs, while Serverless bills only the vCPU/GB-hours you actually consume and starts in seconds. For a steady, all-day, HDFS-heavy pipeline a right-sized EC2 cluster with managed scaling can be cheaper and gives more control; for "a few jobs an hour" Serverless usually wins on the idle-cost elimination alone. If the org already runs everything on Kubernetes, EMR-on-EKS unifies the platform and scheduler.

References

• Hadoop: The Definitive Guide (White) — MapReduce, HDFS, YARN
• Programming Hive (Capriolo et al.) — metastore, partitions, formats
• Apache Tez design ; YARN architecture
• Amazon EMR management guide — instance fleets, managed scaling, spot, EMRFS
• EMR Serverless and EMR on EKS docs

🛸 Hitchhiker's Guide — Phase 07: Hive, Tez, MapReduce & EMR

Read this if: you need the Hadoop lineage you're modernizing and the EMR ops + cost arithmetic, fast. Skim, read WARMUP, build the cost model.

0. The 30-second mental model

MapReduce → Tez → Spark is a story of removing disk between stages. Hive gave us the metastore (table→files+partitions, the lakehouse catalog's ancestor). YARN is the cluster OS that packs jobs into memory+vcore containers. EMR runs all of it on AWS — and your job there is mostly cost: spot, right-sizing, partition pruning, no idle clusters. One sentence: know the legacy to migrate it; operate EMR to make it cheap and reliable.

1. The lineage

MapReduce: map → shuffle/sort → reduce, DISK between every stage (slow, robust)
Tez:       DAG, pipelined, container reuse, dynamic parallelism (Hive got fast)
Spark:     in-memory DAG + lineage recovery (P06)  ← what you migrate TO
through-line: each step removed disk I/O between stages

2. Hive = the metastore

Hive Metastore (or Glue Catalog) = table → S3/HDFS path + partitions + schema + stats
partitions = directories (dt=2026-06-14/region=us); predicate → PRUNE to matching dirs
traps: too-many-partitions (high-cardinality col → metastore meltdown), small files
managed table (drop=delete data!) vs external (drop=metadata only)
ANALYZE TABLE for stats (CBO needs them, P10); native columnar = ORC (P08)

3. YARN containers

container = memory + vcores slice of a node
per node = min(node_mem // task_mem, node_vcores // task_vcores)
the BINDING resource (runs out first) is what you tune
"64 cores but only 8 containers" = memory bound

4. EMR node roles (memorize)

master : 1, runs ResourceManager / AM
core   : NodeManager + HDFS DataNode → losing one risks DATA + the job → ON-DEMAND
task   : compute only, no HDFS → safe to lose → SPOT

5. EMR flavours

EMR on EC2     → you run the cluster+YARN; steady heavy batch; idle costs money
EMR Serverless → no cluster, pay per vCPU/GB-hour WHILE running; spiky/short jobs win
EMR on EKS     → Spark on your Kubernetes; K8s-native orgs, one platform
hidden cost: cluster boot + idle minutes → Serverless eliminates it

6. The cost numbers

spot savings ≈ 70–90%, 2-min reclaim warning
P(≥1 interruption) = 1 − (1−p)^n   ← 50 nodes @2% ≈ 0.64 → expect it
Athena/Trino scan = $5/TB → partition prune + columnar + no small files = the bill
EMR Serverless ≈ $0.052/vCPU-hr + $0.0058/GB-hr

7. Cost-cut playbook (in order of usual impact)

1. attribute (tag by team) — can't cut what you can't see
2. kill idle/bootstrap waste (Serverless / transient clusters)
3. spot the task nodes (~70% off safe compute)
4. scan cost: partition prune + columnar + compaction (often 10×)
5. right-size containers/nodes to the binding resource

8. Beginner mistakes that mark you

1. Core nodes on spot → reclaim kills HDFS + the job.
2. High-cardinality partition column → millions of dirs, metastore meltdown.
3. Dropping a managed table thinking it's external → data gone.
4. Paying for warm idle clusters between hourly jobs (use Serverless).
5. Sizing containers without checking which resource binds.
6. No partition filter → full-table scan → the cost explosion.

9. War stories

• "Spot reclaim killed the cluster." → core node was on spot. Core=on-demand always.
• "Metastore is timing out." → someone partitioned by user_id (millions of dirs).
• "The bill doubled and nobody knows whose job." → no cost-allocation tags.
• "DROP TABLE deleted the lake." → it was a managed table, not external.

10. How this phase pays off later

• Partition pruning → P09 (Iceberg partitioning), P10 (Athena cost).
• Metastore → P10 (Glue Catalog, Lake Formation).
• Migration plan → P15 Round-5 design review.
• Cost arithmetic → P13 cost reviews, P00 tradeoff dials.

Read WARMUP, build the cost model, then P08: where the data actually lives — S3, HDFS, and the columnar formats (Parquet/ORC) that make scans cheap.

👨🏻 Brother Talk — Phase 07, Off the Record

The "legacy and ops" phase. Less glamorous than Flink, but it's where the cloud bill lives and where migrations are won or lost. Let me tell you what actually matters.

Brother, I know how this phase feels. After the intellectual high of Flink and Spark internals, "Hive metastore and EMR operations" sounds like a step backward — like you're being asked to learn the old stuff. And partly you are. But here's the reframe that made me take it seriously: you cannot migrate off a system you don't understand, and migrating legacy warehouses to the lakehouse is one of the highest-value things a principal does. The Round-5 interview is literally "lead the migration from Hive/Tez/Flume to Kafka/Flink/Iceberg." You can't lead that migration if Hive is a black box to you. So this phase isn't about loving the old stack — it's about understanding it well enough to retire it safely.

Let me give you the operational truths.

The EMR bill is your reputation. I mean it. In a lot of orgs, the data platform is one of the biggest line items in the cloud bill, and EMR clusters are a huge chunk of that. The engineer who can walk into a cost review and say "I found $40k/month: these clusters sit idle between hourly jobs so I moved them to Serverless, these task nodes can be spot for 70% off, and this query does a full scan because someone dropped the partition filter" — that engineer gets listened to, because they just found real money. Cost optimization has a scoreboard, and a scoreboard makes you visible. Learn the arithmetic in this lab; it's not academic, it's your next performance review.

The spot/core/task thing will bite someone on your team. Make sure it's not you. Here's the trap: spot instances are gloriously cheap, so the temptation is to put everything on spot. And then a reclamation takes out a core node — which was running HDFS and the application master — and the whole job dies, maybe with data loss. The rule is simple and you must never violate it: core nodes on-demand, task nodes on spot. Core nodes hold state (HDFS) and coordination (AM); losing them is catastrophic. Task nodes are disposable compute; losing one just reschedules work. And do the math from the lab — at 50 spot nodes with a 2% interruption rate, the probability of losing at least one is about 64%. You're not gambling on whether you'll get interrupted; you're guaranteed to. Design for it.

Partitioning is a double-edged sword, and people cut themselves on both edges. Partition your tables and queries can prune to the directories they need — massive cost and speed win. But partition by a high-cardinality column (user_id, session_id, anything with millions of values) and you create millions of tiny directories, the metastore chokes listing them, and you've built a slower system than no partitioning at all. The art is partitioning by something with the right cardinality — usually date, sometimes date + a low-cardinality category — that matches how queries actually filter. When you design a table, the question is "how will people filter this?" and you partition to match. Get this right and you're a hero; get it wrong and you're the cause of the metastore incident.

The DROP TABLE footgun. This one's almost a rite of passage, so let me inoculate you: managed tables, when dropped, delete the underlying data. External tables only drop the metadata. Someone, somewhere, on every team, eventually runs DROP TABLE on a managed table thinking it just removes the catalog entry, and deletes a chunk of the lake. Know which kind your tables are. For anything important, prefer external tables (or table formats with proper governance, P09/P14) so a careless drop is recoverable. The day you prevent this for your team is a day you've earned trust.

Now the migration wisdom, because it's the real prize of this phase. Migrating off Hive/Tez is not a rewrite-and-pray — it's a dual-run-and-validate. You stand the new lakehouse up alongside the old warehouse (they can even read the same S3 data through the catalog), you run both in parallel, you diff the outputs until they match for a soak period, then you cut over, and you keep the old path warm for rollback until you're sure. The engineers who botch migrations are the ones who flip the switch and hope. The ones who nail them treat it like schema evolution (P03) and backfills (P06): always reversible, always validated, never a big-bang. That patience is what makes a migration boring — and boring migrations are successful migrations. Nobody writes a postmortem about the migration that went smoothly because someone validated parity for two weeks first.

Career note: "I migrated us off the legacy Hadoop stack and cut the bill 40% while improving freshness" is a principal-defining sentence. It touches cost, reliability, and modernization all at once, and it requires exactly the cross-phase fluency this whole track is building. This unglamorous phase is where that sentence becomes possible.

Build the cost model. Internalize the spot/core/task rule and the partition-cardinality tradeoff. Then come to P08, where we go under all of this to the bytes themselves — S3 and the columnar formats (Parquet/ORC) whose internals decide whether a scan is cheap or ruinous.

— your brother 👨🏻

Lab 01 — EMR/YARN Cost & Scheduling Model

Phase: 07 — Hive, Tez, MapReduce & EMR | Difficulty: ⭐⭐⭐☆☆ | Time: 4–5 hours

Owning EMR means owning the cluster's cost and placement arithmetic. This lab builds it: YARN container packing (and which resource binds), EMR-on-EC2 cost with a spot mix, EMR Serverless pay-per-use, spot-interruption probability, the core/task node split that makes spot safe, and Hive partition pruning.

What you build

• yarn_plan — bin-pack tasks into containers across nodes; report whether memory or vcores is the binding resource (the "64 cores but memory caps me" lesson)
• cluster_cost_usd — EMR-on-EC2 cost with a spot fraction
• emr_serverless_cost_usd — pay per vCPU-hour + GB-hour (the spiky-workload win)
• spot_interruption_probability — 1 − (1−p)^n (why 50 spot nodes ≈ guaranteed loss)
• safe_spot_plan — on-demand core nodes (HDFS/AM) + spot task nodes
• partitions_scanned — Hive partition pruning (the cost/perf lever; full scan when no filter)

Key concepts

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

• All 14 tests pass.
• You can explain why spot_interruption_probability(50, 0.02) ≈ 0.64 and why that means "don't put HDFS or the driver on spot."
• You can decide EMR-on-EC2 vs EMR Serverless vs EMR-on-EKS for a stated workload.

Extensions

• Add an EMR-on-EKS vs EMR-on-EC2 vs Serverless total-cost comparator for a given job shape (steady vs spiky) and recommend with the deciding dial (P00).
• Model managed scaling: given a job's task curve over time, compute the node count and cost vs a fixed cluster.
• Add bootstrap + idle cost: include cluster spin-up time and idle minutes — the hidden cost that makes Serverless win for short, frequent jobs.

Phase 08 — Storage & Columnar Formats

Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 06 (small files), Phase 07 (partitions, EMRFS)

Why This Phase Exists

This is the byte level the JD demands you debug at: "Parquet internals: row groups, column chunks, pages, encodings, compression, statistics, predicate pushdown, dictionary encoding, and file sizing … Protobuf schema evolution … compare Avro, Protobuf, JSON, Parquet, and ORC … S3 consistency, object layout, request costs, prefix scaling, lifecycle policies, encryption, and cross-region replication." Everything above storage — query speed, scan cost, lakehouse performance — is determined by how the bytes are laid out here. The single most leveraged skill of the whole track is understanding why a columnar file with good stats and partitioning turns a 1 TB scan into a 10 GB one (a 100× cost difference).

Concepts

• S3 as the lake substrate: object storage (not a filesystem); strong read-after-write consistency (since 2020); prefix-based request scaling (~3,500 PUT/s, 5,500 GET/s per prefix); request costs; multipart upload; lifecycle policies (tiering to IA/Glacier); encryption (SSE-S3/KMS, P14); cross-region replication; the small-file & listing cost.
• HDFS (the on-prem ancestor): NameNode (metadata) + DataNodes (blocks), block size, replication factor — and why S3-as-lake replaced it (decoupled storage/compute, elasticity).
• Parquet internals: file → row groups → column chunks → pages; per-chunk statistics (min/max/null) enabling predicate pushdown; dictionary + RLE/ bit-packing encodings; compression (snappy/zstd/gzip); file sizing (128 MB–1 GB).
• ORC: stripes → row index → streams; built-in indexes + bloom filters; Hive-native; Parquet vs ORC tradeoffs.
• Avro (P03 revisited as storage): row-oriented, schema-carrying — great for ingestion/ write-heavy and schema evolution, poor for column-pruned analytics.
• Format selection: row vs columnar; Parquet/ORC (analytics) vs Avro (ingestion) vs JSON/ CSV (interchange) — costed, not by taste.
• Layout for skipping: sorting/clustering on filter columns to tighten row-group stats (the basis of Z-ordering, P09); partitioning vs the small-file/over-partitioning trap.

Labs

Lab 01 — Parquet Reader & Predicate Pushdown (flagship, implemented)

→ Lab folder: lab-01-parquet-reader/

Extension Project A — Real Parquet inspection (spec)

Write Parquet with pyarrow; inspect the footer (parquet-tools/pyarrow): row-group count, column stats, encodings, sizes. Sort on a column, rewrite, and observe tighter stats and better skipping. Measure bytes read for a projected, filtered query.

Extension Project B — Format & layout bake-off (spec)

Same dataset as Parquet/ORC/Avro/JSON: compare file size, write time, and a column-projected filtered read time. Vary row-group size and file size; chart the small-file penalty.

Integrated-Scenario Hooks

• This phase's pushdown + sorting is what Iceberg/Delta exploit (P09) and what makes Athena/Trino cheap (P10).
• Its small-file lens closes the loop with P06 (Spark writes) and P07 (Hive partitions).
• Its S3 layout (prefixes, lifecycle, encryption) feeds P14 governance and P16 capstone.

Guides in This Phase

• HITCHHIKERS-GUIDE.md · WARMUP.md · BROTHER-TALK.md

Key Takeaways

• The bytes' layout decides every downstream cost; this is the highest-leverage phase.
• Columnar = read only the columns and row groups you need; stats enable skipping.
• Stats prove absence, so sorting/clustering on filter columns is what makes skipping work.
• S3 is object storage with prefix scaling and request costs — small files hurt even idle.
• Pick formats by cost: Parquet/ORC for analytics, Avro for ingestion, JSON/CSV for interchange.

Deliverables Checklist

• Lab 01 implemented; all 14 tests pass
• You can draw Parquet's file→row-group→column-chunk→page hierarchy
• You can explain predicate pushdown and why sorting amplifies it
• You can state S3 prefix scaling limits and the small-file cost
• Extension A inspection or Extension B bake-off

Warmup Guide — Storage & Columnar Formats

Zero-to-principal primer for Phase 08: S3/HDFS as the lake substrate, Parquet and ORC internals (row groups, pages, stats, encodings), Avro vs columnar, and the data-layout decisions that determine every downstream query's speed and cost.

Table of Contents

• Chapter 1: S3 — Object Storage as a Lake
• Chapter 2: HDFS and Why S3 Won
• Chapter 3: Row vs Columnar
• Chapter 4: Parquet Internals
• Chapter 5: Encodings and Compression
• Chapter 6: Predicate Pushdown and Why Sorting Matters
• Chapter 7: ORC, Avro, JSON, CSV — Choosing a Format
• Chapter 8: File Sizing and the Small-File Problem
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: S3 — Object Storage as a Lake

S3 is the substrate of the modern lake, and it is object storage, not a filesystem — a distinction that explains many gotchas:

• Objects, not files: a flat key→bytes store. "Directories" are a fiction (key prefixes); there's no cheap rename (a rename is a copy + delete — which is why naive Spark/Hadoop commit protocols are slow/unsafe on S3 and why table formats, P09, exist).
• Consistency: since Dec 2020, S3 has strong read-after-write consistency for all operations — the old "eventual consistency" footguns (S3Guard, etc.) are gone. Know this is current; interviewers test whether you're up to date.
• Request scaling: throughput scales per prefix — roughly 3,500 PUT/s and 5,500 GET/s per prefix, and S3 auto-scales as you spread keys across prefixes. A million tiny files under one prefix → throttling; spreading prefixes (and fewer, larger files) → speed.
• Costs: storage ($/GB-month by tier), requests (PUT/GET per 1,000 — why small files cost money even idle), and data transfer (cross-region/egress). Lifecycle policies tier cold data to IA/Glacier automatically.
• Security (P14): SSE-S3 / SSE-KMS encryption at rest, bucket policies, VPC endpoints, cross-region replication for DR.

Chapter 2: HDFS and Why S3 Won

HDFS was the original lake storage: a NameNode holds the filesystem metadata (the directory tree, block→DataNode map) and DataNodes store fixed-size blocks (default 128 MB), replicated (default 3×) for durability. It's fast for colocated compute (data locality) but has two fatal-at-cloud-scale problems: the NameNode is a memory-bound single point (millions of small files = NameNode heap death — the small-file problem's origin) and storage is coupled to compute (scaling one means scaling the other).

S3 won the lake because it decouples storage from compute (scale independently; spin clusters up/down against durable storage), is effectively infinite and 11-nines durable, and is cheaper. The cost: no data locality (compute reads over the network) and no atomic rename (→ table formats). Know HDFS to understand what EMR/Spark inherited and what you're migrating from (P07/P15).

Chapter 3: Row vs Columnar

The fundamental storage choice:

• Row-oriented (CSV, JSON, Avro): all of row 1, then all of row 2… Great for writing whole records and reading whole rows (OLTP, ingestion). Bad for analytics: to sum one column you read every column of every row.
• Columnar (Parquet, ORC): all of column A, then all of column B… Great for analytics:
  • Projection: reading 2 of 50 columns reads ~4% of the bytes.
  • Compression: a column is homogeneous (all ints, all of one enum) → compresses far better than mixed rows.
  • Skipping: per-chunk stats let you skip data that can't match (Ch. 6). Analytics is overwhelmingly "aggregate a few columns over many rows," so columnar wins — which is why the lakehouse is built on Parquet/ORC.

Chapter 4: Parquet Internals

Parquet's hierarchy (the lab builds it):

File
└── Row Group (≈128 MB of rows — the unit of parallelism & skipping)
    └── Column Chunk (one column's data for this row group)
        └── Page (≈1 MB — the unit of encoding/compression/IO)
    + Column statistics per chunk: min, max, null_count, (distinct_count)
File footer: schema + row-group metadata + column stats + offsets

• A reader reads the footer first (metadata), then reads only the column chunks it needs (projection) from only the row groups whose stats can match (pushdown, Ch. 6) — often a tiny fraction of the file.
• Row-group size trades parallelism vs skipping granularity vs memory; ~128 MB is typical.
• Repetition/definition levels encode nested/optional fields (Dremel encoding) — how Parquet stores arrays/maps/structs and nulls compactly.

Chapter 5: Encodings and Compression

Two layers, and the order matters:

1. Encoding (logical, type-aware): dictionary encoding maps distinct values → small integer codes (huge win for low-cardinality columns — the lab's dictionary_encode), then RLE + bit-packing on the codes (huge win for runs/sorted data — the lab's rle_encode). Delta encoding for sorted integers/timestamps.
2. Compression (byte-level): snappy (fast, default), zstd (better ratio, now often preferred), gzip (smaller, slower). Applied to the already-encoded pages.

The lesson: columnar + dictionary + RLE is why a column compresses 10× where the same data as JSON rows barely compresses — and why sorting the data first (so runs form) multiplies the win. This is also a cost lever (fewer bytes = cheaper scans, P10).

Chapter 6: Predicate Pushdown and Why Sorting Matters

Predicate pushdown = the reader uses per-row-group statistics to skip groups that can't contain a matching row. For WHERE id = 55, a group with min=60, max=69 is skipped without reading its data. The lab's row_group_can_match is exactly this.

The crucial asymmetry: statistics prove ABSENCE, not presence. A group with min=50, max=59 might contain id=55, so it's scanned and row-filtered; a group with min=60 can't, so it's skipped. This is why pushdown's effectiveness depends entirely on how tight the per-group min/max ranges are — and that depends on data layout:

• If the data is sorted by id, each row group covers a tight, disjoint id range, so a point/range query skips almost everything. Massive win.
• If id is randomly distributed, every group's [min, max] spans the whole range, so nothing can be skipped. Pushdown does nothing.

This is the entire motivation for sorting / clustering / Z-ordering (P09): you lay data out so the columns people filter on have tight per-group stats. "Why is my Parquet query still slow despite pushdown?" is almost always "your data isn't sorted on the filter column."

Chapter 7: ORC, Avro, JSON, CSV — Choosing a Format

• Parquet vs ORC: both excellent columnar formats; ORC has historically stronger built-in indexes/bloom filters and is Hive-native; Parquet has the broader ecosystem. Pick by ecosystem; don't agonize.
• Avro's niche: row-oriented + schema-carrying makes it ideal for the ingestion side (Kafka, write-heavy, evolving schemas — P03) where you write whole records and rarely column-prune. The pattern: Avro on the wire/landing, Parquet on the lake.

Chapter 8: File Sizing and the Small-File Problem

The recurring villain, now at its source:

• Target file size: 128 MB – 1 GB. Too small → the small-file tax (metadata blowup, S3 request costs, listing slowness, poor scan throughput, HDFS NameNode pressure). Too large → poor read parallelism, memory pressure, slow single-file operations.
• Where small files come from: Spark writing one file per partition per task (P06), over-partitioning a Hive table (P07), streaming sinks committing frequently (P04), and per-record/per-minute micro-writes.
• Fixes: repartition/coalesce to target before writing (P06); compaction jobs (P09 table formats automate this); right-sized partitioning; batching streaming writes.
• The cost framing: small files inflate scan cost (Athena reads/opens more, $/TB up), request cost (more GETs), and operational cost (slow listing, metastore pressure) — so compaction is a cost program (a JD deliverable), not just tidiness.

Lab Walkthrough Guidance

Lab 01 — Parquet Reader & Predicate Pushdown, suggested order:

1. dictionary_encode / rle_encode / rle_decode — the encodings (and roundtrip).
2. _column_stats + build_parquet — chunk into row groups, compute min/max/null.
3. row_group_can_match — the pushdown logic; remember stats prove absence; all-null group skipped; != not prunable.
4. scan — skip via pushdown, then row-filter + project; report scanned/skipped.
5. Then the extension: sort the data on the predicate column, rebuild, and watch row_groups_skipped jump — Ch. 6 made tangible.