« All Roles

Principal / Distinguished Data Infrastructure Engineer — Curriculum

Target Role: Principal / Distinguished Data Infrastructure Engineer — Real-Time Distributed Data Platform, Streaming Systems, and Lakehouse Architecture (the reference JD — jd.md)

Also prepares you for:

• Netflix / Uber / Stripe / Databricks / Confluent — Staff/Principal Data Platform or Streaming Infrastructure Engineer
• Amazon / AWS — Principal Engineer, EMR / Kinesis / Glue / Lake Formation
• Any enterprise building a petabyte-scale, multi-region, real-time + batch data platform that hundreds or thousands of engineers depend on

Duration: 34 weeks core (≈8 months) — extendable to 12+ months with full capstones Seniority Target: Principal / Distinguished IC (the person who is the escalation point for the hardest failures and defines platform standards for the org) Goal: Make you the engineer who can architect the data infrastructure layer at the system level and debug it at the byte, partition, checkpoint, offset, file, and query-plan level — across streaming, batch, lakehouse, serving, governance, and the Scala/JVM platform code that ties it together.

Why This Curriculum Exists

This is not a "write ETL jobs" role and this is not a "learn one tool" course. The JD is explicit: you build the data infrastructure layer itself — the streaming substrate, the lakehouse architecture, the schema-governance system, the reliability controls, the developer SDKs, the observability, and the operational playbooks that hundreds or thousands of engineers, analysts, data scientists, and product teams stand on.

Most engineers specialise in one or two of: distributed systems, stream processing, batch processing, cloud infrastructure, functional programming, data modeling, query optimisation, data reliability, schema evolution, storage formats, operability, security, governance, cost, and developer-platform design. This role requires practical fluency across all of them — and, harder, the judgment to make the right tradeoff when they conflict (low latency vs low cost; exactly-once vs operational simplicity; flexible schemas vs safe contracts; self-service vs governance).

So this track is built around two non-negotiables:

1. Every technology in the JD gets implemented, not just described. Watermarks, exactly-once sinks, Parquet row-group skipping, Iceberg snapshot isolation, Spark's shuffle, Cassandra's tombstone math, a Cats-style Validated contract engine, IAM least-privilege evaluation — you build a working, tested miniature of each so the concept becomes muscle memory instead of folklore.
2. Everything is taught to the level interviewers actually probe — the numbers, the failure modes, the corner cases, the production tradeoffs. We assume you are aiming for the platform that serves millions of users with correctness guarantees, not a hobby pipeline.

How Each Phase Teaches (the "many ways of explaining")

Every phase deliberately explains the same material through several lenses, because principal-level understanding is the intersection of all of them:

When the JD demands the JVM (Flink, Scala/Cats/ZIO/FS2/Akka), the phase additionally ships a real, sbt test-verified Scala lab. Scala is first-class here, not an afterthought: it is the JVM language of Spark/Flink/Kafka-Streams and is in heavy demand for data-platform roles. Six runnable Scala labs + four runnable capstones (72 ScalaTest specs, all green via sbt test; JDK + sbt required):

What You Will Build

Every phase ships a tested, runnable artifact. By the end you will have hand-built a miniature of essentially every component of a modern data platform:

1. A failure-domain & exactly-once reasoning kit — partitioners, idempotency keys, a two-generals demonstration, and a dedup/outbox engine
2. An ingestion gateway — partition assignment, consumer-group rebalancing, offset management, retention/compaction, DLQ + poison-message quarantine, replay-from-offset
3. A schema-registry & data-contract engine — Protobuf/Avro/JSON compatibility checking (backward/forward/full), a Validated-style accumulating validator, CI gate
4. A Flink-grade stream processor — event-time windows, watermarks, allowed lateness, keyed state, checkpoint/savepoint + restore, exactly-once sink protocol
5. Streaming joins & CDC — stream-table enrichment, sessionization, a CDC apply loop with idempotent upserts
6. A Spark execution simulator — DAG → stages → tasks, shuffle + skew detection, AQE-style coalescing, broadcast-join planning, small-file analysis
7. An EMR/YARN cost & scheduling model — instance-fleet + spot-interruption planner, YARN container packing, EMR-Serverless vs EKS cost calculator
8. Parquet & ORC internals — a row-group/page reader with predicate pushdown and dictionary/RLE decoding from first principles
9. An Iceberg-style table format — snapshots, manifest lists, snapshot isolation, compaction, time travel, partition evolution, and conflict resolution on commit
10. A query optimiser — predicate pushdown, partition pruning, join reordering, a cost-based plan chooser, and an Athena scan-cost estimator
11. A Cassandra/DynamoDB data modeler — partition-key design, tombstone & TTL math, compaction-strategy chooser, consistency-level (R+W>N) calculator, hot-partition detector
12. A typed functional data SDK — Resource (bracket) safety, Validated/EitherT error modeling, an effect-system mini-runtime, FS2-style streaming, retry semantics
13. An orchestration & reliability core — a DAG scheduler with content-hash caching, SLO/error-budget tracker, freshness/volume/uniqueness data-quality checks, lineage graph
14. A governance & security kit — an IAM policy evaluator (least privilege), PII classifier + access policy, Lake-Formation-style column/row masking, audit log
15. Capstone platforms — the JD's five hard problems, composed end-to-end

Folder Structure

data-engineer/
├── README.md            ← you are here
├── jd.md                ← the target role profile (the source of truth for scope)
├── GLOSSARY.md          ← every term in the JD, defined precisely
├── CHEATSHEET.md        ← the numbers, formulas, and decision rules in one place
├── SUMMARY.md           ← mdBook table of contents
├── phase-00-principal-data-infra-engineer/
├── phase-01-distributed-systems-foundations/
├── phase-02-event-ingestion-messaging/
├── phase-03-serialization-schema-contracts/
├── phase-04-flink-stream-processing/
├── phase-05-streaming-alternatives-cdc/
├── phase-06-spark-internals/
├── phase-07-hive-tez-mapreduce-emr/
├── phase-08-storage-columnar-formats/
├── phase-09-lakehouse-table-formats/
├── phase-10-query-analytics-engines/
├── phase-11-serving-cassandra-dynamodb/
├── phase-12-scala-functional-data-engineering/
├── phase-13-orchestration-reliability-observability/
├── phase-14-security-privacy-governance/
├── phase-15-principal-architecture-strategy/
├── phase-16-capstone-systems/
├── interview-prep/      ← the JD's 5-round evaluation loop + per-domain Q&A banks
└── system-design/       ← full walkthroughs of the JD's example hard problems

Each phase contains README.md, WARMUP.md, HITCHHIKERS-GUIDE.md, BROTHER-TALK.md, and at least one lab-NN-*/ (README.md, lab.py, solution.py, test_lab.py, requirements.txt) where all tests pass against the solution.

34-Week Schedule

Prerequisites

• Strong programming — Python 3.10+ (the labs run in Python) and a willingness to read and write Scala (Phase 12 teaches it to platform-author level; you do not need prior Scala, but you need to not fear the JVM)
• SQL fluency — you will optimise queries at the plan level
• Comfort with a terminal, git, and Docker — the extension projects spin up real Kafka/Flink/Trino/Cassandra in containers
• No paid cloud account required — every flagship lab runs locally and simulates the distributed component; the concepts are cloud-portable by design. AWS-specific phases (07, 14) teach the model and provide Terraform you can read and adapt.

Interview Relevance

This track maps directly onto the JD's own five-round evaluation loop:

• Round 1 — Distributed data systems design ("ingest 5M events/sec → Flink → S3/Iceberg → Athena → Cassandra") → phases 01–04, 08–11 + system-design/
• Round 2 — Deep debugging (a Flink job with rising checkpoint duration, growing RocksDB state, delayed watermarks, growing lag) → phases 04, 06, 13
• Round 3 — Spark & lakehouse optimisation (a 9-hour job spilling and producing millions of small files) → phases 06–10
• Round 4 — Scala platform engineering (a type-safe contracts + stream-processing SDK) → phases 03, 12
• Round 5 — Architecture leadership (migrate Hive/Tez/Flume → Kafka/Flink/Iceberg) → phases 07, 09, 15

Start with Phase 00, then read each phase's HITCHHIKERS-GUIDE.md (fast), then its WARMUP.md (slow), then build the lab before moving on. The lab is where folklore becomes knowledge.