Phase 00 — The Principal Data Infrastructure Engineer
Difficulty: ⭐⭐☆☆☆ (conceptual) → the judgment is ⭐⭐⭐⭐⭐ Estimated Time: 1 week (12–18 hours) Prerequisites: none — this is the orientation that makes the other 16 phases cohere
Why This Phase Exists
Before you touch Kafka or Flink or Iceberg, you have to understand what job you are actually being hired to do — because at the principal level the job is not "operate tools," it is "make correct tradeoffs across the entire data lifecycle when the requirements conflict, and encode those tradeoffs into platforms other people build on."
The JD says it directly: "This role is not focused on writing ordinary ETL jobs. It is focused on building the data infrastructure layer itself." And: "The hardest part is not knowing the tools. The hardest part is making correct tradeoffs when the requirements conflict."
So Phase 00 builds the mental model and the calculator for those tradeoffs. Every later phase plugs a real technology into a slot in the model you build here. If you skip this phase you will learn 16 tools and still not be able to architect — which is exactly the failure mode that keeps strong engineers stuck at senior.
Concepts
- Data pipeline vs data platform vs data product — three different things people call "data engineering," with three different success metrics and blast radii.
- The data lifecycle:
creation → ingestion → validation → transport → processing → storage → query → serving → governance → replay → deletion. Every technology in this track lives at one or two of these stages; principals reason about the seams. - The five forces: correctness, latency, throughput, cost, operability — and how every architecture decision trades them against each other.
- The tradeoff dials (low-latency↔low-cost, exactly-once↔simplicity, flexible schema↔safe contract, self-service↔governance, freshness↔replay-correctness, …) — and how to quantify both ends with the workload's numbers instead of arguing taste.
- Latency budgets — decomposing an end-to-end SLA across lifecycle stages so you know where the milliseconds (and dollars) actually go.
- Blast radius / failure domains — the unit of thinking that turns a 30-alarm storm into one root cause (developed fully in Phase 01 & 13).
- The ADR (Architecture Decision Record) — how principals make a decision durable and reviewable instead of tribal knowledge.
Labs
Lab 01 — Tradeoff & Lifecycle Modeler (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build the calculator behind Round-1 architecture interviews: allocate an end-to-end latency budget across lifecycle stages, compute Athena scan cost and storage cost, size Kafka partitions / Kinesis shards, check a quorum for strong consistency, classify delivery semantics, and run a weighted tradeoff resolver that picks between two architectures given the workload's requirements |
| Concepts | The data lifecycle as a typed pipeline; latency budgeting; cost models; the five forces as a scoring function; "name the dial, quantify both ends, then choose" |
| Steps | 1. Lifecycle stage model + budget allocation; 2. cost functions (Athena $5/TB, S3 storage, request scaling); 3. sizing (partitions, shards, quorum); 4. delivery-semantics classifier; 5. weighted tradeoff resolver with explainable output |
| How to Test | pytest test_lab.py -v — covers budget conservation, cost math, sizing edge cases, quorum boundaries, and resolver tie-breaks |
| Talking Points | "Where does the latency go?" "What does exactly-once cost you operationally?" "Defend Kinesis over Kafka with numbers." |
| Resume bullet | Built a decision-support model that allocates end-to-end latency budgets and quantifies cost/correctness/operability tradeoffs to drive platform architecture choices |
→ Lab folder: lab-01-tradeoff-modeler/
Integrated-Scenario Suggestions (carried through the whole track)
These five scenarios (the JD's "Example Hard Problems") are the spine of the capstone (Phase 16). Keep them in mind from now — each phase contributes a real component:
- Global real-time event ingestion — billions of events/day, multi-region, Protobuf, schema validation at ingress, DLQ, Flink, S3/Iceberg, Athena, cost attribution by team.
- Stateful fraud/risk stream processing — event-time, out-of-order, sliding+session windows, Cassandra enrichment, broadcast rules, RocksDB, savepoint upgrades, replay.
- Lakehouse migration — Hive/Tez/Flume → Kafka/Flink/Spark/Iceberg/Athena, small-file elimination, compaction, dual-write, validation, cutover, rollback.
- Cassandra serving layer — low-latency derived features, partition design, TTL, compaction, consistency, streaming upserts from Flink, Spark backfill, multi-region.
- Enterprise governance platform — dataset/schema registry, field ownership, PII, lineage, contract testing, Lake Formation/Glue/IAM integration, breaking-change prevention.
Guides in This Phase
- HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
- WARMUP.md — the full primer; read slowly
- BROTHER-TALK.md — the candid version
Key Takeaways
- The job is tradeoffs across a lifecycle, encoded into platforms, defended in ADRs — not pipelines.
- Every later technology fits a lifecycle slot; learn the slot first, then the tool.
- "Make it fast / cheap / correct" is not a requirement until you attach numbers; Phase 00's lab is where you learn to attach them.
Deliverables Checklist
-
Lab 01 implemented; all tests pass against
solution.py - You can draw the 11-stage lifecycle from memory and name the tech at each stage
- You can take any "design X" prompt and immediately list the dials it forces
- You can write a one-page ADR for a real decision (template in WARMUP)