Warmup Guide — The Principal Data Infrastructure Engineer

Zero-to-principal primer for Phase 00: what the role actually is, the data lifecycle that organises the entire track, the five forces and the tradeoff dials you will spend your career turning, and the artifacts (latency budgets, cost models, ADRs) that turn opinion into engineering.

Table of Contents

• Chapter 1: Pipeline vs Platform vs Product
• Chapter 2: The Data Lifecycle — The Spine of the Track
• Chapter 3: The Five Forces
• Chapter 4: The Tradeoff Dials
• Chapter 5: Latency Budgets — Where the Milliseconds Go
• Chapter 6: Cost Models — Where the Dollars Go
• Chapter 7: Correctness as a First-Class Requirement
• Chapter 8: Blast Radius and Failure Domains
• Chapter 9: The ADR — Making Decisions Durable
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Pipeline vs Platform vs Product

From zero. Three things get called "data engineering," and conflating them is the single most common reason designs are over- or under-built:

• A data pipeline moves/transforms data from A to B. Success = "the job ran and the numbers look right." Blast radius = one dataset. A senior engineer owns pipelines.
• A data platform is the substrate other people build pipelines on: the ingestion SDK, the schema registry, the table-creation service, the orchestration framework, the observability that every pipeline inherits. Success = "hundreds of engineers ship correct pipelines without reinventing architecture." Blast radius = the whole org. A principal engineer owns platforms.
• A data product is a dataset/stream treated as a product: it has an owner, an SLA, a schema contract, documented semantics, a freshness guarantee, and consumers who depend on it. Success = "downstream teams trust it enough to build on it without asking."

The JD is unambiguous that this role is the middle one with strong opinions about the third: "building the data infrastructure layer itself … used by hundreds or thousands of engineers." Why does the distinction matter so much? Because the right answer changes with the noun. Exactly-once semantics in one pipeline is a code review; exactly-once as a platform guarantee every team inherits is an architecture program. Schema validation in one job is an if statement; schema governance as a platform is a registry, a compatibility checker, a CI gate, and an org-wide contract. You will be asked to build the second kind. Everything in this track is "build the platform version of X."

Chapter 2: The Data Lifecycle — The Spine of the Track

Every piece of data, in every system you will ever build, walks this path:

creation → ingestion → validation → transport → processing →
storage → query → serving → governance → replay → deletion

This is not a metaphor; it is the literal table of contents of the JD and of this track. Memorise it, because the principal skill is reasoning about the seams between stages — that's where production failures live:

The reason a principal can interview across all of Kafka and Spark and Iceberg and Cassandra is not that they memorised four manuals — it's that they hold one lifecycle and ask the same five questions (the next chapter) at every stage.

Chapter 3: The Five Forces

Every data-infrastructure decision is scored on five axes. There is no architecture that maxes all five; the art is knowing which the workload actually needs.

1. Correctness — does the output match reality? (no loss, no duplication, no skew, no silent corruption, schema-valid). The one force you can't trade away on critical data, only define precisely (exactly-once-effect, freshness SLA, quality policy).
2. Latency — how fast from event to usable result? (p50/p99; "real-time" is meaningless without a number and a percentile).
3. Throughput — how much per second, sustained and at peak? (events/s, MB/s, and the ratio of read to write).
4. Cost — $/event, $/TB stored, $/TB scanned, $/engineer-hour to operate.
5. Operability — how hard is it to run, debug, upgrade, and stay on-call for? (the force juniors ignore and principals obsess over, because it dominates total cost over a system's life).

The senior framing: "requirements" are not requirements until they are numbers on these five axes. "Make it fast and cheap and correct" is a wish. "p99 < 60s, 5M events/s peak, < $X/month, exactly-once into the lakehouse, runnable by a 4-person on-call" is a spec you can architect against — and the lab makes you compute it.

Chapter 4: The Tradeoff Dials

The five forces conflict in recurring, nameable ways. A principal names the dial, quantifies both ends with the workload's numbers, and chooses — out loud, in an ADR. The canonical dials (straight from the JD):

• Low latency ↔ low cost. Real-time Flink + Cassandra serving is fast and expensive; hourly Spark + Athena is cheap and slow. The number that decides it: what does a minute of staleness cost the business?
• Exactly-once ↔ operational simplicity. Transactional sinks and checkpoint barriers buy correctness and cost you debuggability and throughput. The number: what does one duplicate or one lost event cost? (a duplicated fraud alert vs a duplicated ad impression are not the same question).
• Flexible schema ↔ safe contract. Schemaless JSON ships fast and breaks consumers silently; strict Protobuf contracts slow producers and prevent 3 a.m. outages. The number: how many consumers, and how expensive is their breakage?
• Self-service ↔ governance. Letting teams create tables freely maximises velocity and guarantees a small-file/PII/lineage mess; gating everything is safe and becomes the bottleneck you were hired to remove. The principal answer is almost always "self-service with guardrails the platform enforces."
• Streaming freshness ↔ replay/batch correctness. The Lambda-vs-Kappa argument: maintain two code paths (fast+approximate, slow+correct) or one replayable streaming path. The number: can your streaming path be replayed deterministically? (if yes, Kappa; if no, you've bought Lambda's double maintenance).
• Cassandra serving speed ↔ lakehouse analytical flexibility. A serving store answers one query shape in single-digit ms; the lakehouse answers any query in seconds. You need both, wired safely (P11).
• Cloud-native lock-in ↔ open-source portability. Athena/Kinesis/Glue are fast to adopt and hard to leave; Trino/Kafka/Iceberg are portable and yours to operate.

Hold this: an interviewer is not testing whether you know Flink. They are testing whether, when they add "…and it must also be cheap," you can name which dial that turns and what it costs. That is the whole Round-1 signal.

Chapter 5: Latency Budgets — Where the Milliseconds Go

An end-to-end SLA (say "p99 event-to-queryable < 15 minutes") is meaningless until you decompose it across lifecycle stages, because that's how you find the stage that's eating your budget and the stage where spending more buys nothing:

ingest commit         ▏  50 ms   (acks=all to Kafka)
transport / consumer  ▏ 200 ms   (consumer poll + network)
stream processing     ▏   2 s    (Flink window + watermark delay)
write to S3/Iceberg   ▏  30 s    (batch the writes; small-file avoidance)
catalog commit/refresh▏  10 s    (snapshot commit + metadata)
query availability    ▏   --     (immediate once committed)
                      ───────────
budget consumed       ≈ 42 s     → 15-min SLA has huge headroom; spend it on cost, not speed

Two principal moves fall out of this:

1. Budget conservation — the sum of stage budgets must be ≤ the SLA; the lab enforces it (you can't allocate 110% of the budget).
2. Spend the slack. If the SLA is 15 min and you consume 42 s, the remaining 14 min is a cost lever: batch writes harder, use spot instances, compact lazily. Juniors optimise latency they don't need; principals convert unused latency into saved dollars.

Chapter 6: Cost Models — Where the Dollars Go

You cannot be the "cost efficiency" person the JD asks for without arithmetic. The ones that come up constantly (full derivations in later phases; here are the shapes):

• Athena/Trino scan cost = bytes scanned × $5/TB. This is why partitioning, columnar formats, and projection pruning are cost features, not just speed features. A query that scans 1 TB instead of 10 GB costs 100× — and the fix is layout, not hardware (P08–P10).
• S3 storage = GB-months × tier price; lifecycle policies move cold data to cheaper tiers. Plus request costs (PUT/GET per 1000) — which is why millions of small files cost money even when idle (P08).
• Compute = node-hours × instance price, where spot is up to ~90% cheaper but can be reclaimed in 2 minutes (P07). Cost = nodes × hours × $/hour × (1 − spot_discount).
• The dominant hidden cost is operability — engineer-hours on-call, debugging, and upgrading. A "cheaper" architecture that needs a dedicated team is not cheaper.

Cost attribution by producer team (a JD deliverable) is just tagging every byte and every node-hour with an owner and summing — boring, and the thing that actually changes behaviour, because teams optimise what they're billed for.

Chapter 7: Correctness as a First-Class Requirement

The JD repeats one idea more than any other: treat data pipelines like production software with explicit correctness guarantees. That means correctness is specified, tested, and monitored, not hoped for:

• Specified: "exactly-once-effect into the lakehouse," "99.99% of accepted events durably persisted," "no breaking schema change reaches prod." These are the JD's example SLOs — testable statements, not vibes.
• Tested: every lab in this track ships a property test. The flagship example (Phase 04): shuffled, duplicated delivery produces identical results to clean ordered delivery. If you can't write that test, you don't have correctness, you have luck.
• Monitored: freshness, volume, uniqueness, referential integrity, schema/semantic validity, cardinality drift (P13). The point is to find corruption before customers or executives do.

Correctness is the force you most often see traded away by accident — a processing-time aggregation here, a missing dedup there — and the failures are silent, which is why a principal builds the detection in (P13's silent-corruption chapter).

Chapter 8: Blast Radius and Failure Domains

A failure domain is the set of things that fail together (a partition, a broker, an AZ, a region, a shared metastore, a single hot key). Blast radius is how much breaks when one thing does. The first question in every incident is not "what's wrong with component X?" but "what shared resource explains all these alarms at once?"

This reframing is what lets a principal stay calm in a 30-alarm storm: 12 jobs failing that all read one Hive metastore is one incident (the metastore), not twelve. You design for blast radius (isolate tenants, bulkhead resources, cap fan-out) and you reason with it during incidents (P13). Phase 01 makes this rigorous; Phase 00 just plants the question.

Chapter 9: The ADR — Making Decisions Durable

A principal's decisions outlive their memory of them and must survive their absence. The Architecture Decision Record is the lightweight artifact that makes a decision reviewable, teachable, and reversible-on-purpose. The template you'll reuse all track:

# ADR-NNN: <short title>
Status: proposed | accepted | superseded by ADR-MMM
Date: YYYY-MM-DD

## Context
What forces are in play? What are the numbers (the five forces, the dials)?

## Decision
What we chose, stated as one sentence.

## Alternatives Considered
Each option, and the dial it turned the wrong way for *our* numbers.

## Consequences
What gets better, what gets worse, what we must now monitor, and what would
make us revisit this.

The discipline the ADR enforces is exactly Chapter 4's: name the dial, quantify both ends, choose, and write down what would change your mind. An architecture review (JD Round 5) is just a room full of people pressure-testing your ADRs.

Lab Walkthrough Guidance

Lab 01 — Tradeoff & Lifecycle Modeler, suggested order:

1. LifecycleStage enum and allocate_latency_budget(total, weights) — split a budget by weights and assert it sums to ≤ total (budget conservation; Ch. 5).
2. Cost functions: athena_scan_cost_usd(bytes) ($5/TB), s3_storage_cost_usd(...), spot_compute_cost_usd(...) (Ch. 6) — pure arithmetic, exact unit tests.
3. Sizing: kafka_partitions_needed(...), kinesis_shards_needed(...), quorum_is_strong(n, r, w) (r + w > n) — boundary tests (exactly equal is not strong).
4. classify_delivery_semantics(retries, idempotent, transactional) → at-most / at-least / exactly-once-effect (Ch. 7).
5. resolve_tradeoff(option_a, option_b, weights) — score two architectures on the five forces with workload weights; return the winner and the deciding force (explainable output, Ch. 3–4). Test the tie-break and the "weights change the winner" property.

Success Criteria

You are ready for Phase 01 when you can, from memory:

1. Define pipeline vs platform vs product and say which this role builds.
2. Recite the 11-stage lifecycle and name a technology and a seam failure for each.
3. List the five forces and explain why a "requirement" isn't one without numbers.
4. Name four tradeoff dials and, for each, the single number that decides it.
5. Decompose a 15-minute end-to-end SLA into a stage budget and identify the slack.
6. Compute an Athena scan cost and explain why layout (not hardware) is the fix.
7. Write a one-page ADR for a real decision.

Interview Q&A

Q: "Design a system to ingest 5M events/sec." Where do you start? Not with Kafka. I start by turning the prompt into numbers on the five forces: what's the read throughput vs the 5M write (the ratio drives the storage/serving split); what's the latency SLA and at what percentile; what correctness guarantee (an ad impression and a payment are different); what's the budget; and who operates it. Then I decompose the lifecycle and a latency budget, and only then do tools get chosen — each as an ADR that names the dial it turns. The 5M number alone tells me almost nothing; the ratios and the SLAs tell me everything.

Q: A team wants exactly-once everywhere. Your response? First, "exactly-once" is exactly-once-effect, re-earned at every external boundary — so "everywhere" is a much bigger bill than they think. Second, I'd ask for the cost of a duplicate and the cost of a loss per stream, because most streams don't need it: an analytics impression tolerates at-least-once with dedup at read; a balance transfer does not. Exactly-once is a dial I turn per data product, sized by what an error costs — not a platform-wide default that taxes throughput and operability for streams that never needed it.

Q: How do you make a "make it cheaper" mandate concrete? Attribute cost first — tag every byte stored, byte scanned, and node-hour to an owner, so the bill is a map. Then the levers are mechanical: scan cost falls with partitioning + columnar + projection (often 10–100×); storage falls with compaction (kills the small-file request tax) and lifecycle tiering; compute falls with spot + right-sizing + AQE. Crucially I look for unused latency budget and spend it on cost — most "expensive" pipelines are optimising a freshness no one asked for.

Q: What's the difference between a senior and a principal data engineer, in one sentence? A senior makes one pipeline correct and fast; a principal makes the platform that makes every team's pipelines correct and fast by default, and can defend why with numbers in an ADR.

References

• Martin Kleppmann, Designing Data-Intensive Applications — the whole book is this role's bible; Ch. 1 (reliability/scalability/maintainability) is this phase
• Jay Kreps, The Log: What every software engineer should know (2013) — the lifecycle's unifying abstraction
• Google SRE Book — Service Level Objectives chapter (SLO/SLI/error-budget thinking applied to data in P13)
• Michael Nygard, Documenting Architecture Decisions (2011) — the ADR pattern
• The track's own CHEATSHEET.md and GLOSSARY.md