Warmup Guide — Principal Architecture & Strategy

Zero-to-principal primer for Phase 15: how a principal decides (ADRs, decision matrices), build-vs-buy, tool selection as ADRs, safe migrations, multi-region/DR strategy, reference architectures, design reviews, and technical leadership.

Table of Contents

Chapter 1: The Decision Discipline and the ADR

A principal's primary output is decisions — and a decision isn't done when you've chosen, it's done when you've made the choice legible and durable. The discipline (P00, here formalized):

  1. Frame it as a dial (P00): name the conflicting forces (latency↔cost, exactly-once↔ simplicity, …).
  2. Quantify both ends with the workload's numbers (the five forces — P00).
  3. Score the options on weighted factors (the lab's decision matrix) and identify the deciding factor — the one that actually tipped it.
  4. Write the ADR: Context (the forces + numbers), Decision (one sentence), Alternatives (each, and the dial it turned the wrong way for us), Consequences (what gets better/worse, what to monitor, what would make us revisit).

Why the ADR matters: six months later "why did we do it this way?" is answered by a document, not by memory and ego; a new team member onboards on the reasoning, not just the result; and a decision can be reversed on purpose when its assumptions change (the "what would make us revisit" line). An architecture review (Ch. 7) is a room pressure-testing your ADRs.

Chapter 2: Build vs Buy

The recurring strategic question. The framework:

  • TCO over a horizon (the lab's tco): Build has high upfront (engineering time) and low run cost; Buy/adopt-OSS-managed has low upfront and higher recurring cost. There's a crossover month — below it Buy wins on cost, above it Build might. But cost is only half.
  • Non-cost factors: control (can you fix/extend it?), lock-in (cost to leave), time-to-value (Buy ships this quarter; Build ships next year), team skill (can you operate what you build?), and the decisive one — is it your differentiator?
  • The principal heuristic: build what differentiates you; buy/adopt everything else. Your org's edge is its product, not its message bus — so adopt Kafka, don't write one; adopt Iceberg, don't invent a table format. Build the thin platform layer on top that encodes your standards (the SDK, P12) — that's the differentiator worth owning.

Chapter 3: Tool Selection as ADRs

The JD asks you to "decide when to use Spark, Flink, Hive, Athena, Trino, Cassandra, Kinesis, Kafka, or EMR." Each is a weighted decision (the lab's recommend), and you've built the inputs across the track:

  • Kafka vs Kinesis vs Pulsar (P02): ops vs control vs elasticity/lock-in.
  • Flink vs Spark-SS vs Kafka-Streams vs Akka (P04/P05): latency vs ops vs ecosystem.
  • Iceberg vs Delta vs Hudi (P09): openness vs Spark-features vs upsert pattern.
  • Athena vs Trino vs Spark-SQL (P10): serverless/ad-hoc vs steady-concurrency vs one-stack.
  • Cassandra vs DynamoDB (P11): open/ops vs managed.
  • EMR-on-EC2 vs Serverless vs EKS (P07): steady vs spiky vs K8s-native.

The skill is not memorizing "use X" — it's running the matrix on this workload's weights and naming the deciding factor. The same two tools flip winners under different weights (the lab's test_weights_change_winner), which is why "best tool" is meaningless without the context.

Chapter 4: Migrations Done Safely

The JD's Round 5 and Problem 3: migrate Hive/Tez/Flume → Kafka/Flink/Spark/Iceberg/Athena. The only safe migration pattern (the lab's migration_gate_check):

  1. Dual-write: run old and new in parallel; the new path writes alongside the old without anyone depending on it yet.
  2. Validate parity: compare old vs new outputs continuously (data diffing, audit counts — P13) until they match for a soak period. Never cut over before parity holds.
  3. Cutover: move consumers to the new path — incrementally (a few first), with the old path still warm.
  4. Rollback-ready: at every step, you can revert to the old path instantly. A migration without a rollback is a gamble.
  5. Decommission: only after a soak period on the new path with no issues — and it's the last step, never before cutover is proven.

This is the strangler-fig pattern: grow the new system around the old, shift load gradually, retire the old once it carries nothing. It's the same discipline as schema evolution (P03), backfills (P06), and table-format adoption (P09): incremental, validated, reversible, never big-bang. The big-bang "flip the switch and pray" migration is how careers and quarters get lost.

Chapter 5: Multi-Region and DR Strategy

(Builds on P14 Ch. 8.) The strategic choices:

  • Why: regional outages, data residency, latency for global users.
  • RPO / RTO: the two numbers that define the design and its cost — how much data you can lose, how fast you recover. They're a P00 dial (more nines = more money).
  • Active-passive (standby region, failover): cheaper, simpler, slower RTO, possible data loss to RPO. Active-active (both serve): instant RTO, but you must handle multi-region writes (conflict resolution — P01 vector clocks/LWW, Cassandra LOCAL_QUORUM per DC) and pay for double capacity.
  • The decision is an ADR weighing cost vs RTO/RPO vs consistency vs operational complexity — and DR is only real if game-day tested (P14).

Chapter 6: Reference Architectures and Standards

A principal scales themselves by encoding decisions into standards so teams don't re-decide (or re-mistake) every time:

  • Reference architectures: the blessed shape for common needs (the standard ingestion pipeline, the lakehouse table layout, the streaming-job template) — documented, with the ADRs behind them.
  • Standards: streaming standards (event-time + idempotency + DLQ — P01/P02/P04), lakehouse standards (Iceberg + compaction + partitioning — P08/P09), contract standards (P03), table ownership/SLA/lineage (P13). These are the platform-as-product (P00): self-service with guardrails.
  • The payoff: "engineers can create new pipelines without reinventing architecture" (a JD success metric). Standards are how one principal's judgment reaches a thousand engineers.

Chapter 7: Design Reviews and Presenting to Leadership

  • Running a design review: it's pressure-testing ADRs. Ask for the dials and the numbers; probe the failure modes, the blast radius, the rollback, the cost; check the deciding factor holds under a weight change (sensitivity). The goal is a better, defensible decision, not a winner of an argument.
  • Presenting to leadership: translate to their language — cost, risk, time, and reliability, not Flink internals. "This migration saves $X/month and cuts incident rate Y%, takes Z quarters, with this rollback if it goes wrong" — a sentence an exec can decide on. The JD: "present tradeoffs to senior engineering leadership" and "translate business requirements into platform capabilities."

Chapter 8: Technical Leadership

The seniority-bar items (the JD's list):

  • Influence without authority: you rarely command; you persuade with ADRs, prototypes (build the risky bit to de-risk it), and earned trust.
  • Mentor senior engineers / raise the bar: review designs, teach the reasoning (not just the answer), set the example of writing things down.
  • Simplify without making fragile: the hardest skill — remove complexity while keeping the guarantees. (A simpler system you can operate beats a clever one you can't — P00 operability.)
  • Operate under ambiguity: turn vague fear ("will it scale?") into specific, numbered questions (P00) — calm comes from arithmetic.
  • Balance correctness, latency, cost, operability: the whole track, every day, as tradeoffs (P00).

Lab Walkthrough Guidance

Lab 01 — Architecture Decision Toolkit, suggested order:

  1. weighted_score / rank_options / recommend — and confirm test_weights_change_winner: the same options flip under different weights (the core lesson).
  2. tco — Build-vs-Buy economics.
  3. migration_gate_check — required gates + ordering (validate before cutover; decommission last).
  4. dr_meets_targets — RPO/RTO.
  5. render_adr — the durable artifact.

Success Criteria

You are ready for Phase 16 when you can, from memory:

  1. State the decision discipline and the ADR structure (incl. "what would reverse it").
  2. Apply build-vs-buy (TCO + non-cost factors + "build only your differentiator").
  3. Turn a tool choice into a weighted ADR with a deciding factor.
  4. Lay out a safe migration (dual-write → validate → cutover → decommission, reversible).
  5. Choose active-active vs active-passive with RPO/RTO.
  6. Explain how standards/reference architectures scale a principal's judgment.
  7. Present a tradeoff to leadership in cost/risk/time/reliability terms.

Interview Q&A

Q (Round 5): Lead the migration from Hive/Tez/Flume to Kafka/Flink/Spark/Iceberg/Athena. I'd run it as a strangler-fig, never a big-bang. Phase 0: stand up the new stack alongside the old, reading the same S3 data through the catalog where possible (P07/P09). Phase 1: dual-write — new producers (Kafka/Flink) write to Iceberg tables while Flume/Hive keep running and owning truth. Phase 2: validate parity continuously — data diffing and audit counts (P13) comparing new Iceberg outputs to the legacy warehouse until they match for a soak period; no cutover before parity. Phase 3: cut over consumers (Athena/Trino) incrementally to the new tables, old path still warm for instant rollback. Phase 4: decommission Flume/Hive/Tez only after a clean soak. Cross-cutting: governance (P14) and contracts (P03) ride along, cost is attributed (P07), and each phase has entry/exit criteria. The whole thing is reversible at every step — that's what makes a multi-quarter migration survivable.

Q: How do you decide build vs buy for a message bus? Buy/adopt — almost always. A message bus is not your differentiator; Kafka/Kinesis/Pulsar are mature, and the TCO of building and operating one dwarfs adopting one (the upfront engineering plus the perpetual operational burden). I'd reserve "build" for the thin layer that encodes our standards on top — the ingestion SDK with our contracts, DLQ, and observability baked in (P12) — because that is differentiating and cheap to own. The general heuristic: build what makes you uniquely better, adopt everything commoditized, and be honest that "we could build it" is rarely the same as "we should."

Q: Two senior engineers disagree on Flink vs Spark Structured Streaming. How do you resolve it? Not by authority or by who argues hardest — by the workload's numbers. I'd have us write the decision as a matrix: latency SLO, state size, exactly-once needs, existing stack/skills, ops budget, ecosystem. Score both honestly, weight by this use case, and the deciding factor falls out — and crucially, the same two engines flip winners under different weights, so the disagreement usually evaporates once we agree on the weights (which is the real question). Then we write the ADR with the deciding factor and what would make us revisit. The output isn't "I'm right," it's a documented decision the team owns and can revisit if the assumptions change.

Q: What makes a principal different from a staff engineer here? Scope of judgment and how it's encoded. A staff engineer makes excellent decisions for their systems; a principal sets the standards and reference architectures that make hundreds of engineers' decisions good by default, leads the cross-team migrations, and translates between business and platform. The artifacts are different too — a principal's work is ADRs, design reviews, mentoring, and the platform-as-product, not just shipped components. And the hardest principal skill is simplification: removing complexity while preserving the guarantees, because operability over a system's multi-year life dominates everything (P00).

References

  • Michael Nygard, Documenting Architecture Decisions (the ADR origin) ; adr.github.io
  • Martin Fowler, StranglerFigApplication and Patterns of Distributed Systems
  • Staff Engineer (Will Larson) and The Staff Engineer's Path (Tanya Reilly) — the role
  • Google SRE — DR, RPO/RTO ; AWS Well-Architected Framework — the decision pillars
  • Every prior phase's "decision" sections — this phase decides between them