🛸 Hitchhiker's Guide — Phase 15: Principal Architecture, FinOps & Capstone

Read this if: you've built all fourteen engines and "now architect the whole platform" still feels like a different job. It is — it's composition. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the gate.


0. The 30-second mental model

You are the person who, when a design lands on the table, runs the same five-step review every time — workload facts → two planes → quantify → score & trade → fail it — and writes the ADR. And you are the gate: the deploy request that wants into your platform clears identity, authorization, and policy or it doesn't ship. One sentence to tattoo: a principal doesn't build a fifteenth engine — they compose the fourteen, score the composition with numbers, and admit what gets deployed.

1. The review, one breath

1. WORKLOAD FACTS  (RPS, p99 SLO, availability target, residency, compliance, team size)
2. TWO PLANES      (control: MG/RBAC/policy/IaC/secrets  ·  data: net/compute/events/storage)
3. QUANTIFY        (compose SLA ∏Aᵢ & 1−∏(1−Aᵢ) · $/month · sizing · retry budget)
4. SCORE & TRADE   (five pillars from evidence · name the pillar you trade · the number)
5. FAIL IT         (blast radius · throttling · DR failover · rollback)
→ WRITE THE ADR

No numbers in step 1 = no review. Skip step 2's control plane = the #1 panel anti-pattern.

2. The five pillars (score everything on these)

PillarThe questionScored from
Reliabilitystays up & recovers?composite SLA → nines
Securityidentity-perimeter, secrets safe, private?posture (MI/private/KV/CMK)
Costevery dollar buys something needed?the bill vs reference
Operational Excellencecan a small on-call run it?less to rotate/leak/watch
Performancemeets the SLO at scale?redundancy + private fast-path

Same two designs flip winners when the weights change. "Best architecture" is a category error.

3. The numbers you'll actually use

ThingNumber
Three serial 99.9%99.7% (∏ — ~3× the downtime)
Two redundant 99%99.99% (1−∏(1−A))
99.9% downtime~43 min/month; 99.99% → ~4.3; 99.999% → ~26 s
Nines\( -\log_{10}(1-A) \): 99%→2, 99.9%→3, 99.99%→4
Internet egress~\$0.087/GB (the multi-region ambush)
Hot vs Archive blob~\$0.018 vs ~\$0.001 /GB-month (~18×)
Reservation discount~30–60% off (commit the baseline)
Spot discountup to ~90%, 2-min reclaim
Dev/test auto-shutdown~76% off (nights+weekends)
Scale-to-zero~20× off a spiky always-on VM (cold-start cost)

4. The FinOps levers (and the number that decides each)

right-size            → p95 utilization (NOT the average)
reservations/savings  → break-even on the commitment term
autoscale/scale-to-0  → does p99 SLO tolerate the cold start?
storage lifecycle     → when will you next read it? (Archive = hours to rehydrate)
egress avoidance      → cross-region GB in the hot path
dev/test shutdown     → idle hours per week

The principal habit from P00: find unused headroom, spend it on cost. Most "expensive" platforms over-provision a requirement nobody wrote down.

5. DR in one table (state RPO + RTO before you design)

backup/restore   RPO hours   RTO hours    cheapest, storage only
active-passive   RPO min     RTO min      warm 2nd region, async repl, FD failover
active-active    RPO ~0      RTO sec       both serve, ~2× compute + heavy egress

RPO = data you can lose. RTO = time you can be down. Test the failover — an untested DR plan is a hope. Region pairing ≠ failover; you still design it.

6. The reference architecture (the whole track)

Front Door (WAF) → APIM (JWT aud/iss/scp, rate-limit) → Functions/Durable
   → Service Bus/Event Grid (peek-lock, DLQ, retry) → Cosmos via Private Endpoint
secrets → Key Vault (RBAC, purge protect) · auth → Managed Identity (NO secret)
under: landing-zone policy baseline · observed by Azure Monitor/KQL · SLA-composed redundancy

Serial SLA ≈ 99.88% — below a 99.95% target. Add redundancy on the weak link or relax the SLO; show the math, then decide. No "exactly-once delivery" claim, ever.

7. The admission gate (what the lab builds)

admit():  identity(401) → authorization(403, deny-wins) → policy(403, deny) → SCORE → ADR
  • Auth before authz: bad-token-AND-unauthorized = 401, not 403.
  • Gate vs advice: a deny policy rejects; a low security score doesn't — it's reviewer advice.
  • Enforce the floor as code, advise on the rest, let teams self-serve inside the guardrails. That's platform engineering.

8. War story shapes you'll relive

  • "Our SLA says 99.95% but we're down 40 min/month." → nobody composed the chain; four-nines parts in series give three nines. Show ∏Aᵢ; add redundancy on the weak link.
  • "The bill doubled when we went multi-region." → egress. Nobody estimated the cross-region replication + user-download GB at \$0.087. Cache at the edge, keep compute by data.
  • "We failed over and lost two hours of orders." → RPO was async and nobody stated it. The DR plan was never game-day'd.
  • "A leaked app-settings file became a breach." → a stored secret + public endpoint. Managed Identity + Private Endpoint would have meant nothing to leak.
  • "Owner can't read the blob." → control plane ≠ data plane; Owner has no DataActions. The oldest confusion in the track, still biting in prod.

9. Vocabulary that signals you've run the review

  • Composition — units interact, they don't add; the principal sees the seams.
  • Gate vs advice — non-negotiables as deny; everything else scored, not blocked.
  • Fail closed / deny by default — the security posture of every gate.
  • Confused deputy — a token for service B accepted at service A (wrong aud).
  • RPO / RTO — data-loss window vs time-to-recover; both, before designing DR.
  • Exactly-once-effect — at-least-once + idempotent consumer; the only one that exists.
  • Self-service with guardrails — the only scalable answer to the governance dial.
  • The ADR — the durable record of a decision and what would reverse it.

10. Beginner mistakes that mark you in the final round

  1. Designing only the data plane; forgetting identity/RBAC/policy/landing zone.
  2. Quoting an SLA without composing the dependency chain.
  3. Claiming "exactly-once delivery" (it doesn't exist across a network boundary).
  4. "We'll use Kubernetes" with no workload numbers to justify it over Functions/ACA.
  5. Public PaaS for PII; secrets in app settings instead of Key Vault + Managed Identity.
  6. No rollback story, no DR test, no cost estimate — and no ADR.
  7. Treating the security score as a gate (it's advice) or a deny policy as advice (it's a gate). Know which is which.

11. How this phase pays off

This is the final interview round and your day job: a room hands you a vague design, and you run the five-step review out loud, put numbers on every pillar, name the trade, and write the ADR — calmly, because you did the arithmetic they were too anxious to do. The lab is that review, executable. Build it, then read BROTHER-TALK and go be the principal in the room.