👨🏻 Brother Talk — Phase 14, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview — and again the night before your first real on-call shift.

Listen. This is the phase where you stop being someone who builds things and start being someone who can be trusted with things. Because here's the truth nobody puts on a job description: anyone can stand up a service that works on a good day. The reason principals get paid is the bad day — the 2 a.m. page, the cascading failure, the "everything is red and the CEO is in the channel" moment. And what separates the person who fixes it from the person who makes it worse is not raw intelligence. It's whether they have the arithmetic and the method in their bones, or whether they're improvising under stress. This phase is those two things. Burn them in.

Let me start with the single most useful party trick you'll ever learn, because it changes how people see you in one meeting. Someone says "we have a 99.9% SLA." You say, calmly: "that's forty-three minutes a month — and our chain has four hops, so the system is more like 99.84, which is sixty-nine minutes. We're already over budget on paper. Which hop do we make redundant first?" Watch the room. You just did, out loud, in five seconds, the thing everyone else assumed was true and nobody actually computed. That's not magic — it's ∏ aᵢ and (1 − A) × 43200. But the effect is that you become the person who knows whether the promise is real. The whole game in this phase is turning vague reliability talk into specific numbers, because numbers are where calm comes from, and calm is what they're really hiring at the principal level.

Now the retry stuff, because this is where good engineers blow up production trying to help. Here's the thing that took me a real incident to internalize: a retry is not a safety feature; it's a loaded gun pointed at your dependency. The intuition "if it failed, try again" is right for one client and catastrophic for a thousand. A dependency blips for two seconds; every client dutifully retries; and because they all failed at the same instant, they all retry at the same instant, and now the dependency is getting its normal load plus a synchronized retry wave, which keeps it down, which causes more retries. You didn't recover from the outage — you fed it. The fix is three things and you need all three: jitter (so the herd doesn't move in lockstep), a budget (so the retries can't pile up without limit), and idempotency (so retrying doesn't double-charge somebody's credit card). And when the server hands you a Retry-After, honor it — it's literally telling you when to come back, and arguing with it just keeps you throttled. Most outages I've watched go from "blip" to "incident" were a retry storm. Be the person who knows why.

Here's a mental model that'll serve you forever: a retry and a circuit breaker are opposites, and you need both. A retry says "try again" — it's for a transient failure, a blip, a thing that'll work the second time. A circuit breaker says "stop trying for a while" — it's for a sustained failure, a dependency that's actually down, where trying again just wastes a timeout and piles up. The junior reaches for retry for everything and turns a dead dependency into a self-inflicted DoS. The principal knows: blip → retry (with the three catches); outage → break the circuit, fail fast, fall back, and probe once for recovery. When you can say that distinction crisply in an interview — "retry is for transient, the breaker is for sustained, here's why each state of the breaker exists" — you've shown them you've actually held the pager, not just read the docs.

Let me give you the two incidents you will have, so they're not a surprise.

The 403 that isn't an auth problem. Your Function can see the storage account but gets 403 reading a blob. You'll burn an hour checking the token, the Managed Identity, the network — all fine. The answer is the control-plane vs data-plane split from Phase 00: the identity has Reader (it can see the resource exists) but not Storage Blob Data Reader (it can't read the bytes). Two completely different RBAC worlds, and Azure's role names are designed to trick you. Once you've been bitten by this, 403 on a data operation makes you instantly check the data-plane role first. That instinct is worth a hundred docs.

The DNS failure that isn't a Storage outage. You add a Private Endpoint — good, secure, the right call — and suddenly "name resolution failed." You'll think Storage is down. It isn't. The Private Endpoint gave the resource a private IP, but DNS is still handing out the public one, because the Private DNS zone isn't linked to your VNet or the A record is wrong. This one fools entire teams. The tell is in the symptom — "name resolution failed" — which is why your triage checks the symptom before the status code. A DNS error is a network problem no matter what HTTP status comes after it.

And this is the deepest thing in the phase, so sit with it: triage is a decision tree, not an inspiration. Under stress your IQ drops, your working memory shrinks, and the worst thing you can do is "think hard." The best operators don't think harder at 2 a.m. — they think less, because they've pre-compiled the tree. Signature 429? Throttling, honor Retry-After, check limits. 403? Authz, data-plane role. DNS error? Private DNS zone link. 503? Dependency down, circuit-break. You're not being clever; you're executing a runbook you wrote when you were calm. That's why we build the diagnose function in the lab — to force the tree out of your gut and into something explicit you can trust when your gut is panicking. The calm senior in the incident channel isn't smarter than you. They just already wrote the tree.

One honest thing about the error budget, because the framing flips a career. Reliability beginners treat 100% as the goal and every minute of downtime as a personal failure — and that mindset paralyzes a team, because the only way to never go down is to never change anything. The principal framing is the opposite: that 43 minutes a month is a budget you get to spend on deploys, experiments, and yes, the occasional incident. If you're not spending your error budget, you're being too cautious and shipping too slowly. This is the sentence that marks you as someone who actually understands SRE and not just uptime: "we're under-spending our error budget, so we can afford to move faster here." Say that in a review and watch a director's eyebrow go up. You just reframed reliability from a fear into a resource.

Here's the career angle, brother to brother. Every other phase made you someone who can build a piece of the platform. This phase makes you the person they call when it breaks — and that person is different. That person gets trusted with the production keys, gets put on the escalation path, gets pulled into the architecture review specifically because they'll ask "and what's the composite SLA of that?" before anyone ships. The reliability person is the adult in the room. Not because they know more services, but because when everyone else is reacting, they have a number and a tree. Calm is a skill, and it's built from exactly the arithmetic and the method in this phase. You're not born with it. You compute your way into it.

Go build the resiliency engine. Compose a four-hop SLA and feel it drop below every hop. Watch the retry budget exhaust and stop the storm. Trip the breaker and recover it through half-open. Run a 403 and a DNS error through the tree. Then the next time the channel lights up red, you'll be the one typing the number while everyone else is typing "any ideas?"

Then come find me in Phase 15 — that's the capstone, where everything you've built across the whole track gets composed into one platform and scored on security, SLA, and cost at the same time. This phase's arithmetic is half of that score.

— your brother 👨🏻