👨🏻 Brother Talk — Phase 13, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. Here's a thing nobody tells you about observability until you're three years in and holding the pager: the person who can write the query is the person who runs the incident. Not the person with the most context, not the loudest in the bridge call, not even the one who wrote the broken service. The one who, while everyone else is clicking through dashboards and saying "hm, that looks weird," quietly types a KQL query, hits run, and says "it's the dependency call to Key Vault, p99 went from 40ms to 3 seconds at 14:02, here's the operation_Id of a failing trace." That person just became the most senior engineer in the room, regardless of their title. I have watched it happen dozens of times. The query is the authority.

So let me give you the mindset shifts, brother to brother.

The dashboard is a lie you build for yourself, and the query is the truth. Dashboards are great — for the questions you already knew to ask. But production never breaks the way your dashboard anticipated. The failure that pages you is always the one no chart shows, and the moment you accept that, you stop treating "drop into the Logs blade and write KQL" as a last resort and start treating it as the main event. The principals I respect barely look at dashboards during an incident. They open the query window. Make that your reflex. The portal's pretty charts? Those are auto-generated KQL — you're just learning to write what the portal writes for you, except better, because you know what you're actually looking for.

Stop reporting averages. I mean it, it's banned. Same energy as Phase 00's "real-time" ban. The average latency is the single most misleading number in our field, because it hides the exact thing that matters: the tail. A service can have an 80ms average and a 4-second p99, and that p99 is a real cohort of real humans having a genuinely bad time — and they're the ones who churn, who tweet, who file the ticket your VP sees. The moment you say "p95" and "p99" instead of "average," you sound three levels more senior, because you've revealed you know that the tail is the product. Averages are for people who haven't been paged yet.

Cardinality is where the money quietly bleeds out, and you will be the one who finds it. Here's a career-making move that looks like janitorial work: every few months, run Usage | summarize GB=sum(Quantity) by DataType and find out what's actually eating the bill. Nine times out of ten it's one team logging full URLs with query strings, or stamping a per-request GUID as a metric dimension, and it's 5×'ing the ingestion cost for zero analytical value. You find it, you fix it, you save the org real money, and you do it with a one-line query. Nobody asked you to. That's exactly why it gets noticed. The principals who get promoted aren't only the ones who ship features — they're the ones who quietly make the platform cost less and can prove it with a number.

Alerts are a relationship with your future self at 3 a.m. — don't betray him. Every alert you create is a promise: "this is worth waking a human for." Break that promise — page on noise, flap on a boundary, fire on a transient blip — and the deadliest thing in operations happens: people start ignoring the alert. An ignored alert is worse than no alert, because it gives you the illusion of coverage while the real incident scrolls past muted. So tune them like your sleep depends on it, because someone's does. Debounce (for / N violations). Pick gt vs ge on purpose. Decide explicitly whether "no data" is an emergency or a Tuesday. The engineer who keeps the alert channel trustworthy is worth more than the one who adds fifty new alerts nobody reads.

Tracing is the superpower that ends the blame game. You will sit in incident calls where five teams each insist it's not their service, and it'll go in circles for forty minutes because everyone's looking at their own slice. Then you join requests to dependencies on operation_Id, attribute the latency to the exact hop, and the argument is over — not because you won it, but because there's nothing left to argue about. Data ended it. The first time you do that, people start treating you like a principal. So everywhere you build, ask: is the correlation id propagated? If operation_Id doesn't flow through every hop, you can't reassemble the trace, and you're back to forty-minute blame calls. Protect the correlation id the way Phase 00 told you to protect replay.

The honest truth about this phase: the engine you build is small — a few dozen lines of pure functions over lists of dicts. The judgment around it is the whole job. Knowing which pillar to reach for, knowing that percentile is approximate and being at peace with it, knowing that cardinality is the cost, knowing that an alert is a promise — none of that is in the syntax. It's in having been paged enough times to have opinions. You don't have those scars yet, and that's fine; this lab is how you get the intuition before the pager gives you the scars.

One more thing, and it's the most important. Observability is where you stop being someone who deploys the platform and become someone who operates it — and that transition is the entire difference between senior and principal. Anyone can terraform apply. The principal is the one who's still standing, calm, querying, when the thing they applied is on fire at 2 a.m. — turning a wall of red alarms into one sentence: "it's the dependency on the regional cache, here's the blast radius, here's what I'm doing about it." That calm isn't a personality trait. It comes from the query. From knowing that whatever is wrong, you can ask the data and it will answer. That's not magic. That's this phase. Build the engine, and the calm follows.

Go build the KQL engine. Then come find me in Phase 14 — that's where we turn all this querying into actual reliability: SLOs, error budgets, retries, circuit breakers, and the math that lets you promise an availability number and mean it.

— your brother 👨🏻