Phase 13 — Observability, Azure Monitor & KQL

Difficulty: ⭐⭐⭐☆☆ (the query engine) → ⭐⭐⭐⭐⭐ (operating it at scale without going broke) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (the five forces, latency budgets), Phase 03 (JWT/operation_Id correlation intuition), Phase 11 (the distributed services you'll be tracing). Phase 14 (reliability) consumes the SLO/error-budget thinking you build here.

Why This Phase Exists

Every prior phase built something — a deployment graph, a token validator, an NSG evaluator, a message broker, a Functions runtime. This phase is how you find out, at 2 a.m., which of them just broke and why. The JD is explicit: "Monitor, troubleshoot, and optimize cloud infrastructure to ensure high availability, security, and performance" and calls out Azure Monitor / Log Analytics / KQL and Application Insights / distributed tracing by name. That is not a "nice to have" bullet — it is the difference between an engineer who operates a platform and one who merely deploys it.

Here is the uncomfortable truth most "set up a dashboard" tutorials hide: observability is a query problem, not a UI problem. The portal's pretty charts are auto-generated KQL. When the chart doesn't show what you need — and in a real incident it never does — you drop into the Logs blade and write the query, joining requests to dependencies on operation_Id, summarizing p95 latency by 5-minute bin and service, and dcount-ing distinct failing users to gauge blast radius. The principal is the person who writes that query in 90 seconds while everyone else is clicking. And it is also a cost problem: logs are billed by the GB ingested and retained, traces by the sample, and a single high-cardinality dimension (userId, requestId) can 10× your bill or your query latency.

So this phase makes you build the KQL engine — the read pipeline of tabular operators and aggregations — and the alert evaluator that pages you, because once you've implemented summarize ... by bin() and a window-vs-threshold alert by hand, you stop reciting query syntax and start reasoning about what the query costs and where it lies.

Concepts

• The Azure Monitor umbrella — one brand over several distinct data planes: Metrics (pre-aggregated numeric time series, cheap, 1-minute granularity, near-real-time), Logs (a Log Analytics workspace of typed rows you query with KQL, expensive, flexible), Alerts (rules over metrics or logs → an action group), and Application Insights (APM: requests/dependencies/traces/exceptions correlated by operation_Id). Knowing which store answers which question is half the skill — a metric for "is it up and how loaded," a log for "what exactly happened to this request."
• KQL — the read pipeline — where | project | extend | summarize ... by bin() | join | top | order by | count | take. Each operator is a pure transform over a table; queries compose left-to-right. The semantics of summarize (group-and-reduce) and bin (floor onto an absolute time grid) are the load-bearing two.
• Aggregations & cardinality — count / sum / avg / min / max / dcount / percentile. How a percentile is computed (nearest-rank vs interpolation vs Kusto's T-Digest approximation), why dcount is approximate (HyperLogLog) at scale, and why a high-cardinality by dimension blows up both cost and query latency.
• Distributed tracing — a trace is a tree of spans; Application Insights correlates them with operation_Id (the trace id) and operation_ParentId (the parent span), so a join on operation_Id reassembles a request's path across five services and shows you which hop ate the latency. Sampling (head vs tail) trades ingestion cost for fidelity.
• Alert rules — aggregate(metric over a lookback window) <operator> threshold → action group. The window/lookback, the aggregation granularity, the operator boundary (gt vs ge), and the for/debounce clause are where the signal-vs-noise battle is won or lost. A flapping alert is worse than no alert; a missed breach is worse than both.
• SLO / SLI / error budget — the Google-SRE frame the JD's "high availability" implies: define a Service Level Indicator (a query), set a Service Level Objective (a target on it), and spend the resulting error budget on velocity — so reliability is a number you manage, not a vibe. Developed fully in Phase 14; rooted here.

Concept Map — How the Pieces Fit

   your services emit telemetry
            │
            ▼
   ┌─────────────────────────────────────────────────────────────┐
   │ METRICS (numeric TS)      LOGS (Log Analytics, KQL)          │
   │  cheap, 1-min, "is it up"  $/GB, flexible, "what happened"   │
   │        │                          │                          │
   │        │                          ▼                          │
   │        │           where | extend | summarize ... by bin()   │
   │        │                 | join (operation_Id) | top         │
   │        │                          │                          │
   │        ▼                          ▼                          │
   │   ALERT RULE:  agg( signal over [now-window, now] ) op thr   │
   │                          │                                   │
   │                          ▼                                   │
   │                     ACTION GROUP  →  human / webhook / runbook│
   └─────────────────────────────────────────────────────────────┘
            ▲
            │ APPLICATION INSIGHTS (APM): requests/dependencies/traces/
            │ exceptions, all stamped operation_Id → reassemble the trace

The whole phase is "build the middle of that picture": the KQL operators that turn raw log rows into a number, and the alert rule that turns that number into a page. Metrics sit beside logs as the cheap "is it up" channel; Application Insights feeds the log tables and adds the operation_Id correlation that makes distributed tracing possible.

Labs

Lab 01 — KQL Query Engine + Alert Evaluator (flagship, implemented)

→ Lab folder: lab-01-kql-engine/

Integrated-Scenario Suggestions (carried through the whole track)

These tie this phase's observability layer to the platform you've been building across the track — each is a real query/alert you'd own as the principal:

1. The latency-regression hunt — requests | summarize p95=percentile(DurationMs, 95), n=count() by bin(TimeGenerated, 5m), Service, then render timechart. When a release regresses p95, this is the query that names the service and the minute. (Lab 01 directly.)
2. The distributed-trace reassembly — a slow checkout spans the API gateway (P09), Service Bus (P10), a Function (P11), and Key Vault (P12). join requests to dependencies on operation_Id to attribute the latency to the exact hop — the query that ends the "it's not my service" standoff.
3. Blast-radius gauge — exceptions | where TimeGenerated > ago(15m) | summarize dcount(user_Id), dcount(operation_Id) by ProblemId — how many distinct users/operations a failure touched. The cardinality is the severity.
4. The cost autopsy — Usage | summarize GB=sum(Quantity) by DataType to find which table is eating the ingestion bill, then a sampling/retention/transform decision — the FinOps half of observability (P15 capstone).
5. SLO + error budget — define an availability SLI as a KQL ratio (good_requests / total), set a 99.9% SLO, and a burn-rate alert that fires when you're spending error budget too fast — the bridge into Phase 14 reliability.

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; KQL operators and az monitor one-liners to memorize; read first
• WARMUP.md — the full primer: each concept's mechanism under the hood, the percentile/sampling math, the lab walkthrough, interview Q&A, references; read slowly
• BROTHER-TALK.md — the candid version: what observability really costs you and your career

Key Takeaways

• Observability is a query problem. The chart is auto-KQL; the principal writes the query the chart can't. Build the engine and the syntax becomes mechanism.
• Know which store answers which question — a metric for "is it up and how loaded," a log for "what happened to this request," a trace for "which hop was slow."
• Cardinality is the cost. dcount on a high-cardinality dimension (userId, requestId) is what 10×'s your bill and your query latency; aggregate first, sample deliberately.
• An alert is agg(window) <op> threshold → action group. The window, the boundary (gt vs ge), and the debounce decide whether it's a signal or a 3 a.m. lie.
• operation_Id is how a request becomes a story across services; a join on it is the whole of distributed tracing.

Cross-Cutting Themes (the principal's lenses for this phase)

• Control plane vs data plane, again. Azure Monitor config (workspaces, alert rules, action groups) is control-plane ARM/RBAC — the same idempotent PUT and role model from P00–P04. The telemetry itself (the rows you query, the metrics you read) is a data plane with its own access model (Log Analytics Reader), its own throttling, and its own bill. A Reader on the resource is not a reader of its logs.
• Cardinality is the new "$/TB scanned." Phase 00 taught that layout is a cost feature; here the layout that costs money is dimensional cardinality. Every high-cardinality field you ingest or summarize ... by is a line item — aggregate first, sample deliberately, never group by a GUID.
• Approximate by design. percentile (T-Digest) and dcount (HyperLogLog) are not exact at scale, and that's the correct engineering trade — bounded memory and distributed mergeability beat an exact answer you can't afford to compute. Knowing this is a principal signal; assuming exactness is a junior tell.
• An alert is a promise to your future self. Signal-vs-noise is not a tuning afterthought; it's the core design problem. The window, the gt/ge boundary, the for/debounce, and the no-data policy decide whether the on-call channel stays trusted or becomes muted noise.
• Correlation is causation's only evidence. operation_Id propagated through every hop is what turns five services' separate logs into one request's story. If the correlation id doesn't flow, you cannot trace, and you're back to forty-minute blame calls.

Deliverables Checklist

• Lab 01 implemented; all 35 tests pass against solution.py and your lab.py
• You can write, from memory, a KQL query that computes p95 latency + count by 5-minute bin and service, and explain each operator's effect on the row set
• You can explain nearest-rank vs interpolation vs T-Digest percentile, and the cost/fidelity trade Kusto makes
• You can explain why a high-cardinality dcount dimension is expensive and what you do instead
• You can describe a metric alert vs a log (scheduled-query) alert, the for/debounce knob, and the gt-vs-ge boundary trap
• You can draw the operation_Id / operation_ParentId correlation and explain head vs tail sampling