🛸 Hitchhiker's Guide — Phase 13: Observability, Azure Monitor & KQL

Read this if: you can read a dashboard but "the chart doesn't show what I need, drop into Logs and write the query" still feels like someone else's job. It's yours now. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the engine.

0. The 30-second mental model

Observability is a query problem, not a UI problem. The portal's charts are auto-generated KQL; the principal is the one who writes the query the chart can't. A KQL query is a table piped left-to-right through pure operators (where | extend | summarize ... by bin() | join) until the signal falls out. An alert is just aggregate(metric over a window) <op> threshold → page someone. One sentence to tattoo: metrics tell you that, logs tell you what, traces tell you where — pick the right one before you start clicking.

1. The three pillars (pick one before you panic)

Alert on metrics, investigate in logs, attribute in traces. Wrong pillar = lost first 15 minutes.

2. The KQL operators to memorize (the pipe)

TableName
| where  Col == x and Time > ago(1h)     // filter EARLY — shrinks every later stage
| extend NewCol = expr                    // add a computed column (sees the ORIGINAL row)
| project Keep, Renamed = Old             // pick / rename columns
| summarize Agg = f(Col) by Dim, bin(T,5m)// group-and-reduce (the heart of KQL)
| join kind=leftouter (Other) on Key      // correlate two tables
| order by Agg desc                        // sort
| top 10 by Agg                            // the N biggest (default desc)
| take 100                                  // first N (no order guarantee in real KQL)
| count                                     // scalar: number of rows
| render timechart                          // chart it (portal only)

summarize and bin are the two that carry the weight. bin(t, w) = floor(t/w)*w — absolute grid, not relative to your data; that's why series from different queries line up.

3. The aggregations + the gotchas

Nearest-rank by hand: rank = ceil(p/100 · n). Over 1..100, p95 → rank 95 → value 95.

4. The numbers to tattoo on your arm

5. az monitor one-liners + the KQL that pays the rent

# create a Log Analytics workspace
az monitor log-analytics workspace create -g rg -n my-ws

# run a KQL query from the CLI (great for runbooks / CI checks)
az monitor log-analytics query -w <workspace-id> \
  --analytics-query "requests | summarize p95=percentile(DurationMs,95) by Service" -o table

# list metric definitions for a resource, then pull a metric
az monitor metrics list-definitions --resource <res-id>
az monitor metrics list --resource <res-id> --metric "Http5xx" --interval PT1M

# a scheduled-query (log) alert on a KQL condition
az monitor scheduled-query create -g rg -n high-p95 --scopes <ws-id> \
  --condition "count 'q' > 0" \
  --condition-query q="requests | summarize p95=percentile(DurationMs,95) by bin(TimeGenerated,5m) | where p95 > 1000" \
  --action-groups <ag-id>

# an action group (who gets paged)
az monitor action-group create -g rg -n oncall --action email me me@corp.com

The four KQL queries you'll write a thousand times:

// 1. latency regression by service & 5-min bin
requests | summarize p95=percentile(DurationMs,95), n=count() by Service, bin(TimeGenerated,5m)

// 2. error rate by result code
requests | summarize total=count(), errors=countif(Success==false) by bin(TimeGenerated,5m)
         | extend rate = 100.0*errors/total

// 3. blast radius — distinct users/operations hit by a failure
exceptions | where TimeGenerated > ago(15m) | summarize dcount(user_Id), dcount(operation_Id) by ProblemId

// 4. trace reassembly — attribute latency to downstream hops
requests | join kind=leftouter (dependencies) on operation_Id
         | summarize req_p95=percentile(DurationMs,95), dep=sum(dep_DurationMs) by operation_Id

6. War story shapes you'll relive

• "The dashboard says we're fine but customers are screaming." → the chart averages. avg is 80 ms; p99 is 4 s. Switch to percentile(Duration, 99) and the tail appears. The average is the liar.
• "Our Log Analytics bill 5×'d overnight." → someone started logging full URLs (with query strings) or a per-request id as a dimension. High cardinality = GB ingested = $$. Drop the field or sample it; aggregate it, don't by it.
• "The alert pages every few minutes and we've started ignoring it." → window too short, no debounce, or gt flapping right at the boundary. Add a for/N-violations debounce and a sane lookback. An ignored alert is worse than no alert.
• "It's not my service." (said by all five teams) → join requests to dependencies on operation_Id, summarize the downstream time per hop. The trace ends the standoff with a number.
• "We sample 1%, so why are our error dashboards empty?" → head sampling dropped 99% of an already-rare error class. You need tail sampling for the error/slow paths.
• "No data" alert never fired when the service died. → an empty window can't breach a threshold, so silence ≠ a page unless you explicitly alert on no-data (for a heartbeat, you must).

7. Vocabulary that signals you've held the pager

• Cardinality — distinct values of a column; the thing that costs money in observability.
• operation_Id / operation_ParentId — trace id / parent span id; how a request becomes a tree across services.
• Span / trace — a unit of work / the whole tree of them for one user action.
• Head vs tail sampling — decide at ingress (cheap, drops errors) vs after completion (keeps errors, costs complexity).
• T-Digest / HyperLogLog — the sketches behind approximate percentile / dcount.
• SLI / SLO / error budget — the indicator (a query), the target, and the allowed failure you spend.
• Burn rate — how fast you're spending error budget; the alert that scales with traffic.
• Action group — the who/what that gets notified when a rule fires.
• Metric alert vs log alert — cheap pre-aggregated series vs full KQL on a schedule.
• The four golden signals — latency, traffic, errors, saturation.

8. Beginner mistakes that mark you in interviews

1. Reaching for logs to answer "is it up" (that's a metric) or a metric to debug "this one request" (that's a log).
2. Reporting avg latency instead of a percentile — instantly junior.
3. summarize ... by a high-cardinality column (userId, requestId, raw URL) and being surprised by the cost and the slow query.
4. Saying percentile/dcount are exact — they're approximate at scale by design.
5. Building an alert with no debounce/for and no no-data policy, then drowning in pages.
6. Ignoring the gt-vs-ge boundary, so a value exactly at the threshold either never pages or always does.
7. inner join when hunting for what's missing (it silently drops the very rows you wanted).
8. Sampling at a flat head rate and then wondering why rare errors vanished from the data.

9. How this phase pays off later

• The KQL engine (the lab) is literally how you investigate every incident in P14, and how you measure the SLOs P14 manages.
• operation_Id correlation ties together everything you traced through P09 (gateway), P10 (Service Bus), P11 (Functions), P12 (Key Vault) — one request, one story.
• Cardinality/cost thinking is the FinOps half of the P15 capstone's combined scorer.
• The alert evaluator is the operational backbone of "high availability" the JD demands — and the difference between deploying a platform and running one.

Now read the WARMUP slowly, then build the KQL engine. After that, the portal's Logs blade stops being a mystery box and starts being a REPL you own.