Lab 01 — KQL Query Engine + Alert Evaluator

Phase: 13 — Observability, Azure Monitor & KQL | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

When a service degrades at 2 a.m., the portal is not where you find the cause — the query is. Azure Monitor stores billions of log rows in a Log Analytics workspace, and you interrogate them with KQL (Kusto Query Language): a left-to-right pipeline of tabular operators (where | extend | summarize ... by bin() | join | top) that filters, buckets, aggregates, and correlates until the signal falls out. Pull the cover off and KQL is just a sequence of pure functions over a list of rows, each returning a new list — which is exactly why a query is safe to re-run and reason about. This lab builds that engine: the operators, the aggregations a principal recites in their sleep (count / sum / avg / min / max / dcount / percentile), the bin() that turns continuous time into 5-minute buckets, an inner/leftouter join, and — the thing that actually pages you — an alert-rule evaluator that aggregates a metric over a lookback window and compares it to a threshold.

What you build

• Table(rows) — a fluent wrapper so t.where(...).summarize(...).top(...) reads like the KQL pipe (|). Every method returns a new Table (pure; nothing is mutated). Underneath are the real engine — the operator functions, usable directly on any list[dict].
• Row-shaping operators — where(rows, predicate) (filter), project(rows, columns) (keep/rename columns; takes a list or a {new: old} dict), extend(rows, **computed) (add columns from callable(row) -> value).
• Ordering / windowing — order_by(rows, key, desc) (stable sort), top(rows, n, by, desc) (the n biggest/smallest), take(rows, n), count(rows).
• bin_time(value, width) — the bin() function: floor(value / width) * width, flooring continuous timestamps onto an absolute grid so rows in the same interval group together inside summarize.
• percentile(values, p) — the nearest-rank percentile (rank = ceil(p/100 · n)), chosen because it returns an observed value so a p95 is exactly assertable.
• summarize(rows, aggregations, by) — the heart of KQL: group rows by the by columns and reduce each group with count / sum / avg / min / max / dcount / percentile. Deterministic group order (sorted by the group key). by=[] gives one global summary row.
• join(left, right, on, kind) — a hash join on a single key, inner and leftouter.
• AlertRule + evaluate_alert(rows, rule, *, now) — the alert: window-filter rows to [now - window, now], aggregate the metric, compare to the threshold with gt/lt/ge/le. Empty window → does not fire (no data cannot breach).

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # worked example

Success criteria

• All 35 tests pass against your implementation.
• You can explain why test_percentile_nearest_rank_known_answers asserts percentile([1..100], 95) == 95 exactly: nearest-rank takes rank = ceil(0.95·100) = 95, the 95th-smallest value — an observed point, never an interpolated one. (Then explain why real Kusto's T-Digest answer might be off by a hair, and why that's an acceptable cost/fidelity trade at billions of rows.)
• You can explain why test_bin_time_buckets_to_grid floors 299 → 0 and 300 → 300: bin() lands rows on an absolute grid of width-sized buckets, which is why two queries with the same bin(Timestamp, 5m) line up even though they never coordinated.
• You can explain why test_summarize_count_by_group_is_deterministic matters — real Kusto doesn't promise group order, so a teaching engine that does (sorted by key) is testable and reproducible.
• You can explain why test_alert_boundary_gt_vs_ge is the off-by-one interviewers probe: a metric of exactly 300 against threshold 300 does not fire under gt but does under ge. Pick the wrong operator and you either miss the breach or page on the boundary forever.
• You can explain why test_alert_empty_window_does_not_fire is deliberate: an empty window has no metric to breach, so it returns False — modeling Azure Monitor's "no data" behavior rather than a false page (and why "no data" is sometimes itself worth a separate alert).

How this maps to real Azure (Azure Monitor + Log Analytics + Application Insights)

What the miniature leaves out (and why it's fine): real Kusto is a columnar, distributed, indexed engine that runs over petabytes with extent caching, a cost model, ingestion-time transforms, materialized views, cross-workspace/cross-cluster queries, make-series / time-series functions, mv-expand, parse, and dozens more operators; percentile/dcount are approximate (T-Digest / HyperLogLog) precisely because exact answers don't scale. None of that changes the two mechanisms this lab isolates — (1) a query is a left-to-right pipeline of pure operators that culminates in a summarize-style group-and-reduce, and (2) an alert is an aggregation over a lookback window compared to a threshold. Those are exactly the parts an interview probes and an incident turns on.

Extensions (build these in your own workspace)

• make-series / gap-fill: add a make_series(rows, agg, on_time, step, range) that emits a dense time series (one bucket per step across the whole range, zero-filled), the way KQL make-series does for charting — summarize ... by bin() leaves gaps where a bucket had no rows, and gaps lie to the eye.
• Tail-based sampling: model sample_traces(traces, head_rate) (head sampling: keep each trace with fixed probability, decided at ingress) vs a tail sampler that keeps a trace iff it contains an error or a slow span (decided after the trace completes) — then reason about the cost/fidelity trade in the WARMUP.
• Multi-table correlation: build requests + dependencies tables sharing operation_Id, then join them to compute, per request, the sum of downstream dependency time — the classic "where did the latency go?" trace query.
• Alert debounce (for window): extend evaluate_alert to require the threshold be breached for N consecutive evaluations before firing (Azure's "number of violations" / for clause) — the single most effective noise-reduction knob.
• Wire it to real Azure: az monitor log-analytics workspace create; send a few custom logs; run your KQL in the portal's Logs blade (requests | summarize p95=percentile( DurationMs,95) by bin(TimeGenerated, 5m), Service); then az monitor scheduled-query create an alert on it and watch it fire into an action group.

Interview / resume

• Talking points: "Walk me through a KQL query that finds the p95-latency regression by service and 5-minute bin." / "How does percentile scale to billions of rows — what's T-Digest costing you?" / "Why does high-cardinality dcount (per userId) blow up cost, and what do you do instead?" / "Metric alert vs log (scheduled-query) alert — when each, and how do you stop it flapping?" / "How does operation_Id reconstruct a distributed trace across five services?"
• Resume bullet: Built a miniature KQL query engine — the tabular operators (where/project/extend/summarize ... by bin()/join/top), the aggregations (count/sum/avg/min/max/dcount/nearest-rank percentile), and a window-aggregation alert-rule evaluator — modeling Azure Monitor Logs, Log Analytics, and Application Insights alerting.