Lab 03 — Observability, SLOs & Alerting: Metrics, PromQL-lite, Error Budgets

Phase 28 · Lab 03 · Phase README · Warmup

The problem

Every Kubernetes platform ships the same observability stack — Prometheus scraping metrics, Grafana rendering dashboards, Alertmanager routing pages — and most engineers use it for years without ever being able to answer the three questions that separate an operator from an SRE:

1. What does rate() actually compute, and why is a raw counter useless without it?
2. How does histogram_quantile() turn bucket counts into a p99 — and why is that number an estimate whose accuracy you chose when you chose your buckets?
3. Why does a good paging alert need a for: duration and two burn-rate windows — and what exactly is an error budget, arithmetically?

This lab makes you build all three mechanisms, plus the alert state machine between them (inactive → pending → firing → resolved), so that when a Grafana panel shows histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) you can read it the way you read your own code — because you wrote each function in it.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• Counters are monotonic (negative increment = structured error) and store cumulative values; gauges take the latest write.
• rate() over a known steady series returns the exact per-tick rate, and uses only the trailing window (a past burst outside the window doesn't leak in).
• histogram_quantile() interpolates linearly within the correct bucket, walks cumulative counts across buckets, and returns the largest finite bound for the +Inf overflow bucket.
• sum_by() groups series by a label's value and sums the latest samples.
• An alert breaching its threshold goes pending, fires only after the breach is sustained for for_ticks, and a transient spike shorter than that never fires.
• A firing alert resolves when the expression recovers, and every transition is recorded in order.
• sli/error_budget/burn_rate compute the SRE arithmetic exactly (99.9% target + 99.5% SLI = 5× burn).
• The fast-burn alert triggers when both windows exceed the factor, and stays quiet when the short window has recovered even though the long window still remembers the incident.
• All 32 tests pass under both lab and solution.

How this maps to the real stack

• The metric types are Prometheus' metric types. A counter is cumulative and monotonic so a missed scrape loses resolution, not data — rate() reconstructs per-second behavior from any two samples. A gauge is instantaneous. A histogram is _bucket{le="..."} counters plus _sum and _count; our per-series {"counts", "inf", "sum", "count"} dict is exactly that data layout. And a series being keyed by (name, sorted label pairs) is Prometheus' identity model — which is also why a high-cardinality label (user ID, pod UID, request ID) multiplies your series count and takes the TSDB down. Real Prometheus adds the TSDB (chunked, compressed storage), scraping over HTTP /metrics, and staleness handling; the query semantics are what you built.
• rate(), histogram_quantile(), sum by() are the three PromQL constructs that appear in virtually every production dashboard, usually composed: histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))). Real rate() also handles counter resets (a restarted pod's counter starts back at 0 — Prometheus detects the drop and compensates); ours doesn't, which is a stated limit below. Real histogram_quantile runs over rated buckets (quantile of the last 5m, not of all time); ours runs over the snapshot, same interpolation.
• The alert state machine is Prometheus' alerting-rule lifecycle verbatim: inactive → pending (expr true, for: not yet elapsed) → firing (sent to Alertmanager, which owns grouping, silences, inhibition, and routing to PagerDuty/Slack/Opsgenie) → resolved. The for: duration is the single most important field in a paging rule — without it you page on every scrape blip.
• SLI/SLO/error budget/burn rate are Google SRE's definitions: SLI = good events / valid events; SLO = the target over a window (say 99.9% over 30 days); error budget = 1 − SLO; burn rate = how many times faster than "exactly spend the budget by window's end" you're erring. The multi-window, multi-burn-rate alert (long window ≥ factor AND short window ≥ factor, e.g. 14.4× over 1h AND 5m for a 30-day SLO) is the SRE Workbook's recommended paging setup: the long window gives confidence it's real, the short window guarantees it's current so you don't get paged for an incident that ended an hour ago — exactly the behavior the test suite proves.
• Grafana is the query client on top: every panel is a PromQL expression like the ones you built, rendered over time. The RED method (Rate, Errors, Duration — for services) and USE method (Utilization, Saturation, Errors — for resources) from the Warmup are dashboard design disciplines over these exact primitives.

Limits of the miniature. No scraping, no TSDB, no retention/compaction — the store is an in-memory append-only list. rate() doesn't handle counter resets. histogram_quantile runs on snapshot counts, not rated buckets, and there are no native/exponential histograms. The AlertManager here evaluates and tracks state but doesn't group, silence, inhibit, or route notifications (real Alertmanager's whole job). And logs and traces — the other two observability legs — are covered by Phase 14's tracer and the Warmup, not this lab.

Extensions (your own machine)

• Add counter-reset handling to rate(): when a sample is lower than its predecessor, treat the drop as a reset (add the post-reset value instead of the negative delta) — then write the test that a pod restart mid-window doesn't produce a negative rate.
• Add recording rules: a rule that evaluates an expression each tick and writes the result back into the store as a new series (job:http_errors:rate5m), then alert off the recorded series — the standard trick for making dashboards and alerts cheap.
• Implement Alertmanager grouping: batch firing alerts by a label set within a time window so one incident = one notification, not fifty.
• Run the real thing: prometheus --config.file=... + node_exporter locally, write your HighErrorRate rule with a for: clause, and watch the pending→firing transition in the UI.
• Compute the full SRE Workbook alert table: for a 30-day 99.9% SLO, derive the burn-rate factors and window pairs for page-fast, page-slow, ticket-fast, ticket-slow.

Interview / resume signal

"Built the Prometheus mechanism set from scratch: a label-keyed time-series store with counter/gauge/histogram semantics, PromQL-style rate()/histogram_quantile() (cumulative bucket walk with linear interpolation)/sum by() queries, the alerting-rule state machine with for:-duration debounce and resolution, and Google-SRE SLO arithmetic — SLI, error budget, burn rate, and the multi-window fast-burn paging alert, all deterministic and test-proven."