« Phase 14 · Warmup · Track Overview

Lab 01 — The SRE Console

The problem

It is 03:12. Your phone goes off. Availability is 99.97% against a 99.5% target, latency p99 is normal, error rate is flat, CPU is at 40%.

And the platform is broken:

• One tenant's agent has been in a retry loop since 23:00 and has spent 340% of its monthly budget. Every request succeeded.
• A guardrail change is refusing 60% of retrieval, so answers are being generated from nothing. Every request succeeded.
• The provider quietly moved the model behind a stable version string, and the eval score dropped 0.14. Every request succeeded.

None of the four golden signals moved, because none of these are errors. That is what "tuned for non-deterministic AI workloads" means: correctness is a distribution, cost is a first-class signal, and the thing you page on is not the thing that is wrong.

You build the console that would have caught all three — and, harder, that does not page for the one failure out of two requests on a service nobody uses.

What you build

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

• All 132 tests green against your lab.py.
• A safety block is valid but not good; a client error is neither.
• An empty window scores 1.0, not 0.0.
• latency_sli is inclusive at the threshold, and a fast failure is still not good.
• burn_rate_threshold(0.02, 1.0) == 14.4, derived.
• An alert fires only when both windows agree.
• A recovered short window stops the alert while the long window is still hot.
• One failure in two requests does not page, and the reason names the minimum.
• The budget never goes negative, and overspend is a separate number.
• Two fully-concurrent children do not produce zero self time for their parent.
• Total self time across all spans equals the root's duration.
• A metric with user_id is rejected regardless of its estimated size.
• A rejected metric is not registered.
• The ladder jumps straight to level 3 at a 20× burn rate, and ascends one rung at a time.
• Recovery holds for the configured period before ascending.
• Cost per action doubles when half the requests fail.
• One tenant tripping the breaker leaves another allowed.
• A tenant with no configured budget is denied.
• The same growth curve alerts or not depending only on the lead time.
• A regression with every version unchanged yields the provider-or-drift hypothesis.

How this maps to the real stack

Honest limits. The forecaster fits a straight line — which is exactly wrong for the traffic an AI platform actually sees, where adoption is closer to exponential and demand is strongly diurnal. A real one needs seasonality and a confidence interval, and reporting a point estimate without one overstates what it knows. The alerting has no for: duration and no inhibition, so a real deployment would still fire three alerts for one incident. The cardinality budget works from estimates, and the estimate for tenant is wrong the day someone onboards a customer per branch. Spans have no sampling, which is the decision that actually determines whether tracing is affordable — at 100% sampling an agent platform generates more trace data than log data. And the degradation ladder does not shed anything; it decides what would be shed and in what order, which is the part that has to be decided in daylight.

Extensions

1. Add sampling. Head-based, then tail-based (keep every errored or slow trace, sample the rest). Then answer: what is the storage cost at 100%, at 10%, and what did you lose?
2. Seasonality in the forecaster. Fit weekly and daily components. Compare the projection against the linear one on real-shaped data and see how different the procurement date is.
3. for: durations and inhibition. Make one incident produce one page instead of three.
4. A quality SLO, properly. Sampled offline evaluation as a gated objective: it blocks deploys and is reviewed weekly, and it never pages. Then defend that choice.
5. Burn rate on cost. The same multi-window machinery against a spend budget rather than an error budget. Most of the code is identical, which is the interesting part.
6. Exemplars. Link a metric bucket to a trace id, so "p99 is bad" becomes "here is a p99 trace". It is the single highest-value observability feature most teams have not enabled.
7. A real post-mortem template, generated from the console: the timeline from alerts, the budget consumed, the degradation steps taken, the traces at the p99.
8. Multi-region SLIs. Compose per-region SLIs into a global one, and discover that the ratio composes cleanly while the percentile does not — which is the argument in §2, demonstrated.

Interview / resume bullets

• "Defined the platform's SLIs on an explicit event-ratio model with a written validity predicate, which turned every SLO argument from a disagreement about numbers into a disagreement about the denominator — where it belongs."
• "Kept answer quality out of the availability SLO and ran it as a gated objective instead: it blocks deploys and is reviewed weekly, but it never pages, because a distribution shift is not an incident."
• "Implemented multi-window multi-burn-rate alerting with derived thresholds and a minimum-volume guard, which cut 3 a.m. pages on low-traffic services to zero without raising a single threshold."
• "Made cost per successful action and safety-block rate first-class signals, and added a per-tenant cost circuit breaker — which caught a runaway agent loop that every traditional signal reported as healthy."
• "Instrumented agent runs as OTel span trees with GenAI semantic conventions and self-time attribution, so 'the platform is slow' became 'the reranker is 690 ms of the 4.1 s' in one query."
• "Enforced a cardinality budget at metric-definition time, which prevented the label that would have taken the metrics backend down during an incident."
• "Forecast capacity against provider quota rather than utilization, with alerts sized to procurement lead time — so a GPU quota conversation started at 55% utilization instead of at a hard 429."