« Track Overview · Warmup · Lab 01
Phase 14 — SRE for Non-Deterministic AI Workloads
Answers these JD lines: "Own platform Site Reliability Engineering (SRE), including SLO design, error budget management, observability (OpenTelemetry, traces, metrics, logs at agent and tool granularity), incident response, post-mortems, capacity planning, and cost governance for a growing fleet of agents and AI workloads in production" · "observability tooling … tuned for non-deterministic AI workloads."
Why this phase exists
Phase 00 introduced SLOs, error budgets and burn-rate alerting as architecture inputs. This phase is the run-state discipline that operates them — and the JD's qualifier, "tuned for non-deterministic AI workloads", is doing real work.
Ordinary SRE assumes that "correct" is a predicate. For an AI platform it is a distribution: the same input can produce a good answer, a mediocre one and a wrong one, and none of them is an error in the HTTP sense. That single fact breaks three habits:
- You cannot put quality in the availability SLI. It is not measurable in real time, not attributable to the platform, and it makes the metric un-actionable during an incident.
- Your golden signals are incomplete. Latency, traffic, errors and saturation say nothing about cost per request or safety-block rate, both of which can move catastrophically while every traditional signal is green.
- Debugging is trace-first, not log-first. A run is a tree of model calls, tool calls and retrievals; a flat log stream cannot reconstruct it, and non-determinism means you cannot reproduce it by re-running.
The phase is therefore about instrumenting a system whose correctness is statistical, and about the operational disciplines — incident response, post-mortems, capacity, cost — that the JD names explicitly.
Concept map
- SLIs for an AI platform: request-based availability and latency at the ingress; and, separately, quality (sampled offline evaluation), cost per successful action, and safety-block rate — with a clear statement of which are hard SLOs with pages and which are tracked objectives with gates.
- The event-ratio model:
good / valid, and the fact that the definition of valid is a design decision with a written eligibility predicate. - Error budgets and multi-window multi-burn-rate alerting: the 14.4 / 6 / 1 ladder, derived rather than memorized; minimum-volume guards for low-traffic services.
- OpenTelemetry with GenAI semantic conventions: a span per agent step and per tool call, carrying model, token counts, cost, tenant and trace linkage — so the debugging artifact and the audit artifact are the same object (Phase 01).
- Cardinality governance: the label budget. High-cardinality identifiers live on traces and in the accounting store; metrics carry tenant, deployment and outcome. This decision, made late, is an emergency migration.
- Degradation ladder: the pre-agreed order of what is shed — rerank → cheaper model → cache-only → read-only → queue — decided in daylight and executed at 3 a.m.
- Incident response for probabilistic systems: how you tell "the model changed" from "our prompt changed" from "the corpus changed"; the value of pinned versions and eval baselines in making that distinguishable at all.
- Post-mortems: blameless, with a timeline, contributing factors, and action items whose completion rate is itself a tracked metric.
- Capacity planning: forecasting against provider quotas and GPU lead times; headroom against the provider limit as the operative signal rather than CPU (Phase 05).
- Cost governance / FinOps: attribution, budgets, forecasting, and cost per successful action as the unit economic; cost circuit breakers as an availability control.
The lab
| Lab | You build | Proves you understand |
|---|---|---|
| 01 — The SRE Console | an event-ratio SLI engine with an explicit validity predicate; rolling-window error budgets with per-layer allocation; multi-window multi-burn-rate alerting with derived thresholds and a minimum-volume guard; an OTel-shaped span tree for an agent run with GenAI attributes, and a query that answers "where did the latency go?"; a cardinality budget checker that fails a metric definition exceeding its series limit; a degradation-ladder executor driven by burn rate; a cost-per-successful-action meter with a per-tenant circuit breaker; and a capacity forecaster projecting headroom against a provider quota with a lead-time alert | that operating a probabilistic system requires different signals, and that the discipline is in choosing what not to page on |
132 tests, all green. Test contract: an alert fires only when both windows agree; a low-traffic window does not fire on a single failure; the budget never goes negative and overspend is surfaced separately; the span tree reconstructs a run exactly; a metric exceeding the cardinality budget is rejected at definition time; the ladder sheds in the declared order; and the forecaster alerts with enough lead time to start a procurement conversation.
Documents
| Document | For |
|---|---|
| WARMUP.md | zero to principal on SRE for probabilistic systems — first principles, then the interview answers |
| HITCHHIKERS-GUIDE.md | the fast orientation: what the pieces are and how they fit |
| DEEP-DIVE.md | the mechanisms, in detail, with the failure modes |
| PRINCIPAL-DEEP-DIVE.md | the trade-offs you own at principal level |
| CORE-CONTRIBUTOR.md | what it takes to work on OpenTelemetry, Prometheus or a tracing backend |
| STAFF-NOTES.md | judgment, review signal, war stories |
Deliverables checklist
- You can state an SLI for an AI platform and defend excluding answer quality from it.
- You can derive a burn-rate threshold from a budget-burn tolerance.
- You can name the four golden signals plus the three AI-specific ones.
- You can design a span tree for an agent run and say what each span carries.
- You can compute the series count of a proposed metric and say whether it is affordable.
- You can write a degradation ladder for this platform, in order.
- You can explain how you distinguish a model change from a prompt change during an incident.
Key takeaways
- Correctness is a distribution. Availability is a hard SLO; quality is a gated objective.
- Never page on a distribution shift. Gate deploys on it and review it weekly.
- Cost and safety-block rate are golden signals here. Both can move while everything else is green.
- Traces, not logs. Non-determinism means you cannot re-run to reproduce.
- Decide the label budget early. Cardinality is the metric backend's cliff.
- Write the degradation ladder in daylight.
- Cost per successful action is the unit economic, and a cost circuit breaker is an availability control.
- Capacity is a forecast, because provider quota and GPU lead times are measured in weeks.