« Phase 00 · Warmup · Track Overview
Lab 01 — Platform Reference Model & Budget Calculator
The problem
Your Platform Product Owner needs three numbers by Thursday: what SLO can we offer, what does an agent action cost, and what stops a bad action. Today those answers are opinions. By the end of this lab they are functions with tests.
You will build the arithmetic layer that every later phase leans on — availability composition, error budgets, burn-rate alerting, latency budgets, loop reliability, cost — and then the piece that turns arithmetic into architecture: an admission pipeline that runs all five layers' checks against a proposed action and reports every layer that would have denied it.
What you build
| # | Component | What it does |
|---|---|---|
| 1 | series_availability, parallel_availability, correlated_parallel_availability | compose dependency availabilities, including the common-mode term that ruins naive redundancy math |
| 2 | Component, PlatformModel | the five-layer model, with the degradable flag that separates request availability from quality availability, plus weakest_links to rank where to spend |
| 3 | ErrorBudget, BudgetLedger | budget from an SLO and window, allocation across layers, consumption that never goes negative, and the four-state error-budget policy |
| 4 | burn_rate, burn_rate_threshold, MultiWindowAlertPolicy | derive 14.4 instead of memorizing it; fire only when a long and a short window agree |
| 5 | LatencyStage, LatencyBudget | per-stage allocation where a parallel group contributes its max, headroom, fits_fallback, and a degradation ladder (shed_order, shed_until_fits) |
| 6 | loop_success, effective_step_probability, max_steps_for_target | \( p^n \), retries, and the inversion that tells a team how many steps their target can afford |
| 7 | TokenPrices, CostModel | three-tier token cost, the quadratic scratchpad, cost per successful action, cache arithmetic |
| 8 | AdmissionPipeline | five layers of independent checks; returns all denials, the primary one the caller sees, and defence_depth — the number of distinct layers that denied |
Key concepts
| Concept | Where it shows up | Why it matters |
|---|---|---|
| Series composition | series_availability | every serial dependency you add makes the platform worse; unavailabilities approximately add |
| Degradable dependency | Component.degradable | the highest-leverage availability move in the phase — same components, an order of magnitude less downtime |
| Common-mode correlation | correlated_parallel_availability | two replicas in one region are not independent; a 20% common-mode term costs you two nines |
| Error budget | ErrorBudget, BudgetLedger | the shared instrument of two-in-a-box; the policy is a pure function of budget remaining |
| Multi-window burn rate | MultiWindowAlertPolicy | urgency from the short window, confirmation from the long one |
| Headroom | LatencyBudget.headroom_ms | the headroom line is the fallback decision |
| \( p^n \) | loop_success | 20 steps at 95% is 36%; you fix architecture, not prompts |
| Quadratic scratchpad | CostModel.total_input_tokens | 10 steps at 2 000 tokens each is 100 000 input tokens, not 20 000 |
| Cost per successful action | cost_per_successful_action_micros | makes quality a cost lever |
| Defence depth | AdmissionResult.defence_depth | "defence in depth" becomes a number you can assert on |
Files
| File | Role |
|---|---|
| lab.py | your implementation — every # TODO |
| solution.py | reference; python solution.py prints the full worked example |
| test_lab.py | 103 tests: happy path, malformed input, boundaries, security, invariants, determinism |
| requirements.txt | pytest — everything else is stdlib |
Run
pip install -r requirements.txt
pytest test_lab.py -v # your lab.py — red until you implement
LAB_MODULE=solution pytest test_lab.py -v # the reference — must be green
python solution.py # the worked example
Success criteria
-
All 103 tests green against your
lab.py. -
weakest_linksis deterministic under ties (sort by unavailability desc, then name asc). -
A budget consumed to exactly its limit reports
freeze, notreliability-focus— you usedEPSILON, not0.0. -
A burn rate mathematically equal to 14.4 fires. (
1 - 0.999is0.0009999999999998899; a naive>=misses the boundary.) - A parallel latency group contributes its max: the 3-second budget commits 2 370 ms, not 2 490 ms.
-
fits_fallback(630)is true andfits_fallback(631)is false — the headroom boundary is inclusive. - The dual-control check counts distinct approvers and excludes the agent itself.
- The injected-payment scenario returns denials from four distinct layers.
How this maps to the real stack
| This lab | The real thing | What we simplified |
|---|---|---|
PlatformModel | a dependency graph in a reliability model (or an architecture review spreadsheet) | real dependency graphs are DAGs with fan-out, not a flat chain; real availability is measured from SLI data, not assumed |
ErrorBudget / BudgetLedger | Google SRE error budgets; Nobl9, Datadog SLOs, Grafana SLO, Azure Monitor SLO workbooks | real budgets are computed continuously from event counts over a rolling window, not consumed by manual consume() calls |
MultiWindowAlertPolicy | the multi-window multi-burn-rate alerts in The SRE Workbook, implemented as Prometheus recording + alerting rules | production rules precompute burn rates as recording rules; ours evaluates a callable |
LatencyBudget | a latency budget in a design doc, enforced by per-hop timeouts in Envoy/Istio and client configs | we plan against a sum of p95s; real systems also model queueing and use hedged requests |
loop_success | the reliability argument behind step budgets in LangGraph / Bedrock AgentCore / ADK runtimes | real per-step probabilities are measured per tool from traces, not assumed uniform |
CostModel | token accounting in an LLM gateway (LiteLLM, Azure APIM AI policies, Kong AI Gateway) | real gateways read usage from provider responses and reconcile against billing exports |
AdmissionPipeline | a chain of PEPs: APIM policy → control-plane authorization (OPA/Cedar) → kernel budgets → retrieval authorization → action-gateway contract checks | real PEPs are distributed across services and processes; ours runs them in one function so you can see the ordering. Real ones also short-circuit — ours deliberately does not, so defence_depth is measurable |
The honest limits. This lab models steady-state availability. It says nothing about mean time to recovery, correlated multi-hour outages, or the human factors that dominate real incidents. It also assumes every component's availability is known — in practice, getting trustworthy per-component SLI data is most of the work, and the arithmetic is the easy part.
Extensions
- Make the model a DAG. Replace the flat component list with a graph of nodes and edges, support fan-out (a request that calls three tools in parallel and needs two), and compute availability by enumerating minimal cut sets.
- Continuous budget consumption. Replace
BudgetLedger.consumewith a stream of(timestamp, good, total)buckets and compute a rolling-window budget, sopolicy_statechanges over time. Then implement the alert policy against the same buckets. - Queueing. Add a
utilizationfield toLatencyStageand inflate its contribution by the M/M/1 factor \( 1/(1-\rho) \) — then watch the budget become infeasible at 80% utilization, which is the real reason capacity planning exists. - Measured
pper tool. Feed the loop-reliability model from a table of per-tool success rates and compute a task's success probability from its actual tool sequence. - Emit evidence. Have
AdmissionPipeline.evaluatereturn an audit record (decision, all denials, the policy inputs, a version stamp) and hash-chain successive records. That is Phase 10 in miniature.
Interview / resume bullets
- "Built the platform's availability and error-budget model: composed a six-component request path, identified retrieval as convertible from a serial to a degradable dependency, and raised modelled read-path availability from 99.10% to 99.66% — 6 h 28 m to 2 h 27 m of monthly downtime — without adding hardware."
- "Derived our alerting thresholds from a stated budget-burn tolerance rather than copying 14.4, and implemented multi-window multi-burn-rate rules so pages require a sustained burn confirmed by a live short window."
- "Introduced cost per successful action as the platform's unit economic, which reframed evaluation investment as a cost-reduction programme: raising task success from 70% to 90% cut effective unit cost 22%."
- "Made 'defence in depth' measurable: the admission path reports every layer that would deny an action, and we require at least two independent denials for any money-moving tool."