« Phase 00 · Warmup · Track Overview

Lab 01 — Platform Reference Model & Budget Calculator

The problem

Your Platform Product Owner needs three numbers by Thursday: what SLO can we offer, what does an agent action cost, and what stops a bad action. Today those answers are opinions. By the end of this lab they are functions with tests.

You will build the arithmetic layer that every later phase leans on — availability composition, error budgets, burn-rate alerting, latency budgets, loop reliability, cost — and then the piece that turns arithmetic into architecture: an admission pipeline that runs all five layers' checks against a proposed action and reports every layer that would have denied it.

What you build

#ComponentWhat it does
1series_availability, parallel_availability, correlated_parallel_availabilitycompose dependency availabilities, including the common-mode term that ruins naive redundancy math
2Component, PlatformModelthe five-layer model, with the degradable flag that separates request availability from quality availability, plus weakest_links to rank where to spend
3ErrorBudget, BudgetLedgerbudget from an SLO and window, allocation across layers, consumption that never goes negative, and the four-state error-budget policy
4burn_rate, burn_rate_threshold, MultiWindowAlertPolicyderive 14.4 instead of memorizing it; fire only when a long and a short window agree
5LatencyStage, LatencyBudgetper-stage allocation where a parallel group contributes its max, headroom, fits_fallback, and a degradation ladder (shed_order, shed_until_fits)
6loop_success, effective_step_probability, max_steps_for_target\( p^n \), retries, and the inversion that tells a team how many steps their target can afford
7TokenPrices, CostModelthree-tier token cost, the quadratic scratchpad, cost per successful action, cache arithmetic
8AdmissionPipelinefive layers of independent checks; returns all denials, the primary one the caller sees, and defence_depth — the number of distinct layers that denied

Key concepts

ConceptWhere it shows upWhy it matters
Series compositionseries_availabilityevery serial dependency you add makes the platform worse; unavailabilities approximately add
Degradable dependencyComponent.degradablethe highest-leverage availability move in the phase — same components, an order of magnitude less downtime
Common-mode correlationcorrelated_parallel_availabilitytwo replicas in one region are not independent; a 20% common-mode term costs you two nines
Error budgetErrorBudget, BudgetLedgerthe shared instrument of two-in-a-box; the policy is a pure function of budget remaining
Multi-window burn rateMultiWindowAlertPolicyurgency from the short window, confirmation from the long one
HeadroomLatencyBudget.headroom_msthe headroom line is the fallback decision
\( p^n \)loop_success20 steps at 95% is 36%; you fix architecture, not prompts
Quadratic scratchpadCostModel.total_input_tokens10 steps at 2 000 tokens each is 100 000 input tokens, not 20 000
Cost per successful actioncost_per_successful_action_microsmakes quality a cost lever
Defence depthAdmissionResult.defence_depth"defence in depth" becomes a number you can assert on

Files

FileRole
lab.pyyour implementation — every # TODO
solution.pyreference; python solution.py prints the full worked example
test_lab.py103 tests: happy path, malformed input, boundaries, security, invariants, determinism
requirements.txtpytest — everything else is stdlib

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py — red until you implement
LAB_MODULE=solution pytest test_lab.py -v   # the reference — must be green
python solution.py                          # the worked example

Success criteria

  • All 103 tests green against your lab.py.
  • weakest_links is deterministic under ties (sort by unavailability desc, then name asc).
  • A budget consumed to exactly its limit reports freeze, not reliability-focus — you used EPSILON, not 0.0.
  • A burn rate mathematically equal to 14.4 fires. (1 - 0.999 is 0.0009999999999998899; a naive >= misses the boundary.)
  • A parallel latency group contributes its max: the 3-second budget commits 2 370 ms, not 2 490 ms.
  • fits_fallback(630) is true and fits_fallback(631) is false — the headroom boundary is inclusive.
  • The dual-control check counts distinct approvers and excludes the agent itself.
  • The injected-payment scenario returns denials from four distinct layers.

How this maps to the real stack

This labThe real thingWhat we simplified
PlatformModela dependency graph in a reliability model (or an architecture review spreadsheet)real dependency graphs are DAGs with fan-out, not a flat chain; real availability is measured from SLI data, not assumed
ErrorBudget / BudgetLedgerGoogle SRE error budgets; Nobl9, Datadog SLOs, Grafana SLO, Azure Monitor SLO workbooksreal budgets are computed continuously from event counts over a rolling window, not consumed by manual consume() calls
MultiWindowAlertPolicythe multi-window multi-burn-rate alerts in The SRE Workbook, implemented as Prometheus recording + alerting rulesproduction rules precompute burn rates as recording rules; ours evaluates a callable
LatencyBudgeta latency budget in a design doc, enforced by per-hop timeouts in Envoy/Istio and client configswe plan against a sum of p95s; real systems also model queueing and use hedged requests
loop_successthe reliability argument behind step budgets in LangGraph / Bedrock AgentCore / ADK runtimesreal per-step probabilities are measured per tool from traces, not assumed uniform
CostModeltoken accounting in an LLM gateway (LiteLLM, Azure APIM AI policies, Kong AI Gateway)real gateways read usage from provider responses and reconcile against billing exports
AdmissionPipelinea chain of PEPs: APIM policy → control-plane authorization (OPA/Cedar) → kernel budgets → retrieval authorization → action-gateway contract checksreal PEPs are distributed across services and processes; ours runs them in one function so you can see the ordering. Real ones also short-circuit — ours deliberately does not, so defence_depth is measurable

The honest limits. This lab models steady-state availability. It says nothing about mean time to recovery, correlated multi-hour outages, or the human factors that dominate real incidents. It also assumes every component's availability is known — in practice, getting trustworthy per-component SLI data is most of the work, and the arithmetic is the easy part.

Extensions

  1. Make the model a DAG. Replace the flat component list with a graph of nodes and edges, support fan-out (a request that calls three tools in parallel and needs two), and compute availability by enumerating minimal cut sets.
  2. Continuous budget consumption. Replace BudgetLedger.consume with a stream of (timestamp, good, total) buckets and compute a rolling-window budget, so policy_state changes over time. Then implement the alert policy against the same buckets.
  3. Queueing. Add a utilization field to LatencyStage and inflate its contribution by the M/M/1 factor \( 1/(1-\rho) \) — then watch the budget become infeasible at 80% utilization, which is the real reason capacity planning exists.
  4. Measured p per tool. Feed the loop-reliability model from a table of per-tool success rates and compute a task's success probability from its actual tool sequence.
  5. Emit evidence. Have AdmissionPipeline.evaluate return an audit record (decision, all denials, the policy inputs, a version stamp) and hash-chain successive records. That is Phase 10 in miniature.

Interview / resume bullets

  • "Built the platform's availability and error-budget model: composed a six-component request path, identified retrieval as convertible from a serial to a degradable dependency, and raised modelled read-path availability from 99.10% to 99.66% — 6 h 28 m to 2 h 27 m of monthly downtime — without adding hardware."
  • "Derived our alerting thresholds from a stated budget-burn tolerance rather than copying 14.4, and implemented multi-window multi-burn-rate rules so pages require a sustained burn confirmed by a live short window."
  • "Introduced cost per successful action as the platform's unit economic, which reframed evaluation investment as a cost-reduction programme: raising task success from 70% to 90% cut effective unit cost 22%."
  • "Made 'defence in depth' measurable: the admission path reports every layer that would deny an action, and we require at least two independent denials for any money-moving tool."