« Phase 00 · Warmup · Track Overview

Deep Dive — Mechanism & Internals

The core-contributor lens: the data structures, the algorithms, the invariants, the complexity, and a step-by-step trace of the lab's admission pipeline.

Table of Contents

• 1. The composition engine
• 2. Why degradable is a property of the edge, not the node
• 3. The correlated-redundancy mixture, derived
• 4. The budget ledger's invariants
• 5. Burn-rate evaluation as a two-predicate conjunction
• 6. The latency budget's group algebra
• 7. The admission pipeline: structure and trace
• 8. Complexity and determinism
• 9. Floating point, and why EPSILON exists

1. The composition engine

The entire availability model reduces to two folds over a sequence of probabilities:

series   = reduce(lambda acc, a: acc * a,          values, 1.0)
parallel = 1.0 - reduce(lambda acc, a: acc * (1-a), values, 1.0)

Two details that look trivial and are not:

The identity elements are different. An empty series is 1.0 (a platform with no dependencies never fails because of a dependency). An empty parallel group is 0.0 (a redundancy group with no members cannot serve). Getting these backwards produces a model that reports a perfect platform when a config file is empty — a failure mode that has shipped.

Validation happens per element, not on the aggregate. series_availability([1.5, 0.5]) returns 0.75 if you only validate the result, which is silently wrong. Each element is checked against [0, 1] on the way through.

2. Why degradable is a property of the edge, not the node

The Component.degradable flag looks like it describes the component. It does not — it describes the calling code's behaviour when that component fails.

The same vector store is:

• serial if the retrieval step does results = vector.search(q) and lets the exception propagate;
• degradable if it does try: results = vector.search(q) except: results = []; degraded = True and the answer path tolerates an empty dense result because BM25 also ran.

This is why the lab's model exposes with_degradable(*names) as a transformation: you are not discovering a property, you are recording an architectural decision. It is also why the model returns two numbers:

request_availability()  # product over non-degradable only  — "did we answer at all?"
quality_availability()  # product over everything           — "did we answer well?"

Returning one number is the bug. A platform that reports 99.9% request availability while silently serving BM25-only answers 3% of the time has hidden a defect in a metric, and the first person to notice will be a customer. The invariant the lab asserts:

$$A_{\text{quality}} \le A_{\text{request}}$$

with equality exactly when nothing is degradable.

3. The correlated-redundancy mixture, derived

Independent redundancy multiplies unavailabilities: \( u_{\text{pair}} = \prod_i u_i \). Real replicas share failure modes. Model the failure as a mixture of two regimes:

• with probability \( c \) (common mode), a failure hits all members together — the group is down whenever a "typical" member would be down, so its unavailability is the mean \( \bar{u} \);
• with probability \( 1-c \), failures are independent and multiply.

$$u_{\text{group}} = c,\bar{u} + (1-c)\prod_i u_i$$

Sanity checks the lab tests:

• \( c = 0 \) → \( \prod u_i \) — exactly parallel_availability.
• \( c = 1 \) → \( \bar{u} \) — redundancy buys nothing; for identical members the group's availability equals a single member's.
• Monotone in \( c \): more correlation is never better.

The number that matters: two components at \( A = 0.999 \).

One percent of common mode costs a full nine. That single row is why multi-region and multi-provider architectures earn their complexity while multi-replica-same-cluster designs often do not, and it is why the lab makes you write the mixture rather than the naive product.

4. The budget ledger's invariants

BudgetLedger holds three pieces of state: the ErrorBudget, a per-layer allocations map, and a per-layer _consumed map. Four invariants:

1. Allocations sum to the total. Enforced at construction by allocate(), which rejects weights that do not sum to 1 within tolerance. Silent renormalization would let a typo change the platform's budget.
2. Consumption is monotone non-decreasing. consume() rejects negative minutes. A "correction" that subtracts minutes is a different operation (an incident reclassification) and should be modelled as such, not as a negative consume.
3. Reported remaining is never negative. remaining() clamps at zero and the overspend is surfaced separately by overspend_for(). A negative remainder on one layer would otherwise cancel a positive remainder on another and make the platform total lie.
4. policy_state is a pure function of fraction_remaining(). No hidden state, no time. This is what makes the error-budget policy enforceable: two people reading the same number reach the same conclusion, mechanically.

The overspend/remaining split is worth dwelling on. Consider two layers, each allocated 20 minutes; the model layer burns 35, the kernel burns 0.

The naive version says "we're fine." The lab's version says "we're technically inside the total, but one layer is 75% over its allocation" — which is the sentence that starts the right conversation.

5. Burn-rate evaluation as a two-predicate conjunction

for rule in rules:
    long_burn  = burn_rate(bad_ratio_over(rule.long_window_hours),  slo)
    short_burn = burn_rate(bad_ratio_over(rule.short_window_hours), slo)
    if long_burn >= floor and short_burn >= floor:
        fired.append(rule)

The conjunction is doing two different jobs, and it is worth naming both:

• The long window controls precision. It is the statement "this has been going on long enough to be real." A one-minute blip cannot accumulate enough bad events in an hour-long window to reach a burn rate of 14.4.
• The short window controls recall on the way down. Once the incident is fixed, the long window keeps looking bad for up to an hour (it still contains the bad events). Without the short-window conjunct, the page stays firing long after the problem is gone — which is how teams learn to ignore pages.

The lab evaluates rules in declaration order and returns them in that order, so fast-burn precedes medium-burn precedes slow-burn. highest_severity then folds the list to a single "page" | "ticket" | None. Keeping the list rather than only the max matters for the audit trail: "which rules fired" is a different question from "did we page."

A subtlety in the lab's design: bad_ratio_over is a callable, not a dict. That is deliberate. In production the windows are queried from a metrics backend, and modelling the dependency as a function makes the policy testable with a closure and makes it obvious that each rule performs two queries. A naive implementation that queries once per rule per evaluation does 6 queries per cycle; a real one precomputes burn rates as recording rules.

6. The latency budget's group algebra

committed = Σ_{ungrouped} p95_i  +  Σ_{groups g} max_{i ∈ g} p95_i

Stages without a group are serial and add. Stages sharing a group string run concurrently and contribute only the slowest. The lab's 3-second example commits 2 370 ms; if the parallel group were summed it would commit 2 490 ms and the headroom would drop from 630 ms to 510 ms — enough to change the fallback decision. That 120 ms is the entire value of a Promise.all, made visible.

The degradation ladder is a sort plus a scan:

shed_order()      -> sheddable stages, key = (-p95_ms, name)     # deterministic under ties
shed_until_fits() -> scan the ladder, stopping as soon as fits_fallback(t, shed) is True

shed_until_fits has one deliberately awkward property: it can return the entire ladder without the fallback fitting, and the caller must re-check. The alternative — raising, or returning None — hides the useful information ("we shed everything and it still doesn't fit"), which is exactly the finding you want during capacity planning. The tests assert this case.

7. The admission pipeline: structure and trace

The pipeline is five pure functions, each ProposedAction -> List[Denial], run unconditionally and concatenated:

_channel_checks         → users_and_channels
_control_plane_checks   → control_plane        (short-circuits inside itself if unregistered)
_kernel_checks          → agent_kernel
_knowledge_checks       → knowledge_foundation
_gateway_checks         → action_gateway

Then a stable sort by (LAYERS.index(layer), code) and a wrap into AdmissionResult.

Why run all five instead of short-circuiting? A production PEP chain short-circuits — it is faster and it avoids leaking why a request failed. The lab does not, because AdmissionResult.defence_depth is the phase's teaching instrument: it turns "we have defence in depth" into an integer you can assert on in a test. In production you would run all checks in a shadow mode alongside the short-circuiting path and alert when defence_depth == 1 for a money-moving tool — that is a genuinely useful control, and it is why the lab is built this way.

Worked trace

Input — the injected-payment scenario:

ProposedAction(
    tenant="retail",                       # from the verified token
    agent_id="collections-01",
    tool="payments.release",
    channel="ivr",
    amount_micros=2_000_000_000,           # 2 000 USD-equivalent units
    resource_tenant="wholesale",
    derived_from_untrusted_content=True,
    retrieved_tenants=("retail", "wholesale"),
    step_index=3,
)

Registry entry for collections-01: tenant retail, permitted tools ("crm.read", "collections.note"), granted scopes ("crm.read", "collections.write"), max_action_amount_micros=0, evaluation fresh. Pipeline config: dual-control threshold 100 000 000, approval-capable channels {web, teams}, payments.release requires scope payments.release, side-effecting tools {payments.release, collections.note}.

Eight denials across four distinct layers. Sorted output:

[users_and_channels    ] CHANNEL_CANNOT_APPROVE
[control_plane         ] TOOL_NOT_PERMITTED
[knowledge_foundation  ] CROSS_TENANT_RETRIEVAL
[knowledge_foundation  ] UNTRUSTED_INSTRUCTION_SOURCE
[action_gateway        ] ACTION_LIMIT_EXCEEDED
[action_gateway        ] DUAL_CONTROL_REQUIRED
[action_gateway        ] SCOPE_MISSING
[action_gateway        ] TENANT_MISMATCH

primary = CHANNEL_CANNOT_APPROVE (the earliest layer). defence_depth = 4.

Note what the kernel did not catch: nothing. Step 3 of 25, no cost overrun. That is honest and instructive — the kernel's job is budget, not authorization, and a design that expects the kernel to stop a scope violation has misassigned responsibility.

Two boundary behaviours worth memorizing

• Dual control counts distinct approvers excluding the agent. approvals=("alice","alice") is one approver. approvals=("payments-01","alice") is one approver, because an agent cannot approve its own action. Both are tested; both are real bugs that have shipped.
• The dual-control threshold is inclusive. amount == threshold requires approval. Off-by-one on a monetary threshold is a compliance finding, not a rounding issue.

8. Complexity and determinism

Determinism sources. No clock, no RNG, no UUIDs, no dict-iteration-order dependence in any output: weakest_links sorts with an explicit tie-break, denials sort by (layer index, code), and shed_order breaks ties on name. The test suite asserts repeat-invocation equality for both evaluate and weakest_links.

9. Floating point, and why EPSILON exists

Two boundaries in this lab are unreachable with naive comparisons:

>>> 1 - 0.999
0.0009999999999998899          # NOT 0.001
>>> 0.0144 / (1 - 0.999)
14.400000000001585             # depends on which side you compute from
>>> (1 - 0.999) * 43200
43.19999999999952

Depending on the expression, a mathematically exact 14.4 lands either side of the threshold. In the lab's MultiWindowAlertPolicy, burn_rate(0.0144, 0.999) computes as 14.399999999999986 — just under — so a naive >= does not fire the page. Likewise a budget consumed to exactly its limit leaves a residue of ~3.6 × 10⁻¹⁴, so fraction_remaining() <= 0.0 is False and the policy reports reliability-focus instead of freeze.

Both are real production bugs of the "alert never fires and nobody notices for a quarter" kind. The lab fixes them with a single documented tolerance:

EPSILON = 1e-9
...
if fraction <= EPSILON: return "freeze"
floor = rule.threshold * (1.0 - EPSILON)

The general rule for this kind of code: never compare a ratio of measured floats to a constant with a bare >=. Either compare with a relative tolerance, or restructure to compare integers (counts of bad events against a computed integer allowance) — which is what a production SLO implementation does, and which is the right extension exercise.