« Phase 16 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. Where the shared pager actually bites
• 2. Change classification, and who assigns it
• 3. The exception register, mechanically
• 4. When the budget number is wrong
• 5. Reversibility is a spectrum
• 6. The falsifier test, and its failure modes
• 7. Escalation without losing
• 8. ADR granularity and decay
• 9. Making a design review not adversarial
• 10. ORR evidence, and how it is gamed
• 11. Testing the alerts
• 12. Incident command under two-in-a-box
• 13. Action-item gaming
• 14. Forum preparation
• 15. Standards as controls
• 16. Failure modes

1. Where the shared pager actually bites

The mechanic that sounds performative and is not. Four things change when the product owner carries the pager:

Roadmap decisions change without lobbying. An engineer arguing for reliability work is arguing for their own convenience, as far as anyone can tell. A product owner who was woken at 03:40 arrives at the same conclusion independently, and nobody had to persuade them.

Toil becomes visible. The manual step that "only takes five minutes" is invisible in a backlog and unmissable at 3 a.m. Shared on-call is the fastest way to get toil prioritized.

Alert quality becomes a shared concern. A noisy alert is now costing both owners sleep, which changes who is willing to spend a sprint on alert hygiene.

Empathy runs both ways. The engineer also sees the customer escalation the product owner handles, because the shared surface includes both.

The practical constraints, because "shared on-call" is easy to say and has real mechanics:

That last row is the one to enforce. Shared on-call on a noisy rota is not a leadership practice, it is a punishment, and the correct response to "the PO cannot handle these pages" is usually that the pages are bad (Phase 14).

2. Change classification, and who assigns it

The budget policy permits change classes, which means something must assign the class — and that is where the mechanism can be quietly defeated.

   EMERGENCY_FIX < RELIABILITY < BUG_FIX < CONFIG < FEATURE < EXPERIMENT
                                                    ↑
                                     the boundary everything migrates across

The failure is predictable: during a freeze, a feature is relabelled a bug fix. Nobody is lying — "this fixes the user's inability to do X" is a defensible reading — and the policy is now decorative.

Three defences, in increasing strength:

Classify at PR open, not at merge. A class assigned before the freeze cannot be re-assigned by the freeze.

The class is a label with an owner. Whoever opens the PR proposes it; the other owner confirms it during a freeze. One-person classification during a freeze is self-service.

Audit the reclassification rate. How many PRs changed class after opening, and when? A spike during freezes is the signal, and it is more useful than arguing about any individual PR.

And the definitional line that resolves most cases: a bug fix restores intended behaviour; a feature adds behaviour. If the behaviour never worked, it is a feature — which is unpopular and correct.

3. The exception register, mechanically

   grant(change_class, reason, approvers, ttl)
     → both owners?            no → refuse
     → reason substantive?     no → refuse
     → rate within window?     no → refuse, and say "renegotiate the SLO"
     → record with an expiry

Four design decisions worth defending:

Scoped to a change class, not open-ended. An exception permitting "features" is narrower than one permitting "anything", and the narrowest useful scope is a specific change — which is the version to prefer where it is practical.

Consumed on use. A single-use exception cannot become a standing waiver by accident. If the change fails to ship, request another — that friction is the mechanism working.

Expiring. Seven days is a reasonable default. An exception outliving the situation that justified it is a policy change nobody agreed to.

Rate-limited, with a stated response. Refusing the third exception is not the useful part; saying why is: the policy is being replaced by a habit, so renegotiate the SLO. A rate limit that just says "no" produces resentment; one that names the actual problem produces a conversation.

The register's real output is not the exceptions. It is the rate, reviewed quarterly by both owners, and the question it forces: is our SLO the number we actually hold?

4. When the budget number is wrong

The policy assumes the budget is trustworthy. Sometimes it is not, and the disputes are predictable:

All five are Phase 14 problems surfacing as a Phase 16 argument, and the failure mode is re-litigating the measurement during a freeze — which is both unresolvable in the moment and corrosive.

The mechanism that prevents it:

Disputes about the number are handled outside the freeze. The policy applies to the number as measured; a dispute goes to a scheduled review with data. That is unsatisfying in the moment and it is the only rule that keeps the policy usable.

Recompute historically when the predicate changes. If the validity predicate is genuinely wrong, fix it and recompute — but the recomputation applies from the next period, not retroactively to unfreeze today.

Review the SLO quarterly. Which is where "we breach every month" belongs. A target breached every month is not a target, and reviewing it on a schedule means nobody has to raise it under pressure.

5. Reversibility is a spectrum

The decision router treats reversibility as a boolean. It is not, and the boundary cases are where judgment lives:

The useful reframe is cost of reversal, and a rule of thumb: if reversing would take longer than a sprint, treat it as irreversible.

Two second-order traps:

Accumulation. Ten reversible decisions can compose into an irreversible architecture. Each was a two-way door; the building has one exit. The mitigation is to notice a sequence of related decisions and treat the sequence as one irreversible decision — which is exactly what an ADR is for.

Reversible-in-principle. A change that could be rolled back but never will be, because by then forty things depend on it. Ask "would we actually reverse this?" rather than "could we?"

6. The falsifier test, and its failure modes

"What would change your mind?" is the best question in the protocol and it has three failure modes worth recognizing.

The unfalsifiable falsifier. "Evidence that it is safe" is not a falsifier — it names no experiment. Push for something specific: what result, from what test, at what threshold?

The moving falsifier. The experiment runs, the result comes back, and the falsifier changes. "Well, that test did not account for..." The defence is to agree what each outcome implies before running the experiment, in writing. Without that agreement the experiment settles nothing.

The asymmetric falsifier. One owner names something cheap to test, the other something expensive. That is not bad faith — it may reflect a genuine asymmetry in the risk — but it needs naming, because otherwise the cheap experiment runs, the expensive one does not, and the disagreement resolves by cost rather than by evidence.

And the genuinely hard case: a values disagreement dressed as a factual one. Both owners name falsifiers, the experiments run, both are satisfied by the data, and they still disagree — because the real disagreement was about acceptable residual risk and the data was never going to touch it. The tell is that the argument moves to a new factual question immediately, and the right response is to say so plainly: "I think we agree on the facts and disagree on the risk appetite. Let's escalate that."

7. Escalation without losing

Escalation is expensive: it costs both owners credibility if it looks like they cannot work together. Which produces a bad equilibrium — nobody escalates, and disagreements resolve by attrition.

Four practices that make escalation normal:

Escalate the decision, not the person. "We have a values disagreement about autonomy for irreversible actions and we need whoever owns that risk to choose" is a different sentence from "Omar and I cannot agree".

Both positions, written by their holders. Not a summary by one of them — that is a recommendation with extra steps, and the other owner will experience it as one.

Pre-agree who the escalation point is. Deciding who decides during a disagreement adds a second disagreement.

Escalate early, not at the end. An escalation after three weeks of stalemate reads as failure. An escalation in the first meeting reads as good classification.

And the framing that makes it work culturally: a values disagreement is not a failure of the pair. It is a decision that belongs above them, and recognizing that quickly is a sign the model is working. A pair that never escalates is either identical in judgment or one of them is not saying what they think.

8. ADR granularity and decay

Two failure modes, opposite directions.

Too many. Forty ADRs a quarter, most recording things that were never in doubt. Nobody reads them, so the important ones are lost in the volume, and writing them becomes a chore that gets skipped exactly when it matters.

Too few. Zero ADRs and an architecture that lives in two people's memory — which is the thing the model exists to prevent.

The heuristic: write an ADR when the decision is hard to reverse or expensive to re-litigate. Five to ten a quarter is healthy for a platform.

Then decay, which is the failure mode nobody plans for. An ADR accepted in 2026 describes a decision whose context has moved. Three mitigations:

The third is the strongest and the least common: a comment in the module naming the ADR that governs it means the next person to change that code encounters the reasoning.

And the anti-pattern to name: the ADR written after the fact to justify a decision already made. It is recognizable — one option, no negatives, and the "context" describes the solution. It is worse than no ADR, because it looks like a decision record and is a rationalization.

9. Making a design review not adversarial

The mechanism is a checklist. The culture determines whether the checklist helps or produces documents optimized to survive review.

Five practices:

Publish the rules. An author who knows the standing red flags fixes the design before the review. That is the entire value, and it is lost if the checklist lives in the reviewer's head.

Review the design, not the designer. "This design has no stated blast radius" rather than "you have not thought about failure".

The reviewer's job is to find what hurts at 3 a.m., which is a shared goal rather than an opposing one. Saying so out loud at the start of a review changes the room.

Separate blockers from opinions. A blocker is on the standing list. Everything else is advice the author may decline, and being explicit about which is which stops a review becoming a preference negotiation.

Bring the review earlier. A review of a finished design is a defence; a review of a draft is help. The blockers are cheapest to fix before the implementation exists.

And the structural intervention from the lab: require a structured document. Prose can omit the blast radius without anybody noticing. A form with a field cannot, and the field being empty is a fact rather than a judgment — which takes the reviewer out of the position of having to notice.

10. ORR evidence, and how it is gamed

Requiring evidence is a large improvement over requiring a yes. It is not unfakeable:

Which is why a real ORR has a human panel, and why the reviewer should not be from the building team — the same independence argument as Phase 15's validation.

Two additional properties worth building:

Freshness rules per criterion. A rollback test from last week is evidence; from last year it is history. Attach a maximum age to the criteria where it matters.

Sample and re-verify. Pick one criterion per ORR at random and actually check the evidence end-to-end. The prospect of that check is what keeps the rest honest, and it costs twenty minutes.

And the cultural half: an ORR that never fails is not a gate. If every service passes first time, either the team is extraordinary or the ORR is a formality. A first-pass rate around 60–70% is a healthy gate; 100% is a form.

11. Testing the alerts

ORR row 2 — alerts tested by injecting failure — is the row that is skipped most and matters most. "An untested alert is a belief" is not rhetoric; the ways an alert silently fails to fire are numerous and none of them are visible in the configuration:

Every one of those looks correct on inspection. The only way to know is to make it fire.

The practice, and it is cheap:

   1. inject the failure (a fault-injection header, a killed pod, a synthetic 500 rate)
   2. observe the alert fire
   3. observe the PAGE arrive on the on-call device
   4. open the runbook link and check it describes this
   5. record all four as the ORR evidence

Step 3 is the one people skip, and it is where routing failures live. Step 4 is where you discover the runbook describes a system from two years ago.

Do it quarterly thereafter, not just at the ORR — alerting decays as the system changes, and a quarterly drill is the only thing that notices.

12. Incident command under two-in-a-box

The roles are standard (Phase 14). What two-in-a-box changes:

Either owner commands, depending on who is on the pager. Not "the engineer commands technical incidents" — that recreates the partition inside the incident.

The other owner is not automatically pulled in. Both awake for every incident burns both, and the point of shared on-call is coverage. Escalate deliberately, on stated criteria: severity, duration, or a decision that needs both.

The degradation ladder is pre-authorized. The commander executes it without asking, because it was agreed in daylight — which is exactly why it was agreed in daylight.

Comms are the product owner's strength, and this is where a non-engineer commander is often better: the stakeholder update is a genuine skill, and an engineer commanding tends to under-communicate while debugging.

Two specifics for an AI platform:

Quality incidents need a different tree. "The agent gave a bad answer" is not obviously an incident, and the first question is whether it is availability or quality — completely different work. Most new platforms have a runbook for the second and no way to distinguish it from the first.

"What changed?" needs the pins. Model, prompt, corpus, policy (Phase 15). Without them the incident ends in a shrug, which is a worse outcome than a long incident.

13. Action-item gaming

Tracking completion creates an incentive, and incentives get gamed. The three ways:

Trivial actions. "Add a comment to the code" completes easily and changes nothing. The completion rate rises and the platform does not improve.

Dropping the hard ones. Dropped items leave the denominator, so dropping the difficult action is rewarded by the metric.

Vague actions. "Improve monitoring" can be declared complete by anyone at any time.

Three defences:

The middle one is the strongest. An action item that does not connect to the incident's contributing factors is not an action item, and asking "would this have prevented it?" is a ten-second test.

And the honest framing to keep: the completion rate is a health indicator, not a target. The moment it becomes a target — reported upward, compared between teams — all three games appear. Watching it and asking about the exceptions is the use that survives.

14. Forum preparation

Each forum wants a different artifact, and the preparation is mostly selection rather than creation — because by this point in the track the artifacts already exist.

Which suggests the highest-leverage thing to build: a forum briefing generator that pulls the right artifacts for a named forum. It removes the preparation from the critical path, and it means the artifacts shown are the current ones rather than a snapshot somebody exported last month.

Three practices that make forums go well:

Bring the limitation before they find it. Every forum has one thing you cannot fully answer. Naming it first — "residency is proved two ways and here is what the topology model does not cover" — converts a challenge into a conversation.

Answer in their unit. Cyber thinks in blast radius, Model Risk in validation status, the CTTO in cost per action. Translating once, at the start, saves the whole meeting.

Send the artifact in advance. A forum reading a document live is a forum reading, not deciding.

15. Standards as controls

The phase's organizing idea, and the mechanics of applying it.

For each standard, ask: where in the path can this be enforced?

The pattern: find the function that already stands between the engineer and the outcome, and put the rule there. Not a new gate — an existing one.

Two properties that determine whether a control is accepted rather than resented:

The error message teaches. ValueError: side_effect is required is a rule. "A tool with no declared side-effect class cannot have a retry policy derived for it; declare one of read / write_idempotent / write_non_idempotent / irreversible" is a lesson, delivered at exactly the moment somebody is trying to learn it.

There is a paved road. A control that blocks with no compliant alternative is an obstacle. A control that blocks and points at the one-command module is a curriculum.

And the honest limit: not everything can be a control. "Write good ADRs" cannot be enforced by a function beyond the structural checks. For those, the mechanism is review and mentorship — which is why §9's culture section is not optional decoration.

16. Failure modes