« Phase 15 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. The federation problem
• 2. Join keys, and where they get lost
• 3. Retention tiers
• 4. Chaining across stores
• 5. Redaction without breaking the chain
• 6. Lineage at scale
• 7. Snapshotting a corpus
• 8. What re-execution actually needs
• 9. The inventory as a gate
• 10. Validation for a non-deterministic system
• 11. Residency evidence, precisely
• 12. Detecting a silent provider change
• 13. Exit testing
• 14. Control coverage as a test
• 15. Performance and volume
• 16. Failure modes

1. The federation problem

The lab puts every artifact in one graph. In production they are in six systems, and federating them is most of the work:

Three consequences.

The pack generator is a fan-out with partial failure. One store being slow or down means an incomplete pack, and the generator must distinguish "this artifact does not exist" from "this store did not answer" — they look identical to a naive implementation and mean completely different things.

Query latency is the slowest store. A pack that takes four minutes is one nobody generates proactively, so it only gets generated under pressure — which is when you discover it does not work.

Schema drift is independent. Six teams evolve six schemas. The generator needs a version-tolerant reader per source, and the honest approach is to store the pack once generated rather than re-deriving it later from schemas that have moved.

Which produces the design most banks land on: materialize the pack at action time for anything above a threshold, and keep the on-demand generator for everything else. Materializing costs storage and removes the federation problem for exactly the actions an examiner will ask about.

2. Join keys, and where they get lost

The key is one field. Losing it is easy, and every boundary is a place it goes:

The last row is the quiet one. A team adds trace_id to their events, the downstream store's schema does not have the column, and it is silently discarded. Nobody notices until a pack is generated, which is months later.

Two mitigations that work:

Contract-test the join key. A test per emitter that asserts the id survives a round trip through that store. Cheap, and it is the only thing that catches the dropped-column case.

Make it structurally impossible to omit. The artifact type requires it in its constructor, and the emit helper takes it from an ambient context rather than an argument — so forgetting it is a type error rather than an empty string.

And the propagation mechanism for async work: use span links rather than parent-child for a queue hop (Phase 14), and carry the id in the message envelope rather than the payload, so it survives a schema change to the payload.

3. Retention tiers

Three orders of magnitude apart, which makes them three different systems:

Four consequences worth planning for.

A pack for an old action legitimately lacks the trace. At three years the audit record and the inference record exist; the execution steps do not. That is a design decision, and it must be stated — a predicted gap is a decision, an unpredicted one is a finding.

Sampling must not touch evidence. Phase 14's sampling policy has to be category-based for exactly this reason: side-effecting actions, policy denials and guardrail blocks are 100%, forever, regardless of trace sampling.

Encryption keys must outlive the data. A record encrypted with a 2026 key must be readable in 2033. That needs key versioning in the record, an archival key store, and a rotation procedure somebody has tested — and it is a bigger problem than the chain.

Schema must be readable seven years later. You will read 2026 records with 2033 code. Version the schema from record one, never remove a field, and keep a reader for every version. The alternative is an archive you cannot open.

4. Chaining across stores

Phase 10 hash-chains one log. An evidence pack spans six, and they cannot share a chain because they are written independently and out of order.

The workable structure is a per-pack Merkle-ish chain over the artifacts, anchored externally:

   artifacts sorted causally
     → digest each
     → fold: head = H(head ‖ digest_i)
     → sign the head
     → publish the head hourly to a store you cannot write to

Which gives three properties:

The third is the one that matters and the one the lab cannot do. Without an external anchor, whoever can write the audit store can rewrite the pack and re-sign it — so the chain is tamper-evident against accident and tamper-resistant against nobody. Publishing the head somewhere outside your blast radius (a WORM store, another team's system, a public timestamp authority) is what closes it.

And the ordering subtlety: the chain must be computed over a canonical ordering, or two generators produce different heads for the same pack. Sort by (tick, artifact_id) and make it part of the specification.

5. Redaction without breaking the chain

A pack shown to an external examiner should not contain another customer's data. But redacting an artifact changes its digest, which breaks the chain — and a broken chain is exactly what you were trying to avoid.

The mechanism:

   original artifact  →  digest_i   (in the chain)
   redacted artifact  →  { fields kept, fields withheld: [names], original_digest: digest_i }

The redacted pack carries the original digest per artifact, so the chain still verifies, and the withheld field names are disclosed even though the values are not. Which is the important property: the examiner can see that something was withheld and what kind of thing it was, rather than receiving a pack that silently differs from the record.

Three rules that make it defensible:

Redact values, never fields. Removing a field entirely means the recipient cannot tell it existed.

Log the redaction. Who redacted, when, under what authority, for which recipient. The redaction is itself an action and it is auditable.

Never redact what the question was about. If the examiner asked about customer X, customer X's data is the answer. Redaction is for other customers whose data appears incidentally — usually in a retrieval record that returned several documents.

6. Lineage at scale

The lab walks a graph of ten artifacts. Production is millions of artifacts a day, and the two queries have very different costs:

The forward query is the hard one, and the naive implementation is a full scan. Three approaches:

An inverted index on derived_from. Document → artifacts that used it. A write-time cost per edge, and it makes the impact query a lookup. This is the right default.

Materialized impact sets per document, updated on write. Faster to read, more expensive to maintain, and it is what you need if the impact query runs interactively during a remediation.

A graph database. Neo4j or similar. Real, and it is a new operational dependency for a query you run rarely — usually not worth it below a very large scale.

And the property that makes the forward query answerable at all: document versions, not document ids. "Case note 991 was wrong" is not actionable; "case note 991 version 3 was wrong, versions 1, 2 and 4 were fine" bounds the remediation to decisions that used v3. Without versioning, a bad document taints its whole history.

7. Snapshotting a corpus

The pin everyone forgets, and it is forgotten because it is the hardest one to provide.

Three implementations, in increasing cost:

The first is the minimum and it is genuinely useful: it tells you which documents the agent saw, at which versions, which answers "what data did it use?"

It does not answer "would the same query return the same documents?", because the index changed — new documents may now rank higher. Only a snapshot answers that, and snapshots need the search layer to support them, which is a capability request with a long lead time. Ask for it early.

And the piece people miss even with snapshots: the embedding model is part of the retrieval configuration. Re-embedding the corpus with a new model changes the neighbourhood structure entirely, so an index snapshot taken with embedder v1 is not comparable to one taken with v2. Pin the embedder version alongside the snapshot id.

8. What re-execution actually needs

"Reproducible" in the lab means the pins exist. Actually re-running needs more, and the gap is worth knowing:

The tool-response row is the one that makes true re-execution mostly impossible: the payment has since been released, the customer record has changed, and re-running the agent against today's bank does not reproduce yesterday's decision.

Which means the honest position, and it should be stated in the validation pack rather than discovered:

We reproduce the decision context, not the decision. Given the pinned configuration and the recorded inputs, we can show what the agent was working from and what it was permitted to do. We cannot re-run the world.

That is what an examiner needs. Claiming more is a claim that gets tested, and the test is a request to re-run something.

The one place full re-execution is achievable and worth building: replaying a recorded run against a new configuration, with the tool responses replayed from the record. That is not reproduction — it is a regression test — and it is the highest-value thing to build with a pinned trace, because it answers "would the new prompt have done the same thing?"

9. The inventory as a gate

An inventory that is not the gate is a spreadsheet, and it is stale within a quarter. Making it the gate means the promotion path goes through it, mechanically:

   deploy pipeline ──► inventory.promote(entry_id, autonomy_band)
                          │
                          ├─ validated? (per tier)
                          ├─ validation expired?
                          ├─ enough eval cases? (per tier)
                          └─ autonomy ≤ tier maximum?
                          │
                   refuse, naming EVERY blocker

Three implementation properties:

Report every blocker. A team that fixes one and discovers the next on the next attempt learns to resent the gate. Collect them all and raise once.

The configuration change hook is what keeps it honest. When the deployed configuration differs from the validated one, validation resets and the model drops out of production. Without that hook, the inventory records what was validated once and diverges from what is running — which is the normal end state of a governance register.

The fingerprint is the comparison. Comparing version strings misses a change somebody made without bumping a version. A fingerprint over the fields that determine behaviour catches it.

And the operational half: reconcile the inventory against what is actually deployed, on a schedule. The gate stops new things; reconciliation finds the things that got in another way, and the first run always finds something.

10. Validation for a non-deterministic system

The methods differ from a scorecard's, and knowing which transfer is the substance of the model-risk conversation.

Two that have no traditional analogue and must be argued for:

Red-teaming as validation evidence. A scorecard cannot be talked into a wrong answer; an agent can. So injection, exfiltration and tool-abuse results belong in the validation pack, scored on containment rather than detection.

Human oversight as a control, evidenced. For a Tier 1 agent the human in the loop is part of the model's control environment, so the validation must cover what the reviewer sees — and a reviewer shown "approve?" with no evidence is a control that does not work (Phase 11).

And the statistical point a validator will raise, correctly: a golden set of 500 cases gives wide confidence intervals on a 94% pass rate. The honest response is not to claim precision but to report the interval, and to note that the eval suite's purpose is regression detection rather than absolute measurement — which changes what it needs to be.

11. Residency evidence, precisely

Three levels of claim, and they are not equivalent:

Present all three. They fail differently, which is the point — a modelling gap in the topology analysis does not affect the records, and a bad record does not affect the topology.

The per-record check needs three fields on every processing artifact, and getting them there is the work:

   region                 where it was processed
   data_classification    what was being processed
   transit_regions        what it traversed  ← the one nobody emits

transit_regions is genuinely hard: a request through a global load balancer may traverse a region you did not choose, and the provider does not always tell you. Where it is unavailable, the honest approach is to record its absence and rely on the network proof for that leg — and to say so rather than leaving a gap.

And the rule that makes the check meaningful: a record that does not state its region is a violation. Not a warning, not a data-quality issue. Unprovable is a violation, because the alternative is that missing data reads as compliance.

12. Detecting a silent provider change

The hardest detection problem in this phase, because the only observable is behaviour.

The layered approach:

The canary is the one that works. A fixed eval set, run continuously against production configuration, with the score tracked as a time series. A step change with no deployment on your side is the signal.

Three design points:

Fixed set, never updated. The moment the canary set changes, the time series is broken. Keep it frozen and maintain a separate growing suite for coverage.

Temperature zero. Sampling noise on a small set swamps a real shift.

Run it against production configuration, not a test one — the point is to detect a change in what production is actually using.

And the statistical honesty: on a 200-case canary, a 2% score change is noise. Size the set from the effect you need to detect, and alert on a sustained shift rather than a single run.

13. Exit testing

A tested exit path drifts. Six months is a reasonable interval, and the test has to be real:

The last is the only one that catches the things that actually break an exit: a rate limit you never hit at 1%, a prompt that behaves differently on the other model, a token-count difference that blows a budget, a latency profile that breaks an SLO.

Which is the argument for the alternative carrying live traffic continuously rather than being tested periodically. It is more expensive — two integrations to maintain, two sets of evals — and it is the difference between an exit plan and an exit capability.

And the thing to measure during a test, which people forget: not whether it worked, but what degraded. An exit that works with 40% worse quality is an exit you can execute in an emergency and not one you can execute for a quarter. Record the delta.

14. Control coverage as a test

The catalogue is a document. Making it a test is what keeps it true:

# daily, over a sample of production traces
for trace in sample:
    missing = verify_control_evidence(graph, trace)
    if missing:
        alert(f"{trace}: {missing}")

Which catches the failure mode a document cannot: a control removed by a refactor. The code path is gone, nothing errors, the catalogue still claims it, and nobody notices until an audit.

Three refinements:

Sample across tenants and action types. A control that only fires for payments will look present if you sample only payments.

Track the rate, not just the presence. A guardrail that emitted a verdict on 100% of traces last month and 60% this month has partially stopped running, which is invisible to a presence check.

Alert on the absence of denials. A policy engine that has denied nothing in a month is either perfectly configured or not running, and those look identical from outside (Phase 14's assertion-counter argument).

15. Performance and volume

Two numbers shape the design. Federated pack generation at seconds is why anything above a threshold should be materialized at action time. And seven-year storage is why the evidence tier is a different store from the debugging tier — at 1 KB per artifact and 10 artifacts per action and 100k actions a day, that is ~2.5 TB over seven years for the artifacts alone, before indexes.

What is not worth optimizing: emission. Ten microseconds against a 2-second agent run is noise, and somebody will propose sampling artifacts to save it.

16. Failure modes