« Phase 15 · Warmup · Track Overview

Hitchhiker's Guide — Governance, Model Risk & Evidence

The fast orientation. What the pieces are, what they are called, and how they fit — before the deep dive takes them apart.

Table of Contents

• 1. Don't panic: the one-paragraph version
• 2. The map
• 3. The vocabulary
• 4. The evidence pack, memorized
• 5. The six pins
• 6. Who asks what
• 7. The five things that will surprise you
• 8. Reading a validation pack
• 9. Where the neighbouring phases connect
• 10. What to build first

1. Don't panic: the one-paragraph version

An examiner picks one past action and asks for its complete story: who authorized it, what the agent was permitted to do, what data it used, which model version decided, which policy allowed it, and who reviewed it. Six answers from six systems, which are only joinable if one id was on all of them from day one. So evidence is generated as a by-product of serving, linked by that id, hash-chained, and produced as a pack that refuses to be incomplete. Alongside it sits model risk: an inventory where the agent configuration as a whole is the model, tiered by business impact, where the tier drives the autonomy band, and where independent validation means the validator did not build it. Plus three provable properties — residency per record, reproducibility from pins, and exit readiness that means live traffic rather than a document.

2. The map

   channel ──► session record ─────┐
   control plane ──► decision ─────┤
   knowledge ──► retrieval ────────┤   all carrying ONE trace_id
   gateway ──► inference ──────────┼──────────────────────────────┐
   kernel ──► execution steps ─────┤                              │
   guardrails ──► verdicts ────────┤                              ▼
   HITL ──► approval ──────────────┤                    ┌──────────────────┐
   action gateway ──► audit ───────┘                    │  LINEAGE GRAPH   │
                                                        │  ancestors ◄──►  │
   ┌──────────────────┐   ┌───────────────────┐         │  descendants     │
   │ MODEL INVENTORY  │   │ THIRD-PARTY       │         └────────┬─────────┘
   │ tier ─► autonomy │   │ REGISTER          │                  │
   │ validation gate  │   │ exit readiness    │                  ▼
   └──────────────────┘   └───────────────────┘         ┌──────────────────┐
                                                        │  EVIDENCE PACK   │
   residency check ────────────────────────────────────►│  signed, chained │
   reproducibility check ──────────────────────────────►│  FAILS if a      │
   control coverage ───────────────────────────────────►│  required        │
                                                        │  artifact is     │
                                                        │  missing         │
                                                        └──────────────────┘

3. The vocabulary

4. The evidence pack, memorized

Six required, one conditional. And the rule: if you cannot name whose question an artifact answers, it does not belong in the pack.

5. The six pins

   base_model_version    prompt_version      retrieval_snapshot   ← the forgotten one
   policy_version        tool_set_version    guardrail_version

Every one except retrieval_snapshot is a version somebody bumps deliberately. The corpus changes continuously, by design — so it is the pin that is missing when everything else is present, and it is the one that breaks reproducibility.

And the caveat to state rather than hide: temperature > 0 means the output is not bit-reproducible. The decision context is, which is what an examiner needs.

6. Who asks what

Different audiences, different questions, same records:

Note the last row. Everyone builds backward lineage; the forward direction is the one asked during a remediation, and it determines whether a bad document costs a query or a quarter.

7. The five things that will surprise you

1. Retrofitting the join key does not fix the past. The records already written do not have it, so the first N months are permanently un-assemblable — and that is exactly the period an examiner asks about, because it is finished.

2. An orphaned artifact is worse than a missing one. It looks like evidence while being disconnected from the story, so it survives review and fails at the moment it is needed.

3. A prompt edit is a model change. People resist this correctly — it is expensive. The answer is that a prompt change is expensive, and the cost is paid before deployment or after an incident.

4. Unprovable is a violation. A record that does not state its region is not neutral; it is indistinguishable from a non-compliant one.

5. Retention tiers differ by three orders of magnitude. The audit record lives 7 years, the debugging trace 30 days. So a pack for a 3-year-old action legitimately lacks the trace — which is fine if you predicted it and a finding if you did not.

8. Reading a validation pack

What a good one contains, and what each section is really for:

   1. SCOPE            what it is for — and explicitly what it is NOT for
   2. DATA             corpus provenance, freshness, coverage, KNOWN GAPS
   3. CONFIGURATION    every pin
   4. EVALUATION       golden set, safety suite, results — AND THE FAILURES
   5. RED TEAM         injection, exfiltration, tool abuse (Phase 11)
   6. LIMITATIONS      where it is unreliable, WRITTEN BY THE BUILDERS
   7. MONITORING       what is watched, how often, at what threshold
   8. HUMAN OVERSIGHT  what the reviewer sees and can veto
   9. FALLBACK         what happens when it is unavailable or wrong

Two sections carry disproportionate weight, and they are the ones to read first as a reviewer:

Section 6 — limitations. A pack claiming none has not been thought about. It is the first thing a good validator probes.

Section 4's failures. "94%" is much weaker than "94%, and here are the six failures, categorized". The second is a team that understands its own system.

And the outcome that is usually right: approved with conditions. "Assisted autonomy only, monthly monitoring, revalidation on any prompt change" is a real answer, and better than a binary that forces a validator to reject something usable.

9. Where the neighbouring phases connect

10. What to build first

1. The join key. trace_id on every record, everywhere, before anything else. It costs nothing now and cannot be added retroactively.
2. The artifact schema, including derived_from. A timestamp correlation is guesswork; an explicit parent list is a graph.
3. Region and data classification on every processing record. Same argument — a schema decision now or a migration later.
4. The model inventory as a gate, before the first agent reaches production. An inventory that is not the gate is a spreadsheet.
5. The pins, emitted by whoever owns each one. The retrieval snapshot needs the search layer to support snapshots, which is the longest lead time.
6. The pack generator, early and in non-strict mode, so gaps show up as tickets while there is still time.
7. The control catalogue with its emits column. Generated coverage, not a written matrix.
8. Third-party register and exit testing, once there is a second provider — and get a small share of live traffic onto it.