« Phase 15 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: evidence against velocity
• 2. Winning the "what is the model" argument
• 3. Designing the tiering scheme
• 4. Making validation possible to pass
• 5. What to materialize, and what to derive
• 6. Buy or build the governance stack
• 7. Working with the second line
• 8. Designing to the strictest regime
• 9. The concentration-risk conversation
• 10. Setting the numbers
• 11. Migration: retrofitting evidence
• 12. What I would not build

1. The central tension: evidence against velocity

Every artifact, pin and gate slows something down.

   FAST                                                              DEFENSIBLE
     │                                                                     │
   logs only    + join keys    + pins    + inventory   + validation   + full
                                           gate         gate           reproduction
     │              │             │           │             │              │
   nothing       cheap,        cheap       a release     weeks per      mostly
   provable      day one       forever     step          change         impossible

The failure at the fast end is obvious and slow to arrive: nothing is provable, and you find out during an examination.

The failure at the defensible end is faster and more common: teams route around the governance. An agent that takes six weeks to change gets replaced by a script somebody runs manually, which is outside the inventory, outside the evidence chain, and outside every control in this track. A bypassed control is worse than a loose one because it is also invisible.

So the position I would defend:

Make the evidence free and the gates proportionate. Join keys, artifacts and pins cost nothing at runtime and should be universal. Validation depth scales with tier.

Per class:

The last row is worth arguing for: a tool that can move money changes what the agent is, and tiering by the agent's existing tier understates it.

2. Winning the "what is the model" argument

You will have this conversation with model risk, and how you open it decides how it goes.

The wrong opening: "our system is different, the framework doesn't apply." True in parts, and it sounds like a request for an exemption. You will get more process, not less.

The right opening: "the framework applies, and here is how each requirement maps — including the one place where the mapping is genuinely ambiguous, which is what counts as the model."

Then the argument itself, in the order that lands:

One — start from their definition. SR 11-7 defines a model by what it does: a quantitative method producing output used in a decision. By that definition the agent is the model, and the agent includes its prompt.

Two — make it empirical, not theoretical. Show two eval runs: the same weights with two prompts, scoring 0.94 and 0.71. That is a stronger argument than any amount of reasoning, and it takes an afternoon to produce.

Three — offer the tiering as the resolution. The objection is cost, and the answer is that cost scales with impact. Bring the tiering table.

Four — concede what is genuinely different. Non-determinism, no closed-form performance metric, a provider who can change behaviour under you. Naming the hard parts before they do is what makes the rest credible.

And the concession worth making early: the eval suite is the model-performance metric, and it is weaker than a scorecard's backtest. Saying so buys you the room to explain what it does give — regression detection, safety coverage, and a versioned artifact — rather than defending a claim of equivalence you will lose.

3. Designing the tiering scheme

The scheme is where governance either becomes proportionate or becomes uniform, and uniform is what kills adoption.

Three design decisions:

How many tiers? Three. Two is too coarse — everything lands in "high" and the process is uniform. Four or more and nobody can remember what tier 3 means, so everything defaults to the middle.

What drives the tier? Impact only. Not technique, not model size, not "is it generative". The inputs I would use: irreversible actions, financial magnitude, customer impact, regulatory reporting, data classification.

What does the tier change? This is the one that matters, and the list must be non-trivial:

The autonomy ceiling is the row that makes tiering an engineering control rather than a documentation exercise. Enforced by the inventory refusing to promote, mechanically.

And the failure mode to watch: everything ends up Tier 1. It happens when the criteria are vague and nobody wants to defend a lower tier. Two defences — make the criteria concrete and mechanical, so the tier is derived rather than argued, and make Tier 3 genuinely cheap so there is an incentive to scope a use case down into it.

4. Making validation possible to pass

The most common failure is not a rejected validation. It is a validation that never completes, because the pack is never quite finished and the reviewer keeps finding new questions.

Four things that fix it, in order of effect:

Agree the pack template first. Before anything is built, agree with the validator what a complete pack contains. It turns validation from an open-ended review into a checklist, which is better for both sides.

Give the validator the tools. They should be able to run the eval suite themselves, query the inventory, and pull an evidence pack. A validator who can only read documents will ask for more documents.

Write the limitations section yourself, honestly. A pack claiming no limitations gets probed until one is found, and now the whole pack is suspect. A pack that names six limitations and their mitigations is a team that understands its system.

Use "approved with conditions". It is the most common right answer and it unblocks. "Assisted autonomy only, monthly monitoring, revalidation on any prompt change" is a real outcome, and it beats a binary that forces a rejection of something usable.

And the thing to build that pays for itself immediately: a validation pack generator. Most of the pack — configuration pins, eval results, red-team results, monitoring plan, control coverage — exists in systems already. Generating it means the pack is never stale, and it turns revalidation from a writing exercise into a re-run.

5. What to materialize, and what to derive

The federation problem from §1 of the deep dive forces a choice, and it is a storage-versus-availability trade:

The third, and the threshold is the decision. What I would materialize at action time:

• every irreversible action, regardless of value;
• every action above the dual-control threshold;
• every denial — an examiner asking "has this ever been attempted?" needs the refusals;
• every action by a Tier 1 agent;
• a random sample of everything else, so the derivation path is exercised.

That last one is the operational insight: a code path only used during an examination is a code path that does not work. Materializing a random sample continuously means the generator is tested every day.

And the second-order benefit of materializing: it decouples the pack from schema drift. A pack generated in 2026 and stored is readable in 2033 without needing 2026's schemas. Deriving it later means maintaining six version-tolerant readers for seven years, which nobody does well.

6. Buy or build the governance stack

The line: integrate with the bank's governance systems; build the evidence generation.

The specific trap: a GRC platform will offer to be the model inventory. Accept it as the register — the place risk and audit look — and keep the gate in your deployment pipeline. A register updated by humans diverges from reality within a quarter; a gate cannot, because nothing reaches production without passing it. The two should be synchronized, with the pipeline as the source of truth.

7. Working with the second line

Model risk, compliance and audit are the second and third lines. The relationship determines whether this phase takes a quarter or two years, and most engineering teams get it wrong in the same way: treating them as an obstacle to be satisfied at the end.

Four things that work:

Involve them at design time. A control designed with the validator is a control that passes validation. A control designed alone is a control that gets three rounds of questions.

Speak their vocabulary. "Three lines of defence", "inherent versus residual risk", "control effectiveness", "compensating control". Using their words is not politics — it is what lets them place your work in their framework without translating it.

Give them self-service. A validator who can pull an evidence pack, run the eval suite and query the inventory without asking you is a validator who moves fast. Every access request you require is a delay you own.

Bring problems early. "We cannot pin the retrieval snapshot until Q3, here is the compensating control in the meantime" is a conversation. Discovering it during validation is a finding.

And the framing that changes the dynamic: their job is to be able to defend the platform to a regulator. They are not gatekeepers by preference; they are people who will be asked questions they cannot answer unless you give them the answers. Making them effective is in your interest, and saying so out loud usually changes the relationship.

8. Designing to the strictest regime

Multiple regimes, overlapping and diverging. Two strategies:

Comply with each separately. N control sets, N mappings, N audits. It does not scale past two.

Design to the strictest, map to each. One control set, N mappings.

The second, and today the strictest is the EU AI Act for high-risk systems: risk management, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy, robustness, cybersecurity. Even outside the EU it is the right design target, because it is where everyone else is converging and because retro-fitting to it later is more expensive than building to it now.

Which produces a specific artifact worth maintaining: the mapping table, control → regime → requirement, generated from the control catalogue. Three properties:

• Generated, so it cannot claim a control that does not exist (Phase 11);
• Versioned, because regimes change and a mapping nobody updates is worse than none — it is trusted;
• Gap-explicit. A requirement with no control is listed as a gap with an owner and a date, not omitted. An examiner trusts a document that names its gaps far more than one that does not.

And the honest constraint to raise early: residency can conflict with capability. The best model may not be available in your region, and the answer is a decision — accept a weaker model, self-host, or seek an exemption — made by the business rather than defaulted into by engineering.

9. The concentration-risk conversation

You will be asked, and the answer has three parts.

One — the architecture. A model abstraction layer (Phase 04), policy-driven routing, and a second provider carrying live traffic. The last clause is the whole answer; the rest is preamble.

Two — the measured degradation. "On the alternative, quality drops 8% on our eval suite and latency rises 40%." That is a much stronger answer than "we have an alternative", because it shows the alternative has been used rather than configured.

Three — the honest limits. Where you are genuinely concentrated and cannot easily move: a specific capability only one provider has, a fine-tune you cannot port, an embedding model whose replacement means re-indexing the corpus. Naming those is what makes the first two credible.

The embedding one is worth calling out because it is the concentration people miss: changing the embedding model means re-embedding everything, which is a re-index of the whole corpus and a re-validation of retrieval quality. It is a much bigger exit than changing the generation model, and it usually is not in the plan.

And the question to expect as a follow-up: "how long would a migration take?" Have a number, derived from a test, with the degradation attached.

10. Setting the numbers

Tier thresholds. From the bank's existing risk appetite, not invented. There is already a materiality threshold for operational risk; use it.

Revalidation intervals. From the bank's model-risk standard. Annual for Tier 1 is typical, and arguing for something different needs a reason.

Evidence retention. 7 years for anything regulatory. Longer for anything under litigation hold, which is a different mechanism with its own process.

Eval suite size. From the effect you need to detect, not from a round number. On 200 cases a 2% change is noise; if you need to detect 2%, you need a larger set. This is a statistics conversation and it is worth having properly once.

Exit test interval. 6 months, and the test is a real traffic shift with the degradation measured.

Deprecation notice to accept. 90 days minimum — one validation cycle. Below that you cannot validate a replacement before the original is gone, and that is a procurement term rather than an engineering problem.

Concentration threshold. 80% of traffic on one provider as the point where it becomes a finding. And measure it per provider, not per model.

Materialization threshold. §5 — irreversible actions, above dual control, all denials, all Tier 1, plus a continuous random sample.

11. Migration: retrofitting evidence

Starting state: a platform in production, logs in six systems, no join key, no inventory.

Phase 1 — the join key, everywhere. Nothing else matters until this is done, because everything after it depends on records having it. And the honest framing: the past is unrecoverable. Say so early, in writing, with the date the chain begins. An examiner accepts a stated boundary; they do not accept discovering one.

Phase 2 — region and classification on every processing record. The same argument, and it is the other field that cannot be added retroactively.

Phase 3 — the artifact schema and derived_from. Now the graph exists.

Phase 4 — the inventory, populated from what is actually running. Not from the spreadsheet. You will find agents nobody owns, and that discovery is the business case for the rest.

Phase 5 — the inventory as a gate, in the deployment pipeline. Stops the problem growing.

Phase 6 — pins. The retrieval snapshot has the longest lead time because the search layer must support snapshots; start that conversation in Phase 1.

Phase 7 — the pack generator, in non-strict mode, so gaps surface as tickets.

Phase 8 — strict mode, and materialization.

Phase 9 — validation of the existing estate, worst tier first.

The mistake is starting at Phase 9 because that is what the regulator asked for. Validating agents whose evidence chain does not work produces validation packs that cannot be supported by evidence, which is a worse position than not having validated — you have now attested to something you cannot demonstrate.

12. What I would not build

A GRC platform. The bank has one. Integrate.

A lineage format. OpenLineage exists. A proprietary format is a migration you will do later.

An immutable store. WORM blob storage and Object Lock exist with compliance attestations you cannot self-certify.

A model-risk framework. The bank has one, descended from SR 11-7. Map to it; do not write a parallel one, because a parallel framework is one the second line did not agree to.

Automated validation. Validation is a judgment by an independent human. Automate the evidence they need — the pack, the evals, the coverage — and leave the judgment alone. A system claiming to automate validation is one that will be rejected, correctly.

A universal explainability layer. "Why did the model say that?" is not answerable for a large language model in the way a regulator's phrasing implies, and building something that gestures at it creates a false expectation that is much harder to retract than a clear "we can show what it used and what it was permitted to do, not why the weights produced that token."

A separate audit trail for AI. The bank has an audit trail. Emit into it, with your artifact kinds. A parallel AI audit trail is a second thing to reconcile and a second thing to explain.

A "compliance dashboard" nobody asked for. The artifacts an examiner wants are packs, not dashboards. Build the pack generator; the dashboard is a by-product if anyone wants one.