« Phase 15 · Lab 01 · Track Overview

Warmup — Governance, Model Risk & Evidence, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: a log is not evidence
• 2. The evidence chain
• 3. Join keys are the design
• 4. Reproducibility, and the pin everyone forgets
• 5. Model risk, and what SR 11-7 actually says
• 6. What is "the model"?
• 7. Risk tiering
• 8. Independent validation
• 9. Lineage, in both directions
• 10. Residency as a provable property
• 11. Third-party model governance
• 12. Concentration risk
• 13. Drift and silent provider changes
• 14. The regulatory frames
• 15. A control that emits no artifact does not exist
• 16. Numbers worth carrying
• 17. Interview questions, answered
• 18. References

0. Where this sits

This phase consumes every other phase, and it is the one that makes them defensible.

The phase's own contribution is small and load-bearing: the join keys, the pack generator, and the argument about what the model is.

1. From first principles: a log is not evidence

Start with what an examiner is actually doing. They are not auditing your code. They are testing a claim — "this platform is under control" — and the way to test a claim is to pick one specific past event and ask for its complete story.

So the question is never "do you log?" It is:

"On 12 March, an agent initiated a payment of AED 250,000 for customer X. Show me who authorized it, what the agent was permitted to do at that moment, what data it used to decide, which model version produced the decision, which policy version allowed it, and who reviewed it."

Six sub-questions, answered by six different systems. A log line answers one of them, badly.

The three properties that separate evidence from logging:

Linked. The six records must be joinable. If the policy decision, the inference and the action each have their own id and no shared key, they are three logs about the same event that nobody can prove are about the same event.

Complete. Every required record present. A pack missing the approval is not "mostly good" — it is a pack that cannot answer the question it was made for.

Tamper-evident. The chain must show that nothing was edited after the fact (Phase 10). Otherwise the examiner is trusting your database permissions, which is not a control they can verify.

And the property that makes all three possible:

Evidence is generated as a by-product of serving. Anything assembled later is a reconstruction — and a reconstruction has gaps it cannot explain.

That is not a stylistic preference. A reconstruction is assembled by somebody searching for what they can find, so its shape is determined by what survived, and an examiner probes exactly the parts that did not.

2. The evidence chain

Six artifacts, six emitters, six questions:

Two design rules that follow.

Every artifact names what it derived from. Not just a timestamp — an explicit parent list. That is what makes "what did the agent use to decide?" a graph walk rather than a correlation exercise over timestamps, which is guesswork with extra steps.

If you cannot name whose question an artifact answers, it does not belong in the pack. Evidence packs bloat, and a bloated pack is harder to defend than a small one because every field invites a question.

3. Join keys are the design

The single most important decision in this phase, and it costs nothing on day one.

    trace_id  ──►  session record        (channel)
              ──►  policy decision       (control plane)
              ──►  retrieval record      (search service)
              ──►  inference record      (gateway ledger)
              ──►  execution steps       (kernel)
              ──►  audit record          (action gateway)

Six stores. Six retention policies. Six teams. One id, present on every record from the first line of code.

Why it must be day one: retrofitting a join key does not fix the past. The records already written do not have it, so the first N months of operation are permanently un-assemblable. And N months is exactly the period an examiner will ask about, because that is the period that is finished.

The keys worth carrying, and what each is for:

And the operational consequence people underestimate: the six stores have different retention. The debugging trace is kept 30 days; the audit record is kept 7 years. So a pack for a 3-year-old action can contain the action and not the trace — which is fine and must be stated, because a gap you predicted is a design decision and a gap you did not is a finding.

4. Reproducibility, and the pin everyone forgets

"Explain this decision" six months later requires re-deriving the context it was made in. Which needs pins:

The retrieval snapshot is the one people forget, and it is the one that most often makes a decision irreproducible. Every other pin is a version string somebody bumps deliberately; the corpus changes continuously, by design, because documents are added and updated all day. Without a snapshot id you can pin everything else perfectly and still not know which documents the agent saw.

And the honest caveat that must be stated rather than hidden:

Even with every pin, a temperature above zero means the output is not bit-reproducible.

What is reproducible is the decision context — the same inputs, the same permitted behaviour — which is what an examiner actually needs. Claiming bit-reproducibility for a sampled model is a claim that will be tested, and failing that test costs more than the caveat would have.

5. Model risk, and what SR 11-7 actually says

Most banks' internal model-risk standards descend from the US Federal Reserve's SR 11-7 (2011). It predates all of this and its four ideas transfer well:

One — a model is a quantitative method that produces an output used in a decision. Note what this covers: it is not about the technique. A spreadsheet is a model; so is an LLM.

Two — model risk is the risk of adverse consequences from decisions based on incorrect or misused model output. Two sources: the model may be wrong, or it may be used wrongly. The second is half the framework and the half people forget.

Three — three lines of defence. Development (build it, test it, document it), independent validation (challenge it, with authority to block), and audit (check the process is followed).

Four — the lifecycle. Inventory, development, validation, approval, monitoring, revalidation, retirement.

What SR 11-7 did not anticipate is generative, non-deterministic, tool-using models. Which produces four genuine tensions worth naming before somebody else does:

None of those breaks the framework. They change what the artifacts look like, and being able to say that clearly is what makes the conversation with model risk constructive rather than adversarial.

6. What is "the model"?

The question this phase turns on, and it gets contested every time.

For a credit scorecard, "the model" is unambiguous: coefficients in, score out. Validation tests the coefficients.

For an agent, the output distribution is determined by:

    weights  ×  system prompt  ×  retrieval config  ×  tool set  ×  guardrails  ×  temperature

Change any one and the behaviour changes — and a prompt change often changes behaviour more than a weights change would. So the defensible position:

The agent configuration as a whole is the model.

Which has consequences people resist, correctly identifying them as expensive:

The objection is always the same: "we cannot revalidate for every prompt tweak." Three answers, in the order that works:

One — you are not revalidating fully. Tiering (§7) means a Tier 3 agent's prompt change needs an eval run, not a validation committee. The depth scales with impact.

Two — the eval suite is the mechanism. A prompt change triggers the eval gate (Phase 09), which is automated. Validation reviews the gate, not every run through it.

Three — and this is the one to say plainly: a prompt change is expensive, because it changes what the system does. The only question is whether the cost is paid before deployment or after an incident.

7. Risk tiering

Tier by impact, never by technique. "Is it an LLM?" is not a risk question; "can it move money?" is.

Two properties make a tiering scheme real rather than decorative:

The tier must change something. If Tier 1 and Tier 3 differ only in a field in a register, the tiering is a labelling exercise. The tier must drive validation depth, approval level, monitoring frequency, eval-suite size, and the autonomy band.

That last one is the link that matters. Risk tier → autonomy band → Phase 10's side-effect policy. A Tier 1 agent cannot be promoted to autonomous, mechanically, by the inventory refusing. That is the difference between governance that is enforced and governance that is documented.

And record the reason for the tier. A bare tier is one nobody can challenge, and being challengeable is the point of writing it down — a validator's first question is "why this tier?"

8. Independent validation

The second line of defence, and the whole of "independent" is one property:

The validator did not build it and does not report to whoever did.

A validation function inside the building team is a review. The distinction is not bureaucratic — the value of validation is adversarial, and you cannot be adversarial about your own design.

What a validation pack contains for an agentic system, which is different from a scorecard's:

Two sections carry disproportionate weight.

Limitations, written by the builders. A validation pack claiming no limitations is a pack that has not been thought about, and it is the first thing a good validator probes.

The cases it fails. An eval report showing 94% is much weaker than one showing 94% and the six failures, categorized. The second is a team that understands its system.

And the outcome that is often right: approved with conditions. "Approved for assisted autonomy only, with monthly monitoring, revalidation on any prompt change" is a real answer, and it is better than a binary that forces a validator to reject something usable.

9. Lineage, in both directions

Lineage is the graph from an output back to every input. Everyone builds the backward direction and forgets the forward one.

Backward — the examiner's question. "What did this decision use?" Walk from the action to the session, through the inference, the retrieval, the documents, the policy decision.

Forward — the impact query. "This document was wrong. What did it affect?"

The forward direction is the one asked during a remediation, and it is the one that determines how expensive a data-quality incident is. A corpus document that was wrong for three weeks: which decisions used it? Without forward lineage the answer is "we don't know", which means the remediation is "review everything", which is a quarter of work.

Two properties worth enforcing at write time:

Causal order. An artifact may only derive from artifacts that already exist. Enforcing that on write is what stops a graph you cannot walk, and it catches emitters that record their inputs incorrectly.

Acyclicity. A cycle means an output that is its own ancestor — either an emitter bug or an unintended feedback loop. Both are findings.

And orphans: an artifact that derives from nothing and that nothing derives from. Usually an emitter that forgot to record its inputs, and it is worse than a missing artifact — because it looks like evidence while being disconnected from the story.

10. Residency as a provable property

Phase 13 proved no path can leave the region. This phase proves none did.

Both are needed, and the difference is what an examiner asks for:

Together they are hard to defeat: the network says it is impossible and the records say it did not happen, and they are produced by different systems.

The per-record check reads the actual inference records — region, data classification, and any transit regions — against a rule per classification. Three verdicts:

• region outside the permitted set → violation;
• transit through a non-permitted region → violation, unless the rule permits transit;
• the record does not state its region or classification → violation.

The third is the one to insist on. Unprovable is a violation. "We think it stayed in region" is not evidence, and a record that cannot demonstrate its own compliance is indistinguishable from one that was non-compliant.

And the deployment implication: the platform must emit region and classification on every processing record, which is a schema decision made at the start or a migration made later.

11. Third-party model governance

A third-party model is an outsourcing arrangement, and banking regulators have decades of practice about those. The register:

Two rows produce more work than the others.

Deprecation notice. A provider retiring a model version with 30 days' notice, when your revalidation cycle is 90 days, means you cannot validate the replacement before the old one is gone. That is a contractual problem you solve at procurement, not an engineering problem you solve later.

Retention and abuse monitoring. Most providers retain prompts for abuse monitoring for some period. For a bank that is customer data at a third party, and whether there is an opt-out is a question with a specific answer that belongs in the register.

12. Concentration risk

"What happens if your model provider has an outage, changes their terms, or exits the region?"

The answer is an architecture, not a paragraph — and it has four rungs:

An exit plan that has never been executed is a document.

The only convincing answer is that some traffic already runs on the alternative — which is exactly what Phase 04's model abstraction was for. The gateway's routing rules are the exit plan, and a small percentage of live traffic on the second provider is the proof that they work.

Two refinements:

Aggregate per provider, not per model. Two models from one vendor is one concentration.

Test on a schedule. A "tested" exit path tested eighteen months ago has drifted. Six months is a reasonable interval, and the test is a real traffic shift rather than a tabletop.

13. Drift and silent provider changes

Two distinct problems that get conflated:

Model drift — the world changed; the model is the same. New products, new fraud patterns, new regulations. Detected by monitoring quality against a fixed eval set over time.

Silent provider change — the model changed; you were not told. A provider updates weights behind a stable version string, or routes you to a different deployment.

The second is specific to third-party models and it is genuinely difficult, because the only thing you can observe is behaviour. The defences:

The canary is the one that does the work: a fixed eval set, run continuously against production configuration, with the score tracked. A step change in the score with no deployment on your side is the signal, and without it a silent change is discovered by a user weeks later.

And the reason pinning matters more than it looks: without a pinned version, "the provider changed the model" is a hypothesis you can never confirm or exclude — so every quality incident ends in a shrug (Phase 14).

14. The regulatory frames

Five, and knowing which is which prevents a lot of confused conversation.

CBUAE — the UAE central bank. What matters here: outsourcing and cloud requirements, data residency, operational resilience, and the expectation that you can demonstrate control. The residency requirement is the one that shapes architecture.

SR 11-7 — US Federal Reserve model-risk guidance, 2011. Not binding in the UAE, and most banks' internal standards descend from it, so it is the vocabulary in the room. §5.

NIST AI RMF — voluntary, US, and useful as a structuring vocabulary: Govern, Map, Measure, Manage. Good for organizing a programme; not a compliance obligation.

EU AI Act — the strictest reference regime. Risk-based, with obligations for high-risk systems: risk management, data governance, technical documentation, logging, human oversight, accuracy, robustness, cybersecurity. Even outside the EU it is worth designing against, because it is the ceiling everyone else will converge toward.

OWASP LLM Top 10 — a technical control checklist rather than a regulation (Phase 11). Useful because it is concrete and because auditors have started citing it.

The practical approach: design to the strictest, map to each. One control set, several mappings. And the mapping is a maintained artifact, not a one-off — regimes change, and a mapping nobody updates is worse than none because it is trusted.

15. A control that emits no artifact does not exist

The phase's operating principle, and the one that changes engineering behaviour.

A control that runs and leaves no record cannot be shown to have run. For an examiner that is indistinguishable from not having run — not because they are unreasonable, but because they have no way to tell the difference.

So every control gets a third column:

The blank rows are silent controls, and they are not necessarily wrong — egress allow-listing is enforced by the network and genuinely hard to attach to a trace. But they must be named, because at audit time they need some other form of evidence: a configuration attestation, a policy compliance report, a signed validation document.

Knowing which controls are silent is the difference between a prepared answer and a scramble. And the way to know is to generate the mapping from the control catalogue rather than write it — a hand-written matrix documents intentions (Phase 11).

16. Numbers worth carrying

17. Interview questions, answered

Q1. "An examiner asks you to explain a specific agent decision from six months ago. What do you show them?"

A generated evidence pack — one signed, hash-chained bundle with six artifacts joined on a trace id: the session record for who asked and how they authenticated; the policy decision for what the agent was permitted to do and under which policy version; the retrieval record for which documents at which versions; the inference record for the model version, tokens and region; the execution steps; and the action record with its idempotency key and approvals.

The important word is generated. Every layer emits its artifact as a by-product of serving, joined by one id present from the first line of code. If I had to assemble it afterwards it would be a reconstruction, and a reconstruction has gaps it cannot explain — which is exactly what an examiner probes.

And the generator refuses to produce an incomplete pack: it fails, naming the missing artifact. That is deliberate, because it turns an evidence gap into an engineering ticket during development rather than an audit finding two years later.

Q2. "What is 'the model' for an agentic system?"

The agent configuration as a whole: the weights, the system prompt, the retrieval configuration, the tool set, the guardrails and the temperature. Change any one and the output distribution changes — and a prompt change often changes behaviour more than a weights change would.

Which means a prompt edit is a model change, with revalidation and a version bump. That is contested every time, and the objection is always "we cannot revalidate for every prompt tweak".

Three answers. The depth scales with tier — a Tier 3 agent's prompt change needs an eval run, not a committee. The eval gate is the mechanism, and validation reviews the gate rather than every run through it. And the honest one: a prompt change is expensive because it changes what the system does; the only question is whether the cost is paid before deployment or after an incident.

Q3. "How do you tier models for risk?"

By impact, never by technique. "Is it an LLM?" is not a risk question; "can it move money?" is.

Tier 1 for irreversible actions, material financial impact, or anything feeding regulatory reporting. Tier 2 for customer impact or restricted data. Tier 3 for internal productivity with no material impact.

The tier has to change something or it is a label. It drives validation depth, approval level, monitoring frequency, eval-suite size — and the autonomy band, which is the link that matters. A Tier 1 agent cannot be promoted to autonomous, mechanically, because the inventory refuses. That is the difference between governance that is enforced and governance that is documented.

And I record the reason for the tier, because a validator's first question is "why this tier?" and a bare tier is one nobody can challenge.

Q4. "What does reproducibility require?"

Six pins: base model version, prompt version, retrieval snapshot, policy version, tool set version, guardrail version.

The one people forget is the retrieval snapshot, and it is the one that most often breaks reproducibility. Every other pin is a version somebody bumps deliberately; the corpus changes continuously by design. So you can pin everything else perfectly and still not know which documents the agent saw.

And I would state a caveat rather than hide it: even with every pin, a temperature above zero means the output is not bit-reproducible. What is reproducible is the decision context — same inputs, same permitted behaviour — which is what an examiner actually needs. Claiming bit-reproducibility for a sampled model is a claim that gets tested.

Q5. "How do you prove data residency?"

From two independent directions, because either alone is weak.

The network side proves no path can leave the region — a reachability analysis over the topology, run on a schedule, with retained output. The evidence side proves none did — a per-record check over inference, retrieval and tool-call records, verifying that region and data classification match a rule per classification.

They are produced by different systems, so together they are hard to defeat: the network says impossible and the records say it did not happen.

The rule I would insist on is that a record which does not state its region is a violation. Unprovable is a violation — "we think it stayed in region" is not evidence, and a record that cannot demonstrate its own compliance is indistinguishable from a non-compliant one.

Q6. "What is your answer on concentration risk?"

An architecture, not a paragraph. There is a model abstraction layer, routing is policy-driven, and there is a second provider carrying live production traffic — a small percentage, deliberately.

Because an exit plan that has never been executed is a document. The four rungs are none, identified, tested and live, and only the last is convincing: it proves the path works today, not that it worked in a tabletop eighteen months ago.

Two refinements I would mention. Concentration aggregates per provider, not per model — two models from one vendor is one concentration. And a "tested" path needs re-testing on a schedule, six months being reasonable, with a real traffic shift.

Q7. "How do you detect that a provider silently changed the model?"

Pin the version, and run a continuous canary eval — a fixed evaluation set against production configuration, with the score tracked over time. A step change in the score with no deployment on our side is the signal, and it gives detection within a day rather than a quarter.

Supporting evidence: record gen_ai.response.model, which sometimes differs from what you requested; shadow the previous version after a switch so comparison beats memory; and a contractual change notification, which works when it works.

The reason pinning matters more than it looks is diagnostic: without a pinned version, "the provider changed the model" is a hypothesis you can never confirm or exclude, so every quality incident ends in a shrug.

Q8. "How do you know your controls are working?"

Every control has three columns: what it is, which component implements it, and what artifact it emits. A control that emits no artifact cannot be shown to have run, which for an examiner is indistinguishable from not having run.

Then two mechanisms. A generated coverage matrix that maps controls to frameworks and surfaces the silent ones — egress allow-listing, for example, which is enforced by the network and hard to attach to a trace. Those are not wrong, but they need some other form of evidence and I need to know which ones they are; that is the difference between a prepared answer and a scramble.

And continuous verification: run a check over a sample of production traces daily that asks "did every evidence-emitting control leave an artifact?" It is how you find out a control was removed by a refactor — which otherwise you discover during an audit.

18. References

Regulation and supervisory guidance

• CBUAE Rulebook · outsourcing regulation
• SR 11-7 — Guidance on Model Risk Management
• EU AI Act · high-risk obligations
• Basel — Principles for effective risk data aggregation (BCBS 239)
• EBA guidelines on outsourcing

Frameworks

• NIST AI Risk Management Framework · Generative AI Profile
• ISO/IEC 42001 — AI management systems
• OWASP Top 10 for LLM Applications

Lineage and provenance

• OpenLineage · Marquez
• Microsoft Purview · Azure AI Foundry governance
• in-toto · SLSA — provenance for artifacts
• RFC 6962 — Certificate Transparency — tamper-evident logs

Model documentation

• Model Cards (Mitchell et al.) · Datasheets for Datasets (Gebru et al.)
• Azure OpenAI — data, privacy and security