Lab 01 — The Evidence Engine

The problem

An examiner sits down with one question:

"On 12 March, an agent initiated a payment of AED 250,000 for customer X. Show me: who authorized it, what the agent was permitted to do at that moment, what data it used to decide, which model version produced the decision, which policy version allowed it, and who reviewed it."

You have logs. You have traces. You have an audit table. And you cannot answer, because:

the policy decision is in the control plane's store, the inference record is in the gateway's ledger, the retrieval is in the search service's log, and nothing joins them;
the model version was recorded but the prompt version was not, and somebody edited the prompt in February;
the retrieval snapshot was never pinned, so nobody can say which documents the agent saw;
and the approval is a Teams message.

Six months of work exists and none of it is evidence. Evidence is a linked set of records with shared join keys, generated as a by-product of serving — and anything assembled afterwards is a reconstruction, which an examiner can tell.

You build the engine that generates it, and that refuses to produce a pack with a hole in it.

What you build

#	Component	What it does
1	`assign_tier`, `TIER_POLICIES`	impact-based tiering that drives autonomy, validation and evals
2	`ModelConfiguration`	the agent configuration is the model — fingerprinted
3	`ModelInventory`	the register, and the promotion gate
4	`LineageGraph`	ancestors (the examiner's question) and descendants (the impact query)
5	`ResidencyChecker`	proved per record, not asserted from config
6	`check_reproducibility`	the six pins, and the caveats you must state
7	`ThirdPartyRegister`	concentration risk, and exit readiness that means something
8	`EvidenceGenerator`	a signed, hash-chained pack that fails loudly
9	`control_coverage`	a control that emits no artifact does not exist

Key concepts

Concept	Where	Why it matters
Tier by impact, not technique	`assign_tier`	"is it an LLM?" is not a risk question
A tier must change something	`TierPolicy`	one that changes nothing is a label
Tier drives autonomy	`max_autonomy`	wired to Phase 10, not filed
The reason is recorded	`tier_reasons`	a bare tier is one nobody can challenge
The configuration is the model	`ModelConfiguration`	so a prompt edit is a model change
A config change invalidates validation	`record_change`	and drops it out of production
The owner cannot validate	`validate`	that check is the whole of "independent"
The inventory is the gate	`promote`	the only thing that keeps it current
Every blocker at once	`promote`	one-at-a-time teaches teams to resent the gate
`trace_id` is the design	`Artifact`	retrofitting it loses the first N months forever
Causal order at write time	`add`	a graph you cannot walk is not evidence
Ancestors = the examiner's question	`ancestors`	"what did it use to decide?"
Descendants = the impact query	`descendants`	"this was wrong; what did it affect?"
Orphans are worse than gaps	`orphans`	a disconnected artifact looks like evidence
Residency is proved per record	`ResidencyChecker`	config says what should; records say what did
Unprovable is a violation	`check_trace`	a record without a region is not evidence
The forgotten pin	`retrieval_snapshot`	the corpus moves; the query does not reproduce
State the caveat	`caveats`	bit-reproducibility for a sampled model will be tested
An untested exit is a document	`ExitReadiness`	the only convincing answer is live traffic
Concentration aggregates per provider	`assess_concentration`	two models, one vendor, one risk
Generated, not assembled	`EvidenceGenerator`	a reconstruction has gaps it cannot explain
Missing artifacts fail loudly	`generate`	an engineering problem, not an audit finding
Chain and signature	`verify`	one catches an edit, the other catches a re-hash
Silent controls must be named	`control_coverage`	they need other evidence, and you must know which

Files

File	Role
lab.py	your implementation
solution.py	reference; `python solution.py` runs an eight-part worked session
test_lab.py	124 tests
requirements.txt	`pytest`

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

All 124 tests green against your lab.py.
An irreversible action is Tier 1 regardless of value; a Tier 3 assignment still gives a reason.
A higher tier caps autonomy lower, demands more eval cases, and monitors more often.
The fingerprint ignores config_id and version, and changes for prompt, retrieval, tools, guardrails and temperature.
A model with no owner, no business sponsor or no purpose cannot be registered.
The owner cannot validate their own model.
promote reports every blocker, not the first.
A prompt change resets validation to not_submitted and removes it from production.
An artifact deriving from an unknown parent is refused at write time.
A diamond reports each ancestor once; a cycle is detected.
An out-of-region inference is flagged and a record with no region is flagged.
A trace missing only the retrieval snapshot is reported as non-reproducible.
A non-zero temperature is a caveat, not a failure.
Traffic share aggregates per provider, not per model.
A LIVE exit path needs no exit test; an IDENTIFIED one is a finding.
A 250,000 action with no approval record fails with the artifact named.
Editing, dropping or re-signing an artifact all break verification.
Coverage is generated from the catalogue and names the silent controls.

How this maps to the real stack

This lab	The real thing	What we simplified
`ModelInventory`	an internal GRC system, ServiceNow, or a model-risk platform	no workflow, no approvals routing, no attestations
`assign_tier`	a bank's own model-risk tiering standard	four inputs; a real one has twenty and a committee
`LineageGraph`	OpenLineage/Marquez, Purview, or an internal lineage service	in-memory; no persistence, no cross-system federation
`Artifact`	records in six different stores, joined on a trace id	one store; the join is the hard part in reality
`ResidencyChecker`	a query over inference logs + Azure Policy compliance state	no real logs, no region metadata from the platform
`check_reproducibility`	a deployment manifest plus an eval harness	pins as strings; no actual re-execution
`ThirdPartyRegister`	a vendor-risk system with contracts attached	no contract text, no renewal dates, no assessments
`EvidenceGenerator`	a report generator over the audit store	no persistence, no access control, no redaction for the recipient
`CONTROL_CATALOGUE`	a control library mapped to a framework, in a GRC tool	hand-maintained; a real one is versioned and attested

Honest limits. The lineage graph is in-memory and single-system; the genuinely hard part in production is that the six artifact kinds live in six stores with different retention, different access control and different query languages, and federating them is most of the work. There is no redaction: an evidence pack shown to an external examiner should not contain another customer's data, and deciding what to withhold without breaking the chain is a real design problem. The reproducibility check verifies that pins exist, not that re-execution works — the only honest test is to actually re-run against the pinned snapshot, which needs the snapshot to still exist, which is a retention decision. Residency is checked from records the platform emitted about itself, so it inherits their trustworthiness — pair it with Phase 13's topology proof rather than relying on either alone. And tiering here is a function of four booleans; a real standard has a committee, precedent, and appeals.

Extensions

Federate the lineage. Put the artifacts in three different stores with three different schemas and join them at query time. The pain you feel is the actual problem.
Redaction for external disclosure. Produce a pack for an examiner that omits other customers' data while keeping the chain verifiable — which means hashing what you withhold and proving the omission is complete.
Actually re-execute. Take a pinned trace, restore the retrieval snapshot, re-run, and diff. Then discover which pins were insufficient.
Retention tiering. Evidence for 7 years, debugging traces for 30 days. Model the archive, the restore path, and the cost.
Attestation. Publish the pack's chain head to an append-only external store hourly, so the chain becomes tamper-resistant rather than tamper-evident (Phase 10).
A validation pack template for an agentic system: scope, limitations, eval results, red-team results, monitoring plan, and the conditions under which approval lapses.
Continuous control monitoring. Run verify_control_evidence over a sample of production traces daily, and alert when a control stops emitting — which is how you find out a control was removed by a refactor.
Map to the EU AI Act. Add its obligations as a framework in the catalogue and see which controls you already have and which you do not.

Interview / resume bullets

"Built the platform's evidence engine: every layer emits a lineage artifact keyed on a shared trace id, so an examiner's question — who authorized this, under which policy, using what data, from which model version — is answered by a generated, signed, hash-chained pack rather than by a reconstruction."
"Made the pack generator refuse to produce an incomplete bundle, naming the missing artifact — so an evidence gap surfaces as an engineering ticket during development rather than as an audit finding two years later."
"Argued and implemented that the agent configuration as a whole is the model: a prompt, retrieval, tool-set or guardrail change invalidates validation and removes the agent from production, which brought agentic systems inside the bank's existing SR 11-7 framework."
"Tiered models by business impact rather than technique, and wired the tier to the autonomy band — so risk classification determines what an agent may do without a human, instead of sitting in a register."
"Proved data residency per inference record rather than asserting it from configuration, and paired it with a network reachability proof so the claim held from two independent directions."
"Replaced a paper exit plan with an exit-readiness ladder where the top rung is live production traffic on the alternative — which turned concentration risk from a paragraph into an architecture."

Enterprise AI & Agentic Platform — Engineering & Architecture