« System Design · Track Overview

Design 05 — The Regulator-Grade Evidence Platform

"An examiner picks one agent action from eleven months ago and asks you to justify it. Design the system that answers."

The question it turns on: can you design so that evidence is generated, not assembled?

Table of Contents

• 1. Constraints before components
• 2. The seven questions an examiner asks
• 3. Generated, not assembled
• 4. The artifact set, and the join key
• 5. The six pins, and the one everybody forgets
• 6. Tamper evidence
• 7. Completeness: a missing artifact names itself
• 8. Lineage, and the questions it answers
• 9. Residency, proved per record
• 10. Retention, storage and cost
• 11. Failure modes and blast radius
• 12. What you build first
• 13. What changes at 10×
• 14. The questions you will be asked

1. Constraints before components

The two that shape the design most:

Seven years outlives everything. The service that wrote the record will have been rewritten twice. So the record must be self-describing: versions, not references to a config service that will not exist.

A non-engineer reads it. The pack has to render into something an examiner can follow without a query language. That is a product requirement, not a nice-to-have, and it is the difference between "we have the data" and "we can answer."

2. The seven questions an examiner asks

Everything in this design exists to answer these, in this order:

Two properties of that list are the design:

Each question maps to exactly one artifact type. If answering a question requires joining four sources and reasoning, the answer will be produced late, by an engineer, under pressure, and it will be wrong once.

Question 5 includes non-blocking denials. A barrier that removed a document is a control acting. An evidence pack showing only the controls that halted the request understates the platform's behaviour, and understating your controls to an examiner is a strange choice.

3. Generated, not assembled

The distinction the whole design rests on.

Assembled — after the run, a collector walks logs, traces and database rows and builds a pack. This is what most platforms do, and it fails in a specific way: the collector asks "what can I find out?" rather than "what did I do?" Anything not logged is unrecoverable, and you discover which things those are during an audit.

Generated — each step emits its artifact at the moment it has the information, into the run's artifact list, which is written once at the end.

def emit(kind, **attrs):
    artifacts.append({"kind": kind, "trace_id": request.trace_id,
                      "tick": now(), **attrs})

Three consequences worth stating:

The join key is set in one place. Not by seven emitters, six of which remember. The realistic failure mode is that the approval record is the one missing the trace id — and the approval is the record the examiner most wants to link.

Emission is on the request path, and that is deliberate. ~5 ms. If evidence emission is asynchronous and best-effort, then under load — exactly when incidents happen — evidence is the thing that gets dropped. Buffer the write to storage; do not make the capture optional.

A denial emits too. policy_decision with effect="deny" is written on refusals. Without it, "the control refused" and "the control never ran" are indistinguishable in the record, and the second is the thing an auditor is actually testing for.

4. The artifact set, and the join key

                        trace_id: t-2026-03-11-771  ◄── the join key, on ALL of them
   ┌──────────────────────────────────────────────────────────────────┐
   │ session         user, channel, tenant, chain, auth method        │
   │ policy_decision effect, policy_version, reasons     (even deny)  │
   │ retrieval       partitions, doc_id@version[], snapshot, removed[]│
   │ guardrail       stage, verdict, score, document/tool             │
   │ inference       the six pins, tokens, cost, region, temperature  │
   │ delegation      to, full chain, depth                            │
   │ execution_step  step number, action                              │
   │ approval        approvers[], rationale, timestamp                │
   │ action          tool, args, idempotency key, reference, outcome  │
   │ span            name, start, end, self-time, parent              │
   │ sli_event       valid?, latency bucket                           │
   └──────────────────────────────────────────────────────────────────┘

One key, one place. Every artifact carries trace_id. The pack is only a pack if the pieces link, and the failure mode is precisely one artifact type missing the field.

Records are self-describing. policy_version: "2026-03-11.4", not policy_ref: "current". In 2033 the policy service is gone; the string is still meaningful, and the bundle itself is retained alongside.

Structured, not prose. {"tool": "payments.release", "value_micros": 250000000000} rather than "released payment PMT-771 for 250k". Prose is unqueryable and it drifts.

5. The six pins, and the one everybody forgets

Reproducibility means: given this pack, could a competent third party re-run the decision and understand the output? That requires six things pinned:

The sixth is the one that gets missed, and it is the one that quietly breaks reproduction: pin the other five, re-run six months later against a re-indexed corpus, and you get a different answer with five matching pins and no explanation for the difference.

And temperature. temperature: 0.0 in the record. Not because it makes the model deterministic — it does not, entirely — but because the value used is part of the configuration, and an examiner asking "was this sampled?" deserves an answer.

The framing that lands with Model Risk:

"The agent's configuration IS the model." Same weights, different prompt, different tools, different policy — a different model for risk purposes, requiring its own validation. The pins are that configuration's identity, and the fingerprint over them is what tells you whether two runs used the same one.

6. Tamper evidence

A hash chain over the artifacts:

head = previous_head or "0" * 64
for artifact in artifacts:
    material = json.dumps(artifact, sort_keys=True, separators=(",", ":"))
    head = sha256(head + material).hexdigest()

Three details that are correctness rather than style:

Canonical JSON. sort_keys=True and fixed separators. A chain computed over a non-canonical encoding verifies only on the machine that wrote it — a different library, a different insertion order, and every historical head becomes unverifiable.

Tamper evidence, not tamper prevention. Anyone who can rewrite the store can recompute the chain. Say this before you are asked; claiming otherwise is the fastest way to lose a security reviewer.

Which is why the head is published outside. Anchor the daily head somewhere the platform cannot reach: a WORM blob with a legal hold, a different trust domain, or an external transparency log. The chain only proves something if verification does not depend on the thing being verified.

The upgrade path, if the requirement grows: a Merkle history tree (Crosby & Wallach) gives inclusion and consistency proofs without rereading the whole log, which is how Certificate Transparency works and is the right model if a third party must verify independently.

7. Completeness: a missing artifact names itself

required = {"session", "policy_decision"}
if outcome in (COMPLETED, DEGRADED):
    required |= {"retrieval", "inference", "execution_step"}
if any_action:
    required.add("action")
    if any(value >= dual_control_threshold):
        required.add("approval")
missing = sorted(required - present)

The contract depends on the outcome. A denied run has no inference to record; requiring one makes every correct denial look like an evidence failure. So the check runs after the outcome is known.

Missing artifacts are named. evidence_complete: false sends somebody hunting. evidence_missing: ["approval"] sends them to the approver. One line of code; enormous difference during an incident.

Fail loudly, at development time. A pack with a hole discovered in development is an engineering ticket. The same hole discovered in an audit is a finding, a remediation plan, and a follow-up examination.

And the honest limit, stated first: the check verifies presence, not truth. An artifact can be present and wrong. Presence is mechanically checkable; truth needs independent validation (SR 11-7's effective challenge) and a human panel. Anyone claiming their evidence system proves correctness has not thought about it.

8. Lineage, and the questions it answers

Artifacts answer "what happened in this run". Lineage answers "what else is affected".

   dataset ──► embedding model ──► index snapshot ──► retrieval ──► inference ──► action
      │                                                                │
      └──────────────────► eval run ──► validation ──► approval ───────┘

Two directions, two different bad days:

Ancestors — "this action was wrong; what produced it?" Walk back to the documents, the index snapshot, the model version, the policy bundle.

Descendants — "this document was wrong / this model was withdrawn; what did it affect?" Walk forward to every action that depended on it. This is the query that runs during an incident, at 2 a.m., and it is the one people do not build until they need it.

Store lineage as an append-only edge list with causal-order enforcement: an edge from a node created later to one created earlier is rejected at write time, because it is a bug and the graph is useless once it contains one.

9. Residency, proved per record

"Our data stays in the UAE" is an assertion. This is proof:

{ "trace_id": "...", "data_classification": "confidential",
  "processing_region": "uaenorth", "storage_region": "uaenorth",
  "model_endpoint": "gpt-frontier-uaenorth", "index_region": "uaenorth",
  "egress": [] }

Per record, per stage. The examiner's question — "show me that no confidential data was processed outside the country in March" — becomes a query with a count, not a conversation about architecture diagrams.

Unprovable is a violation. A record whose processing region is absent is not "probably fine". It is a gap, and it must be reported as one; treating missing evidence as compliance is exactly the habit the whole design is against.

And the fallback path is where this breaks. The primary is in-region and everyone knows it. The fallback is chosen at 03:00 by a router, and if the residency gate is not inside the router, the first time you learn about it is from this query. (Design 02.)

10. Retention, storage and cost

Rough sizing: ~5 KB per action across all artifacts × 8,000/day ≈ 40 MB/day, ~15 GB/year, ~100 GB over the retention period. Storage is not the problem — say this, because people assume it is. The costs that matter are:

• Query cost at the cold tier — an examination that scans two years of object storage is a real bill and a real wait. Partition by date and tenant, and keep an index of trace ids.
• Rendering — turning a pack into an examiner-readable document is engineering work, and it is the part that gets skipped.
• Residency of the evidence itself. Evidence about UAE data is UAE data. It cannot all land in one global log sink, which is a constraint on your observability vendor.

Legal hold overrides retention. When litigation or an examination is live, deletion stops — for the specific traces, which means the deletion path must be selective. A retention job that cannot be scoped is a compliance incident waiting to happen.

11. Failure modes and blast radius

Completeness rate belongs on a dashboard. It is a slow, silent failure: a refactor stops emitting approval, nothing breaks, tests pass, and eleven months later an examiner asks. A daily completeness percentage catches it in a day.

12. What you build first

1. The trace id and emit(). Before anything else. Every artifact written before the join key exists is unlinkable forever, which makes this the most expensive thing to retrofit in the entire platform.
2. session, policy_decision, action. The three that answer who, permitted? and what happened. Those three alone answer most questions.
3. The completeness check. Cheap, and it stops the set from silently shrinking.
4. inference with the six pins. Before the first model upgrade, because the first upgrade is when unpinned records become unreproducible.
5. retrieval with versions and the snapshot. Before the first re-index.
6. The hash chain. Once the artifact schema has stopped moving — chaining a schema still in flux just produces unverifiable history.
7. Lineage. Descendants-first: "what did this affect" is the incident query.
8. Rendering. The examiner-readable document. Last, and do not skip it — an unreadable pack is a pack you cannot use under time pressure.

13. What changes at 10×

80,000 actions/day, 200M records/year.

Hot-tier query cost dominates. Trace-id lookup must be an index, not a scan. Partition by (date, tenant) and keep a secondary index on user and agent — those are the two dimensions investigations actually use.

Sampling becomes tempting, and must be refused for evidence. Sample spans; never sample artifacts. A sampled evidence pack is not an evidence pack, and the action that was sampled out is the one you will be asked about.

Schema governance becomes real. Twelve teams emit artifacts; the schema needs a registry, a compatibility rule (backward for readers) and a deploy order — Phase 12's argument applied to your own telemetry.

Rendering becomes a product. With ten examinations a year, someone will build a self-service evidence portal. Better that it is you, with the pack as its API.

Cross-region evidence federation. UAE evidence in the UAE, EU evidence in the EU, and a query layer that fans out without moving data. Which is the same structural-isolation argument as Design 04, applied to the log.

14. The questions you will be asked

"How do I know the agent didn't do something you're not logging?" — Because the action gateway is the only path to a side effect, and it emits before it executes. If an action has no artifact, it did not go through the gateway — and nothing else can reach the estate, which is enforced at the network layer, not by convention.

"Can you prove these records weren't altered?" — The chain gives tamper evidence: altering a record invalidates every subsequent head. It is not tamper prevention — anyone who can rewrite the store can recompute the chain, which is why the daily head is anchored outside the platform, in a WORM store the platform cannot write to.

"Reproduce this decision from eleven months ago." — Six pins plus the retrieval snapshot. I can reproduce the configuration exactly and the inputs exactly. The model's sampling is not bit-reproducible even at temperature zero, so I report reproduction with that caveat stated rather than claiming determinism I do not have.

"Your evidence says the control ran. How do I know it worked?" — You do not, from this. Presence is checkable; correctness is not. That is what independent validation, the red-team suite and the defence-depth harness are for — and they produce their own evidence, which is what I would hand you for that question.

"What if a control silently stops emitting?" — Completeness rate is an SLI with an alarm. It is the specific slow failure this design is most exposed to, so it is the one I instrument most directly.