« Phase 11 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: containment against capability
• 2. Deterministic or model-based
• 3. Setting the thresholds
• 4. Where masking happens
• 5. Buy or build the detectors
• 6. The approval budget
• 7. What to tell the business about injection
• 8. The MNPI conversation
• 9. Third-party MCP servers
• 10. Measuring a control that has never fired
• 11. Migration: adding guardrails to a live platform
• 12. What I would not build

1. The central tension: containment against capability

Every control here costs the agent something it could otherwise do.

   CAPABLE                                                        CONTAINED
     │                                                                 │
   no taint     taint blocks    taint blocks      structured      Dual LLM /
   tracking     irreversible    all writes        extraction      CaMeL
     │              │                │                │               │
   anything      most flows      approvals        no prose       67% of tasks,
   possible      still work      everywhere       crosses        provably safe

You cannot sit at one point for everything, and the principal move — the same as in Phase 09 and Phase 10 — is per class of action:

Stating that table is the answer in a design review. Choosing one point for everything fails in both directions: maximum containment produces an agent that cannot act, which gets exemptions; minimum containment produces the incident.

2. Deterministic or model-based

That last row deserves attention: a model-based guardrail is a model reading attacker-controlled text. It has the same vulnerability as the thing it protects, and there are published attacks that defeat the guardrail and the primary model with one payload.

The rule I would defend:

Deterministic controls enforce. Model-based controls observe.

Run the classifier asynchronously, feed its score into the anomaly signal (Phase 09), and let that affect authorization on the next request. You get the recall without putting a probabilistic, injectable, 300 ms component on the synchronous path.

The counter-argument — "but the classifier catches things patterns miss" — is true and does not change the placement. It catches them a few seconds later, into a signal that has consequences.

3. Setting the thresholds

Injection block threshold. High — 0.85. The cost of a false positive is a legitimate document dropped from context, which produces a wrong answer with no indication anything was removed. That is worse than it sounds: the agent does not know it is missing something, so it answers confidently from what remains. Because the taint rule catches what the scanner misses, the scanner can afford to be conservative.

Escalate threshold. 0.5. Retain the content, flag it, feed the anomaly score. This band is where most true positives live.

PII detection. Only classes with a checksum or a rigid format enabled by default. Adding name-detection means adding NER, which means measuring precision on your own corpus first. A class you cannot measure is a class that will be exempted.

Approval threshold. Not an engineering number. Get it from operational risk, in writing, with a name attached — see §6 for why the number is really about volume.

Egress allow-list. If it exceeds ~20 hosts, it has stopped being a control. Every addition needs an owner and a review date; without a review date the list only grows.

Barrier sync latency. Removals within minutes, additions within hours. The asymmetry is the same as revocation everywhere else in this track: a late grant is an inconvenience, a late removal is a finding.

4. Where masking happens

Three placements, and the choice is genuinely contested:

Mask at output only is usually right, and it is only defensible because of the other controls: the model holds real values, but egress is allow-listed, the output is scanned, and the context is not logged. Remove any one of those and it becomes indefensible.

Mask at retrieval is right when the model is external — a third-party API, a shared endpoint, a region you do not control. Then the data has left your boundary and no downstream control helps.

The decision rule: mask at retrieval when the inference boundary is outside your trust boundary; mask at output when it is inside. Which makes it a consequence of the deployment decision from Phase 05 rather than an independent choice — and that link is worth making explicitly, because the two decisions are usually made by different people.

5. Buy or build the detectors

The line: buy the classifiers, build the architecture. A PII classifier is a machine-learning problem with a vendor. Taint propagation, the barrier model and the containment rule are your control design and nobody will ship them for you.

And the trap: buying a "guardrails platform" and believing it covers this phase. Those products are strong on detection and, in every one I have evaluated, silent on containment — no taint model, no tainted-action rule. Detection is the part you can buy; containment is the part that works.

6. The approval budget

The number that actually determines whether HITL works is not the threshold. It is the volume.

   approvals per reviewer per day  ×  seconds of genuine attention  =  the budget

At 200 approvals a day, a reviewer has under two minutes each including context switching, and they will approve almost everything. At 10, they can read the evidence.

So set the threshold from the volume you can staff, not from a risk number in isolation:

1. Measure the action distribution — how many actions per day at each value band?
2. Decide how many approvals a reviewer can do thoughtfully. Ten to twenty per day is realistic; fifty is not.
3. Set the threshold where the volume lands in that budget.
4. If the threshold that fits the budget is higher than risk accepts, you need more reviewers or fewer actions — not a lower threshold and the same reviewers.

The metric to watch afterwards is the rejection rate. If it is under 1%, either the agent is excellent or the reviewers are clicking. Distinguishing those two requires sampling approved actions and having someone independent review them — which is worth doing, because the answer changes what you do next.

And the escalation path when the answer is "they're clicking": reduce volume, improve the evidence shown, or accept that approval is a logging mechanism rather than a control and stop claiming otherwise.

7. What to tell the business about injection

You will be asked: "is our agent safe from prompt injection?" The honest answer is uncomfortable and it is much better to give it early than after an incident.

What not to say: "Yes, we have guardrails." It is not true, and the person asking will repeat it to a regulator.

What to say: "Prompt injection cannot be prevented — it is a property of how language models work, and no vendor has solved it. What we do is bound the consequence. An injected instruction can make the agent read things and can make it produce a wrong answer. It cannot move money, cannot send data outside the bank, and cannot act on anything consequential without a human who sees where the instruction came from."

Then the residual risks, named:

Naming the last one is what makes the rest credible. It is also what justifies the red-team suite's budget, because the suite exists to catch exactly that.

And the framing that lands with a risk committee: this is the same posture as with people. An employee can be socially engineered. You do not solve that by making them un-foolable; you solve it with dual control, limits and audit — which is precisely what this is.

8. The MNPI conversation

This is the one where an engineering decision becomes a regulatory one, and it usually surfaces late.

The sequence that catches teams out:

1. The knowledge platform indexes "all internal documents" because that is what makes it useful.
2. Somebody notices deal documents are in the index.
3. Compliance asks who can retrieve them.
4. The honest answer is "anyone who asks the right question", because relevance ranking has no concept of a wall.

By then the index is built and re-indexing with barrier metadata is weeks of work.

The position to take early: documents without a classification and a barrier field are not indexed. It slows the initial rollout and it is much cheaper than the retrofit. Expect resistance — "we'll add metadata later" — and the counter is that the retrieval index is the control point, so metadata that arrives later arrives after the exposure.

Two more things worth raising before anyone asks:

Embeddings leak. A vector derived from an MNPI document sits in the same index as everything else. Even with filtering, an attacker with query access can probe the space and infer that something exists about Zenith Bank. The answer is separate indexes per barrier, not a filter on a shared one — which costs money and is the correct answer.

Retrieval logs are MNPI. "Layla searched for Zenith Bank acquisition terms" is itself material information about a deal. The log needs the same barrier as the documents, which surprises everyone the first time.

9. Third-party MCP servers

The supply-chain question specific to this phase, and it will be asked.

A third-party MCP server supplies tool descriptions that enter the model's context on every turn, and receives whatever arguments the agent sends. It is simultaneously an injection channel and an exfiltration channel, and it is registered as configuration.

The tiering I would defend:

And the controls that make tier 2 workable:

• Hash-pin the tool descriptions at approval. Re-approval on any change. This closes the rug pull, and it costs a dictionary lookup.
• Scan descriptions with the same injection scanner as documents.
• Scope the credential to what that tool needs (Phase 08) — the server sees only what it was sent.
• Classify what may be sent to it. A third-party server does not receive restricted data, regardless of what its tool schema accepts.
• Egress-list its host, so the arguments' destination is the one you approved.

The question that decides tier 3: "would we let this vendor read every argument the agent sends, forever, without a contract?" If the answer is no — and it always is — the server is not registered, however useful it looks in a demo.

10. Measuring a control that has never fired

A guardrail with zero blocks in six months is either working perfectly or not running at all, and the log looks identical.

Five ways to tell them apart:

Canary documents. Plant documents containing known injection payloads in the index. The scanner must find them on schedule. If it stops, you know within a day rather than after an incident.

Canary tokens. A unique fake account number in the system prompt. If it ever appears in output or an egress attempt, you have detected a leak with certainty rather than a heuristic — and the value is that it distinguishes "no leaks" from "no detection".

Synthetic traffic. Run the red-team suite against production configuration continuously, not only in CI. Configuration drifts; a new tool or a widened exemption breaks a case that CI passed last week.

Assertion counters. Count how often the taint rule evaluated, not just how often it blocked. If evaluations drop to zero, the code path is gone — a refactor removed it, and nothing failed.

Chaos. Deliberately disable a control in a non-production environment and confirm the red-team suite fails. If it still passes, the control was not what was protecting you, and you have learned something important.

The fourth one is the cheapest and the most often missing. guardrail_evaluations_total next to guardrail_blocks_total turns "no blocks" from ambiguous into diagnosable.

11. Migration: adding guardrails to a live platform

Starting state: agents in production, no taint tracking, PII flowing to logs, no egress control.

Phase 1 — observe. Deploy the whole chain in log-only mode. Every stage records what it would have done. Free, and it tells you the true rate of tainted-side-effecting actions — which is usually higher than anyone expects and is the business case.

Phase 2 — egress allow-listing. Enforce first, because it is the lowest-false-positive control and it closes the highest-severity channel. The allow-list is short and the breakages are obvious and quick to fix.

Phase 3 — masking at output. Second, because it does not affect agent behaviour at all — only what humans see. Zero risk to functionality.

Phase 4 — the barrier filter. Before the index grows. This is the one with a regulatory deadline attached, and it is the one that gets harder every week.

Phase 5 — the taint rule, irreversible actions only. The narrowest possible enforcement of the most important control. Low volume, high value, and it proves the approval path works.

Phase 6 — extend the taint rule to all writes. By now the approval flow is exercised and the reviewers are trained.

Phase 7 — injection scanning in enforce mode. Last, deliberately. It has the highest false positive rate and the lowest marginal value once the taint rule holds.

The ordering is the opposite of what teams do naturally — scanning first, because it is the visible one that a demo shows. Scanning first means false positives before value, which spends the goodwill you need for the controls that matter.

12. What I would not build

A prompt-injection classifier. It is an active research area with a full-time adversary. Buy one, run it asynchronously, and do not put it on the enforcement path.

A NER model for PII. Presidio exists, it is good, and the marginal quality you would add is negative.

"Instruction hierarchy" in the prompt. Elaborate delimiters, XML tags, "the following is untrusted". These help marginally and encourage exactly the belief this phase exists to demolish. Use one clearly-marked block, and put nothing load-bearing on it.

A guardrail that reasons about intent. "Does this action seem like something the user wanted?" is a model call, and it is injectable by the same content it is judging.

My own tokenization scheme. FF1, key rotation, a vault with its own residency story. Buy it.

A universal guardrail service for the whole bank. The controls are context-specific — what counts as sensitive, which barriers apply, which egress hosts are legitimate. A shared service ends up with the union of every exemption, which is no control at all.

Semantic exfiltration detection. "Is this response leaking data?" is unanswerable in general, and attempting it distracts from the allow-list that actually works.