« Phase 10 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 10 — Principal Deep Dive: Agent Security — Prompt Injection, Threat Modeling & Guardrails

The lab is a 300-line synchronous function. A production agent platform is a fleet of gateways, a queue, a policy service, an audit pipeline, and a human-approval UI — but the architecture is the same five layers, and a principal's job is to know where each one physically lives, what it costs, how it fails, and how big the crater is when it does. The harness collapses that into one process so the shape is legible. This doc re-expands it into a system.

Where the trust boundary actually lives

In the lab, the boundary is if defenses.trust_boundary and item.trust is not Trust.TRUSTED: continue. In production, that line is an architectural seam, and the strong form of it is the dual-LLM / quarantine pattern (Simon Willison) and its formalization CaMeL (DeepMind). The design principle: the model that acts (has tools, holds credentials, can spend money) must never receive untrusted text in a position where that text can become an instruction. Untrusted content — a fetched page, a RAG chunk, a tool result — goes to a quarantined model with no tools and no secrets, whose only permitted output is structured data against a schema (extract these fields, return this JSON), never free-form directives. The privileged planner LLM then operates on that structured data as values, not as prose to obey. CaMeL goes further: a trusted planner emits a control-flow program, untrusted content only ever populates variables, and a capability system tracks data provenance so tainted values are physically barred from flowing into privileged sinks.

The principal insight is that this boundary is not free and not always affordable. It roughly doubles model calls for any untrusted-content path, adds a schema-design burden per data source, and constrains what the agent can express to what your schemas anticipated. So the real decision is scoping: which data paths cross a genuine trust boundary and warrant quarantine, versus which are low-stakes enough for delimiting/spotlighting plus an output guard. Answering that requires a threat model, not a vibe — enumerate assets (secrets, customer data, the power to spend/send/change state), the boundaries where untrusted text enters (user, RAG, fetched, tool results, memory, connected MCP servers), and the reachable sinks. The trust boundary goes on the paths where a tainted value can reach a high-value sink.

The performance and cost envelope

Every layer is on the request's critical path, and they have wildly different cost shapes:

• Trust boundary / dual-LLM: +1 model round-trip on untrusted paths — the dominant latency and dollar cost. At 200–800 ms per quarantine call and per-token pricing, this is the line item a CFO notices. You amortize it by caching quarantine results keyed on content hash (a fetched page's extraction is stable) and by only quarantining paths the threat model flags.
• Input/output guards: if regex/heuristic (like the lab's detect_injection), microseconds and free; if a classifier (Llama Guard, a Lakera/Rebuff-style detector, Bedrock Guardrails), another model call — cheaper than the main model but real, and now two extra calls per turn (input + output). Guard latency is the tax you pay on every turn, benign or not, which is why the cheap regex prefilter in front of the expensive classifier is standard.
• Least privilege: effectively free — it is a set membership test at tool-dispatch time. This is why it is the highest-leverage control per dollar: deterministic, zero marginal latency, and it bounds blast radius regardless of model behavior.
• HITL: latency is unbounded (a human might approve in seconds or never), so it cannot sit in a synchronous request. It must be a durable suspend — the workflow parks holding no compute (this is Phase 08's signal pattern), an approval task lands in a queue/UI, and the run resumes on the signal. Designing HITL as a blocking call is the classic junior mistake; designing it as a durable pause is the principal move.

Budget it as: one main model call is your latency floor; guards add a bounded multiple; the trust boundary adds a full extra call on tainted paths; HITL removes itself from the latency budget entirely by going async. If your p99 is blowing up, the culprit is almost always synchronous classifier guards on the benign 99% of traffic — move them behind a cheap prefilter and sample.

Failure modes and blast radius

Reason about each control's failure independently, because defense in depth is exactly the claim that they fail independently:

• Mislabeled provenance (a tool result marked TRUSTED) silently disables the trust boundary for that path — the worst failure because it is invisible and the tests still pass. The mitigation is that provenance labeling is centralized at ingestion and fails closed: unknown source ⇒ UNTRUSTED.
• Guard bypass by paraphrase: an adversary rewrites "ignore previous instructions" into something the regex/classifier misses. Blast radius here is bounded by least privilege — this is why you never let the guard be the only thing between untrusted content and a high-value sink.
• Over-broad allow-list: the summarizer that "temporarily" got an email tool for a demo and never lost it — OWASP LLM08 excessive agency. Blast radius = everything that tool can touch. Least privilege must be reviewed as tools are added; the threat model is revisited every time a tool or data source is added, not once.
• Approval fatigue: gate too much and humans rubber-stamp; test_hitl_approves_lets_it_through is the encoded warning. The design lever is keeping HIGH_IMPACT short so approvals stay rare, legible, and meaningful.
• Empty allow-list fails open: if defenses.allowed_tools treats the empty set as "no policy." In the lab that is pedagogical clarity; in production the default must invert — no explicit grant means deny.

Cross-cutting concerns a principal owns

Multi-tenancy: the trust boundary is per-request, but the blast radius is per-tenant. A hijacked agent that stays inside its tenant is an incident; one that crosses tenants is a breach (Phase 13). Least-privilege tool scopes must be tenant-scoped — "send email" means "send email as this tenant, to this tenant's contacts," enforced below the agent, not by the agent.

Observability: blocked_by is not a lab artifact — it is the security event schema. Every block is a signal: a spike in output_guard blocks means an active exfil attempt or a broken upstream control; a spike in trust_boundary drops means your quarantine is doing its job. You want per-layer counters, the injected content captured for red-team review, and alerting on the layer that fired, because a leak caught only by the last line (output guard) means the first four failed and that is a design regression even though no data left.

Cost of the LLM-judge guard: contextual-grounding and toxicity classifiers are models, and models are a budget line and a latency line and a new attack surface (the judge can itself be injected). Principal judgment is knowing that a classifier reduces risk and never eliminates it, so it is layered with the deterministic controls, never instead of them.

The "looks wrong but is intentional" decisions

Three design choices that reviewers flag and shouldn't. First: the output guard is last, after the model has already decided to leak. That looks like closing the barn door too late — but it is the deliberate assumption that every earlier layer can fail, and it is your last chance to stop bytes before they hit a browser or an outbound socket. Second: HITL is allowed to let the leak through when the human approves. That looks like a hole — it is honesty: the control's strength is exactly the human's, and pretending otherwise builds false confidence. Third: the controls are five small, dumb, independent checks rather than one smart unified policy engine. That looks unsophisticated — it is the opposite. Independence is the security property; a single clever control is a single point of failure, and against an adaptive adversary the only durable posture is layers that fail for different reasons. The naive approach — one great system prompt, or one great classifier — fails not because it is weak but because it is singular.