« Phase 10 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 10 — Core Contributor Notes: Agent Security — Prompt Injection, Threat Modeling & Guardrails

Our harness is a miniature. The real systems it mirrors — NeMo Guardrails, Llama Guard, Bedrock Guardrails, Lakera, Rebuff, and the CaMeL/dual-LLM research line — each implement the same five ideas, but the interesting knowledge is in how they do it, what they got wrong first, and where our stdlib version deliberately lies to keep the point visible. This is the committer's-eye view.

Llama Guard: the input/output guard as a fine-tuned model, not a regex

Our detect_injection is a list of seven regexes. Meta's Llama Guard is the production shape of that same layer: a Llama model fine-tuned as a safety classifier that takes a conversation plus a taxonomy of hazard categories and emits safe/unsafe and, if unsafe, which categories. The non-obvious design decision that a maintainer internalizes: Llama Guard is deliberately a separate model from the one being guarded, because a model cannot reliably police its own output in the same forward pass — the guard has to be an independent judge with its own prompt it does not share with the untrusted content. The taxonomy is promptable: you pass the categories at inference time, which is why Llama Guard 2/3 shipped as much a format and taxonomy as a weight file. The sharp edge committers hit: it classifies safety (violence, hate, self-harm, etc.), and prompt injection is not natively one of its core categories — teams bolt injection detection on as a custom category or a separate detector and are surprised Llama Guard alone does not stop an exfil beacon. Our lab's split between detect_injection (injection heuristics) and generic content safety is the honest version of that gap.

NeMo Guardrails: Colang and the runtime, and why it is not the trust boundary

NVIDIA's NeMo Guardrails implements guards as a dialog-flow runtime. You write rails in Colang, a DSL where you define canonical user intents, bot flows, and — critically — input rails and output rails that run before and after the LLM. Under the hood it embeds your defined intents, does semantic matching to route an utterance to a flow, and can invoke check_facts / check_jailbreak / moderation actions as rail steps. The maintainer's nuance: NeMo's power is that a rail can short-circuit the conversation deterministically — if the input rail matches a jailbreak pattern, the flow never reaches the main LLM. That is exactly our input_guard quarantine. But — and this is the misconception NeMo's own docs fight — a dialog rail is still a heuristic; the jailbreak check is a classifier or an LLM self-check, and an adversary paraphrases around it. NeMo gives you the machinery to place a control; it does not give you the trust boundary, because Colang rails still operate on untrusted text in the same privileged context. The categorical fix (dual-LLM) sits architecturally above what a rails framework can express. Knowing that boundary is the difference between "I configured NeMo" and "I understand what NeMo can and cannot enforce."

Bedrock Guardrails: content policy is not access control

AWS Bedrock Guardrails productizes the input/output guard as a managed policy object: content filters (hate/violence/etc. with tunable strength), denied topics, word/PII filters with redaction, and — the underrated one — contextual grounding checks that score whether a response is supported by the provided source and whether it is relevant to the query. A maintainer's framing that consistently sorts senior from junior: Guardrails is content safety, it is not authorization. It answers "is this text OK to show a human," never "is this agent allowed to call this tool" — that is IAM's and the application's job. The classic interview trap "so your guardrail blocks unsafe tool calls, right?" is answered no; conflating the two tells a reviewer you understand neither. Contextual grounding is the feature teams wire up last and need first: content filters look for toxicity, they cannot detect a fluent, polite, entirely fabricated answer — grounding is the only mechanism aimed at "is this claim supported by anything real." Our lab does not model grounding, which is a deliberate simplification; in a real RAG-backed agent it is a first-class control.

Rebuff and Lakera: the multi-layer detector, and canary tokens

Rebuff is worth studying because it made its layered design explicit: a heuristic filter, a dedicated detection LLM, a vector store of known-attack embeddings for similarity matching, and — the clever bit — canary tokens. Rebuff injects a secret canary into the prompt; if that canary ever appears in the model's output or in an outbound request, you have proof an injection succeeded and leaked, and you can add that attack to the vector store to harden future detection. That closes a learning loop our static harness cannot: our _INJECTION_PATTERNS never learn. Lakera (Gandalf's makers) productizes the same detector-as-a-service idea with a continuously updated model trained on a large corpus of real injection attempts. The committer's takeaway from both: injection detection is an adversarial, self-updating discipline; a fixed pattern list is a snapshot that decays. Our detect_injection is honest about being a heuristic — the lab's own README says an adversary paraphrases around it — precisely so nobody mistakes the snapshot for the discipline.

CaMeL and dual-LLM: the one control that is categorical

Everything above is a detector — probabilistic, defeatable, a layer. The research line our trust_boundary toggle mirrors is the only thing in the space that is categorical. Willison's dual-LLM: a quarantined LLM handles untrusted content and can return only structured values to a privileged LLM that never sees the untrusted prose. DeepMind's CaMeL (Debenedetti et al., 2025) formalizes it as a system where a trusted planner emits a program, untrusted content only ever binds to variables, and a capability/taint system tracks provenance so a tainted value cannot flow into a privileged sink — the paper's claim is defeating injection by construction, not by detection. The maintainer's honest caveat, which the authors themselves make: this is not free — it constrains expressiveness (the agent can only do what the planner's program shape allows), doubles model calls, and requires per-source schemas. Our lab's trust boundary is the degenerate one-line version — drop directives from non-TRUSTED items — which captures the principle (untrusted text is data, never instruction) while eliding the capability system, the taint propagation, and the planner/quarantine split that make CaMeL actually hold under a real workload.

What our miniature deliberately simplifies

A committer should be able to name every place the lab lies for clarity. The policy is injected, not an LLM — naive_following_policy is deterministic so the harness is testable and reproducible (test_deterministic); a real model's susceptibility is stochastic and that is why injection is unsolved. Trust is binary — real systems have gradations (a signed internal service vs. a user vs. the open web) and taint that propagates through tool chains; our two-valued enum flattens that. Detection is static regex — no classifier, no embeddings, no canary, no learning loop. HITL is a synchronous callable — a real gate is a durable async suspend (Phase 08). The output guard sees a clean rendered string — real exfil hides in base64, homoglyphs, split tokens, and DNS lookups, which no single regex catches. Provenance is handed to us pre-labeled — the hardest real problem is assigning trust correctly at every ingestion edge, especially for MCP tool results and multi-hop tool chains where a tainted value launders itself through an intermediate tool. Each simplification is chosen so exactly one idea stays in focus per layer; the value of studying the real systems is seeing what each simplification costs you when the adversary is not a fixed test fixture but a person iterating against your specific defenses.