« Phase 10 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 10 — Hitchhiker's Guide
30-second mental model
LLMs mix instructions and data in one channel, so any text the agent reads can become a command — prompt injection, unsolved, no general fix. You can't prompt your way out (the attacker writes prompt too). Contain it with layered architecture: trust boundary (instructions only from the trusted channel) + least-privilege tool allow-list + output exfil scanning + human-in-the-loop. Defense in depth — each layer alone is imperfect.
The facts to tattoo on your arm
| Fact | Why |
|---|---|
| instructions + data = one channel | the root cause; there's no reliable separator |
| prompting is a mitigation, not a control | the attack is in the prompt channel |
| indirect > direct injection | fetched/RAG/tool/memory content, victim didn't write it |
| markdown-image beacon |  exfils with no click |
| trust boundary = the fix for indirect | untrusted text is data, never instruction |
| least privilege | a hijacked agent can only use tools it was granted |
| output guard > input guard | never trust the model's output; last line before a leak |
| OWASP LLM01 (injection), LLM08 (excessive agency) | the two agent-defining risks |
Framework one-liners
- Dual-LLM / CaMeL — quarantine untrusted content in a non-privileged LLM that can only return data, never instructions.
- Llama Guard / NeMo Guardrails / Lakera / Rebuff — input/output guard + injection detectors (a layer, not the control).
- Spotlighting / delimiting — mark untrusted spans as data (helps, not sufficient).
- OWASP LLM Top 10 / MITRE ATLAS — the threat vocabularies to cite.
War stories
- The summarizer that emailed the customer list. Indirect injection in a fetched page + an email tool it never should have had (excessive agency).
- The invisible image that stole a token. A RAG doc told the agent to render a markdown image to an attacker host; the token rode out in the URL.
- The memory that fired a week later. A poisoned memory entry triggered on an unrelated turn — the hardest vector to catch.
Vocabulary
Prompt injection (direct/indirect/memory) · exfiltration (image beacon / URL query) · trust boundary · least privilege / excessive agency · guardrails (input/output) · dual-LLM / quarantine / CaMeL · human-in-the-loop · defense in depth · OWASP LLM Top 10 · threat modeling.
Beginner mistakes
- "Fixing" injection with a system-prompt rule.
- Filtering inputs and calling it done (adversary paraphrases around it).
- Giving a broadly-exposed agent a high-impact tool with no gate (excessive agency).
- Trusting model output — not scanning for the image/URL exfil beacon.
- Forgetting indirect + memory vectors; only defending the user channel.
- Treating security as a later feature instead of the trust-boundary architecture.