Lab 01 — Least-Privilege Agent Tool Gateway

Difficulty: 4/5 | Runs locally: yes

Pairs with the Phase 11 WARMUP Chapters 1, 4, 8, 9 (untrusted proposer, tool calling, why system prompts aren't authorization, the secure agent architecture) and the HITCHHIKERS-GUIDE. This is the flagship lab of the phase.

Why This Lab Exists (Purpose & Goal)

An LLM agent is an untrusted, probabilistic component that processes instructions and data in the same token stream — which means prompt injection is not reliably preventable, and you must assume the model can be hijacked. The one architectural principle that makes agents safe anyway is: the model proposes; deterministic code outside the model decides and authorizes. The goal of this lab is to build that decision point — a tool gateway that validates every proposed action against policy the model cannot influence.

The Concept, In the Weeds

The gateway is a policy decision point for agent actions (the same pattern as cloud admission, the release gate, object-level authZ — now for tool calls). It validates each typed tool request against:

user/tenant scope — cross-tenant resource IDs are denied (tenant isolation, Phase 07);
per-tool resource policy — only declared tools/actions; hidden/unknown tool names fail closed;
risk and approval — high-risk writes require a human approval artifact (not the model's say-so);
idempotency and replay — replayed requests are rejected;
budget — excessive batch size / cost is denied (denial of wallet);
egress — sending data to an unapproved destination is denied (the exfiltration leg).

The load-bearing idea — and the most common real-world mistake — is that a prompt injection can change the proposal but cannot change the gateway's evaluate logic. Authorization lives in code, not in the system prompt; untrusted retrieved text is data and can never grant authority. A system prompt is a suggestion to a probabilistic function, not an access control; if your authorization can be altered by text in the context window, you have no authorization.

Why This Matters for Protecting the Company

Agents are being given real tools — they file tickets, run queries, send messages, modify infrastructure — and an agent with broad credentials and an injectable context is a new, powerful attack surface. The difference between "a contained AI feature" and "an injected agent that exfiltrated customer data" is whether a deterministic gateway bounded what the agent could do. As companies deploy AI agents into security operations, customer support, and engineering workflows, this gateway pattern — least privilege, model-independent authorization, human approval for consequential actions, audited — is the control that makes them safe to deploy at all. The bar: 0 unauthorized tool actions across the adversarial suite.

Build It

Build the gateway: validate typed tool requests against user/tenant scope, per-tool resource policy, risk, approval, idempotency, and budget. Untrusted retrieved text is data and cannot grant authority.

LAB_MODULE=solution pytest -q

Attack Fixtures

Cross-tenant resource IDs; hidden tool names; excessive batch size; high-risk writes without approval; replayed requests; attempts to send data to an unapproved destination.

Secure Implementation Patterns

The anti-pattern. Executing whatever the model proposes, with the agent's broad credentials:

result = eval(model_output.tool_call)        # injection -> arbitrary action with full agent authority

The secure pattern — a deterministic gateway the model can't influence (mirrors solution.py):

TOOLS = {
    "scanner-read": {"actions": {"read"},   "risk": "low",    "max_items": 100},
    "ticket-write": {"actions": {"create"}, "risk": "medium", "max_items": 20},
    "patch-open":   {"actions": {"create"}, "risk": "high",   "max_items": 5},
}
ALLOWED_DESTINATIONS = {None, "tickets.internal", "git.internal"}

def authorize_and_reserve(req: ToolRequest, state: GatewayState) -> tuple[bool, str]:
    if not req.request_id or req.request_id in state.completed_request_ids:
        return False, "replay-or-missing-request-id"             # idempotency / replay
    policy = TOOLS.get(req.tool)
    if policy is None:                       return False, "unknown-tool"          # unknown tool fails closed
    if req.action not in policy["actions"]:  return False, "action-not-allowed"
    if req.tenant != req.resource_tenant:    return False, "tenant-mismatch"       # tenant isolation
    if req.destination not in ALLOWED_DESTINATIONS: return False, "destination-not-allowed"  # egress allowlist
    if not (0 < req.item_count <= policy["max_items"]): return False, "item-limit" # batch bound
    if policy["risk"] in {"medium", "high"} and not req.approved:
        return False, "human-approval-required"                  # consequential actions need a human
    if req.item_count > state.remaining_budget: return False, "budget-exhausted"   # denial of wallet
    state.remaining_budget -= req.item_count
    state.completed_request_ids.add(req.request_id)
    return True, "reserved"

The decisive property: a prompt injection can change req, but not authorize_and_reserve. Authorization is deterministic code over a typed request — not the system prompt, not the model's "intent." Untrusted retrieved text is data; it can never reach this function as authority.

Production practices to carry forward:

Typed tool schemas — the model emits structured args you validate like any API input; never an eval-able string.
Least privilege per tool/user/tenant, read-only by default; the executor gets a one-use, scoped capability, not the agent's broad credential.
Human approval (with a transaction preview) for writes, privilege changes, disclosure, external comms, and spend — dual control for the highest risk.
Egress allowlist + sandboxed execution (Phase 06) breaks the exfiltration leg of the lethal trifecta (see the trifecta guard).
Audit every decision to a tamper-evident ledger (approval ledger).

Validation — What You Should Be Able to Do Now

Explain why the model is an untrusted proposer and what "proposal vs authority" means architecturally.
Explain why system prompts and output filters are not authorization controls.
Build a least-privilege tool gateway with model-independent policy, tenant isolation, approval gating, idempotency, budget, and egress control.
Treat retrieved/untrusted text as data that can never grant authority.

The Broader Perspective

This lab reveals that AI agent security is mostly systems security, not "AI" security — the hard, valuable work is the deterministic policy engine, the sandboxing (Phase 06), the tenant isolation (Phase 07), and the audited approval flow around the model, not anything inside it. The same enforcement-point pattern you've now built across the entire curriculum (engagement guard, object-level authZ, release gate, sandbox compiler, K8s admission) applies one more time, to the newest and most dynamic actor. The reframe — bound what a compromised/hijacked component can do, rather than trying to make it un-hijackable — is the defining mindset for securing AI systems, and it's why an engineer with strong systems-security fundamentals is exactly who should be building agent guardrails.

Interview Angle

"Where does authorization live in an agent system?" — In deterministic code outside the model. The model proposes; the policy engine decides on a typed request against policy. If your authZ can be changed by text in the context window, you have no authZ.
"Design a least-privilege code-fix agent." — Read-only by default; proposes a structured diff with evidence; never auto-merges; typed scoped tools with no prod creds; applying the fix is a separate human-approved, idempotent/rollback-able action behind the policy engine.

Extension (Stretch)

Add a sandboxed executor (Phase 06) with default-deny egress, and integrate the approval ledger for a tamper-evident proposal→policy→approval→ execution chain. See the trifecta guard for the lethal-trifecta exfiltration defense.

References

Phase 11 WARMUP Chapters 1, 4, 8, 9; OWASP Top 10 for LLM Applications; OWASP Agentic Security; MITRE ATLAS; NIST AI RMF; Google SAIF.

Security Engineer