Phase 11 - AI, LLM, Agentic Security, and Security Automation
Duration: 8 weeks (Months 22-23)
Objective: secure LLM systems as untrusted probabilistic components and build automation that
assists security decisions without silently acquiring authority.
Deep guides: WARMUP.md models LLMs as untrusted proposers; HITCHHIKERS-GUIDE.md provides injection fixtures, tool-gateway policy, executor isolation, audit, and evaluation guidance.
Required Theory
- Tokens, context, embeddings, RAG, reranking, tool calling, planners, memory, multi-agent patterns, evals, and model/runtime supply chain.
- Direct and indirect prompt injection, untrusted retrieval, data exfiltration, cross-tenant memory, insecure output handling, over-permissioned tools, confused deputy, excessive agency, denial of wallet/service, model/plugin/connector provenance, and privacy.
- Secure agent architecture: typed tool schemas, model-independent authorization, capability tokens, sandboxing, egress control, data labels, approval gates, transaction boundaries, idempotency, budgets, audit, rollback, and human ownership.
- AI red teaming in owned systems, capability vs safety evaluation, attack datasets, false refusal, hallucination, groundedness, reproducibility, and regression.
- Security assistants for code review, triage, remediation, cloud posture, logs, threat models, compliance, and disclosure drafting.
8-Week Lab Sequence
| Week | Labs | Required proof |
|---|---|---|
| 1 | security-doc RAG | source ACLs, citations, tenant filters, deletion, poisoned-document fixture |
| 2 | code-review assistant | structured findings, static-tool evidence, confidence, no automatic merge |
| 3 | vulnerability triage/remediation | deterministic schema, severity rationale, human correction dataset |
| 4 | cloud/log assistants | read-only scoped tools, query budgets, privacy redaction, evidence links |
| 5 | prompt-injection test app | direct/indirect injection and data-boundary eval corpus |
| 6 | secure tool gateway | authorization outside model, allowlists, capability scope, egress and sandbox |
| 7 | approvals and audit | two-person/high-risk gates, transaction preview, tamper-evident event chain |
| 8 | evaluation harness | task success, groundedness, unsafe action, leakage, refusal, cost, latency regression |
Runnable lab portfolio
- Least-Privilege Agent Tool Gateway
- Tenant-Safe RAG and Poisoned Document Evaluator
- Tamper-Evident Agent Approval Ledger
These keep proposal separate from authority, enforce retrieval ACL/provenance, detect injection content, and preserve proposal-to-policy-to-approval-to-execution audit integrity.
All test data is synthetic. Injection labs target only the owned toy application. Do not build agents that autonomously exploit systems, steal credentials, establish access, or take destructive actions.
Capstone - AI Security Engineering Platform
Build a platform that ingests structured outputs from code and cloud scanners, retrieves authorized security guidance, drafts triage/remediation, and proposes a ticket or patch for human approval.
Required controls:
- secure RAG with document provenance, ACL filtering, citation, and poisoning tests;
- typed tools behind a policy enforcement point independent of the model;
- read-only default, least privilege, per-tool and per-user scopes;
- isolated execution and default-deny network egress;
- human approval for writes, privilege changes, disclosure, and external communication;
- RBAC, tenant isolation, rate/cost limits, redaction, retention, and audit;
- rollback/idempotency for approved actions;
- threat model and abuse-case library;
- evaluation suite with injection, exfiltration, hallucination, unsafe remediation, and availability cases.
Measures
- 0 unauthorized tool actions across the maintained adversarial suite.
- 100% privileged actions require a valid policy decision and approval artifact.
- Every material claim in a drafted finding links to scanner evidence or approved source.
- Cross-tenant canaries never appear outside their tenant.
- Eval dashboard tracks task quality and safety separately, including confidence intervals and known blind spots.
Tools and Sources
Use a local or hosted model according to data policy, a typed web framework, isolated containers, OPA/Cedar-style policy concepts, vector/search store with ACL metadata, scanner fixtures, and an evaluation harness such as promptfoo/garak-style tooling plus custom tests.
Read OWASP LLM/Agentic guidance, MITRE ATLAS, NIST AI RMF, Google SAIF, provider system cards, secure tool-use documentation, and original papers on indirect prompt injection, RAG poisoning, agent evaluation, and model supply-chain risk.
Evaluation Checklist
- Explain why system prompts and output filters are not authorization controls.
- Contain malicious instructions in retrieved documents.
- Separate model proposal, policy decision, user approval, execution, and audit.
- Evaluate false refusals alongside unsafe successes.
- Trace data provenance, tenant, retention, and deletion through the full system.
- State which security judgments require a qualified human.
Common Mistakes and Interview Questions
Mistakes: giving the model broad credentials, mixing retrieved data with instructions, relying on prompt secrecy, accepting model-generated shell/code directly, no transaction preview, evals made only of known prompts, and calling hallucination detection deterministic.
Questions: secure an email-reading agent; prevent indirect injection from a ticket; design a least-privilege code-fix agent; isolate a model tool runner; investigate cross-tenant RAG leakage; choose approval gates; measure an AI security assistant without trusting an LLM judge blindly.
Portfolio and Stretch
Publish the toy platform, synthetic eval corpus, architecture, policy tests, and red-team report. Stretch: provenance-aware retrieval, dual-control workflows, signed tool manifests, model gateway, multi-model disagreement experiments, or a benchmark for secure remediation quality.