Design 04 - AI Security Agent Platform
Prompt
Design an assistant that reads scanner findings and internal documentation, proposes remediation, creates tickets, and can open code patches after human approval.
Architecture
- Ingestion validates scanner schemas and records provenance; documents retain tenant, ACL, source, version, and deletion metadata.
- Retrieval enforces identity/tenant ACLs before context reaches the model.
- Model produces a typed proposal, never direct execution.
- Policy engine independently authorizes tool, resource, action, tenant, risk, and budget.
- Approval UI shows evidence, diff/action preview, affected resources, rollback, and model uncertainty.
- Executor runs with a short-lived capability in an isolated environment with constrained egress.
- Ticket/patch operation is idempotent and auditable; merge/deploy remain separate controls.
Abuse Cases
Indirect prompt injection in code/docs/tickets, poisoned guidance, cross-tenant retrieval, model hallucination, excessive tool scope, approval fatigue, malicious patch, secret exposure, denial of wallet, compromised model/plugin, and audit tampering.
Validation
- Injection and poisoned-document corpus.
- Tenant canaries and deletion/reindex tests.
- Unauthorized tool/action/resource combinations.
- Patch semantic/security tests and static analysis.
- False-refusal and unsafe-success metrics.
- Human factors test for approval clarity and fatigue.
- Kill switch, credential rotation, rollback, and compromised-model tabletop.
Security Invariants
Untrusted content cannot grant authority. Model output cannot bypass policy. Approval cannot exceed the approver's own authority. Execution credentials are scoped and short-lived. Every action maps to source evidence, policy decision, approval, executor, and result.
Residual Risk
Novel injection, colluding content, reviewer automation bias, model/provider data handling, incomplete eval coverage, and plausible-but-wrong remediation remain. Maintain conservative defaults and qualified human ownership.