AI Security, Privacy, and Governance — Overview

Phase 14 · Document 00 · Security, Privacy and Governance Up: Phase 14 Index · Next: 01 — Prompt Injection

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

LLM applications open a new class of vulnerabilities that traditional application security (AppSec) training doesn't cover — and most teams ship without addressing them. A prompt injection can override your entire application logic with attacker text (01). An LLM can leak PII at scale through its outputs, logs, or memorized training data (02). A secret in a prompt can be extracted by a single clever message (03). A multi-tenant RAG system can serve one customer's documents to another (04). And an agent with real tools can be turned into a confused deputy that acts on an attacker's behalf (Phase 10.05). On top of security sit privacy (whose data, retained how long, used for what) and governance (the policies, audit trails, and compliance that let you operate in the real world). This phase is the security/privacy/governance baseline every LLM engineer needs — and this overview is the threat model that frames the seven dedicated docs that follow.

2. Core Concept

Plain-English primer: the LLM is a new, untrusted execution surface

In a traditional app, you control the code. The dangerous inputs are things like SQL strings or file paths, and you have decades of patterns (parameterized queries, input validation, least privilege) to contain them. An LLM breaks a core assumption: it mixes instructions and data in the same channel — natural-language text — and it cannot reliably tell them apart. Everything the model sees (your system prompt, the user's message, a retrieved document, a tool result) arrives as one stream of tokens, and any of it can read like an instruction. That is the root of most LLM security problems: untrusted text can become an instruction.

So the mental shift is: treat the model's output as untrusted, treat everything in the model's context as potentially adversarial, and never give the model — or text it ingests — the authority to do something you wouldn't let an anonymous internet user do.

TRADITIONAL APP:  code (trusted) + data (untrusted, but separable) → you can sanitize the boundary
LLM APP:          instructions + data are the SAME channel (text) → the model can't reliably separate them
                  → untrusted text (user input, retrieved docs, tool results) can become an INSTRUCTION

The three pillars (and why they're one phase)

• Security — keeping attackers from subverting the system: prompt injection (01), secret exfiltration (03), cross-tenant leakage (04), unsafe agent actions (Phase 10.05).
• Privacy — handling personal/sensitive data correctly: what you collect, where it goes (your provider, 02), how long you keep it, and honoring deletion rights.
• Governance — the operational layer that proves and enforces the above: policy/guardrails (05), audit logs (06), and compliance (07).

They're one phase because in practice they're inseparable: a privacy control (PII scrubbing) is enforced by a guardrail, recorded in an audit log, and demonstrated for compliance.

The OWASP LLM Top 10 as the threat map

The industry-standard catalogue is the OWASP Top 10 for LLM Applications. The headline risks (and where this phase covers them): prompt injection (01), sensitive information disclosure (02/03), insecure output handling (treat output as untrusted, 05), excessive agency (agents, Phase 10.05), supply chain / model poisoning, and system prompt leakage. Use it as a checklist, not a curriculum.

The cardinal principles of this phase

1. The model proposes; the app executes (the trust boundary). The LLM never holds authority — it suggests; deterministic, permission-checked code decides and acts (Phase 10.05). This single principle defuses most agent attacks.
2. Defense in depth. No single control (especially not "prompt hardening") is sufficient — layer input filtering, privilege separation, output validation, sandboxing, and monitoring (01/05).
3. Prompting cannot fix a security problem. You cannot reliably instruct the model out of being manipulated; security must live in the architecture (permissions, isolation, gates), not in the prompt.
4. Least privilege + isolation. Give the model/agent the minimum capabilities and data access; isolate tenants and scope every key/tool (03/04).
5. Fail closed. When a guardrail, policy check, or residency rule is uncertain, deny — don't default to allowing (05).
6. Assume breach; log everything. You will be probed; audit logs (06) are how you detect, respond, and prove compliance (07).

How the phase is organized

00 OVERVIEW (this doc) — threat model + principles
 ├─ 01 Prompt Injection ........ the #1 risk: untrusted text → instruction; layered defense
 ├─ 02 Data Retention & Privacy . PII lifecycle, provider data use/ZDR, retention, deletion rights
 ├─ 03 Secrets & API Keys ....... never in prompt/weights; secret mgmt, scoping, rotation, exfil
 ├─ 04 Tenant Isolation ......... multi-tenant data separation, RAG ACLs, per-tenant keys/context
 ├─ 05 Policy & Guardrails ...... input/output guardrails, moderation, PEP/PDP, fail-closed
 ├─ 06 Audit Logs ............... what/why to log, immutability, PII-safe, traceability, IR
 └─ 07 Compliance Readiness ..... GDPR/SOC2/HIPAA/EU AI Act, DPA, residency, model governance

3. Mental Model

   ★ ROOT CAUSE: an LLM mixes INSTRUCTIONS + DATA in one text channel and can't reliably separate them
                 → untrusted text (user input / retrieved docs / tool results) can become an INSTRUCTION

   THREE PILLARS (inseparable): SECURITY (don't get subverted) · PRIVACY (handle personal data right) · GOVERNANCE (prove + enforce)
   THREAT MAP: OWASP LLM Top 10 — injection [01] · info disclosure [02/03] · insecure output handling [05] · excessive agency [10.05] · supply chain · system-prompt leak

   CARDINAL PRINCIPLES:
     1. model PROPOSES, app EXECUTES (trust boundary [10.05]) — the LLM never holds authority
     2. DEFENSE IN DEPTH — no single control suffices (esp. not prompt hardening)
     3. PROMPTING CAN'T FIX SECURITY — it lives in architecture (permissions/isolation/gates), not the prompt
     4. LEAST PRIVILEGE + ISOLATION — minimum capability/data; scope keys/tools, isolate tenants [03/04]
     5. FAIL CLOSED — uncertain → deny [05]
     6. ASSUME BREACH; LOG EVERYTHING — audit [06] → detect/respond/prove [07]

   DOCS: 01 injection · 02 privacy/retention · 03 secrets · 04 tenant isolation · 05 guardrails · 06 audit · 07 compliance

Mnemonic: the LLM can't tell instructions from data, so untrusted text can hijack it — the model only proposes, your permission-checked code executes; layer defenses, never rely on the prompt for security, least-privilege everything, fail closed, and log it all.

4. Hitchhiker's Guide

What to look for first: where does untrusted text enter the model's context (user input, retrieved docs, tool results, memory) and what authority does the model/agent have (tools, data access, keys)? Those two questions map your entire attack surface.

What to ignore at first: exotic jailbreak taxonomies and the latest "DAN" prompt. The defensible posture isn't enumerating every jailbreak — it's the architecture (trust boundary, least privilege, isolation, fail-closed) that holds regardless of the specific attack.

What misleads beginners:

• Thinking prompt hardening secures the system. It raises the bar slightly; it never closes the hole (01). Security is architectural.
• Trusting model output. Treat it as untrusted input to the next stage — validate, schema-check, permission-check (05).
• Putting secrets in the prompt. Anything in context can be extracted (03).
• Assuming the provider doesn't retain your data. Check the DPA / data-use terms / ZDR (02).
• Ignoring multi-tenancy. Shared indexes/caches/keys leak across customers without explicit isolation (04).

How experts reason: they threat-model the data flow (every untrusted-text entry point and every privileged action), enforce model-proposes/app-executes with least privilege and tenant isolation, layer input/output guardrails that fail closed, never rely on the prompt for security, keep PII-safe audit logs of every consequential action, and treat compliance as a continuous evidence-gathering process, not a one-time audit. They red-team their own system regularly (01).

What matters in production: the trust boundary holds under adversarial input; no PII/secret leakage (output, logs, cross-tenant); guardrails fail closed; complete, tamper-evident audit trail; and demonstrable compliance for your data/jurisdiction.

How to debug/verify: trace an attacker's text from entry → can it reach a privileged action or another tenant's data? If yes, the architecture is wrong (not the prompt). Red-team each entry point; verify logs capture every consequential action; verify a deletion request actually purges data everywhere (02).

Questions to ask: where does untrusted text enter context? what can the model/agent do? does the app (not the model) hold authority? are tenants/keys isolated and least-privileged? do guardrails fail closed? is everything logged PII-safely? can I prove compliance?

What silently gets you breached/fined: prompt-only "security," trusted model output, secrets in context, shared multi-tenant state, fail-open guardrails, missing/PII-leaking logs, and unexamined provider data-retention terms.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The root cause of LLM security risk: instructions and data share one text channel the model can't reliably separate — untrusted text can become an instruction (01).
• Three inseparable pillars: security, privacy, governance — a privacy control is enforced by a guardrail, recorded in an audit log, and proven for compliance.
• The OWASP LLM Top 10 is the standard threat map (injection, info disclosure, insecure output handling, excessive agency, supply chain, system-prompt leak).
• Cardinal principle: the model proposes; the app executes — the LLM never holds authority (Phase 10.05); this defuses most agent attacks.
• Prompting cannot fix a security problem — security lives in architecture (permissions, isolation, gates), not the prompt.
• Defense in depth + least privilege + isolation + fail-closed are the load-bearing principles across all seven docs.
• Treat model output as untrusted input to the next stage — validate/schema-check/permission-check it (05).
• Assume breach: log everything (PII-safely) and red-team regularly — audit logs (06) are how you detect, respond, and prove compliance (07).

9. Observations from Real Systems

• Prompt injection remains unsolved — there is no general fix; production systems contain it with architecture (trust boundary, isolation, gates), not prompts (01).
• The biggest real incidents are data leaks — system-prompt/secret extraction, PII in logs, and cross-tenant leakage — more than dramatic "jailbreaks" (02/03/04).
• Enterprise buyers gate on governance — DPAs, data residency, SOC 2, audit trails decide deals as much as model quality (07); the gateway's policy engine is often where this lives (Phase 8.09).
• Agents raise the stakes — "excessive agency" turns a text vulnerability into real-world actions; the model-proposes/app-executes boundary is the standard containment (Phase 10.05).
• OWASP/NIST/MITRE ATLAS/Google SAIF have converged into a shared vocabulary — teams audit against these rather than inventing their own.

10. Common Misconceptions

11. Engineering Decision Framework

SECURE / PRIVATE / GOVERNED LLM APP:
 1. THREAT-MODEL the data flow: list every UNTRUSTED-TEXT entry (user, RAG docs, tool results, memory)
    and every PRIVILEGED action/data access the model/agent can reach.
 2. TRUST BOUNDARY: app (deterministic, permission-checked) executes; model only PROPOSES [10.05]. LEAST PRIVILEGE everything.
 3. INJECTION [01]: layered defense (input filtering + privilege separation + output validation + sandbox) — never prompt-only.
 4. PRIVACY [02] + SECRETS [03]: minimize/scrub PII, check provider retention/ZDR; no secrets in prompt/weights, scope+rotate keys.
 5. ISOLATION [04]: per-tenant data/keys/context; RAG ACLs at retrieval; no shared mutable state across tenants.
 6. GUARDRAILS [05]: input + output policy checks that FAIL CLOSED; moderation; structured-output validation.
 7. AUDIT [06]: log every consequential action PII-safely, immutably, traceably; red-team regularly.
 8. COMPLIANCE [07]: map to GDPR/SOC2/HIPAA/EU AI Act for your data+jurisdiction; gather evidence continuously.

12. Hands-On Lab

Goal

Produce a threat model + security baseline for a sample LLM app, mapping every untrusted-text entry point and privileged action to a control across the seven topics.

Prerequisites

• A sample LLM app (e.g., a RAG support assistant with one tool) and the OWASP LLM Top 10.

Steps

1. Data-flow map: diagram the app — every place untrusted text enters the model's context (user input, retrieved docs, tool results, memory) and every privileged action / data access the model can reach.
2. Map to OWASP LLM Top 10: for each risk, note whether/how your app is exposed.
3. Apply the trust boundary: identify where the model proposes vs the app executes; flag any place the model holds authority (Phase 10.05).
4. Assign controls: for each entry point/action, pick the control and the doc that covers it (injection [01], privacy [02], secrets [03], isolation [04], guardrails [05], audit [06], compliance [07]).
5. Identify the gaps: which risks are only mitigated by prompting (i.e., not really mitigated)?
6. Prioritize: rank fixes by blast radius (cross-tenant leak / destructive action > minor info disclosure).

Expected output

A one-page threat model (data-flow + entry points + privileged actions), an OWASP mapping, a control-per-risk table pointing into docs 01–07, and a prioritized gap list — your security baseline for the app.

Debugging tips

• If a control is "the system prompt tells it not to" → that's a gap, not a control.
• If untrusted text can reach a privileged action or another tenant → architecture fix, not prompt fix.

Extension task

Run a quick red-team against one entry point (e.g., indirect injection via a retrieved doc) and confirm the architectural control holds (01).

Production extension

Turn the control table into a pre-deployment security checklist (the artifact in §15) and wire the highest-priority controls (trust boundary, isolation, fail-closed guardrails, audit) into the app.

What to measure

Coverage (entry points/actions with a real control), prompt-only "controls" (gaps), prioritized risk reduction.

Deliverables

• A data-flow threat model + OWASP LLM Top 10 mapping.
• A control-per-risk table referencing docs 01–07.
• A prioritized gap list and a starter security checklist.

13. Verification Questions

Basic

1. Why can't an LLM reliably separate instructions from data, and why does that matter?
2. What are the three pillars of this phase and why are they inseparable?
3. State the "model proposes; app executes" principle.

Applied 4. Why is prompt hardening not a security control? 5. What does "fail closed" mean and when do you apply it?

Debugging 6. You find an attack where a retrieved document makes the agent send an email. Where's the real fix — prompt or architecture? 7. How would you check whether your provider retains your prompts?

System design 8. Threat-model a multi-tenant RAG agent: list the untrusted-text entry points and the controls per OWASP risk.

Startup / product 9. Why does enterprise governance (DPA, residency, SOC 2, audit) often gate deals as much as model quality?

14. Takeaways

1. The root cause: LLMs mix instructions and data in one text channel — untrusted text can become an instruction (01).
2. Security, privacy, and governance are one inseparable discipline — controls, logs, and compliance reinforce each other.
3. The model proposes; the app executes — the LLM never holds authority (Phase 10.05); this defuses most attacks.
4. Prompting can't fix security — it lives in architecture: defense in depth, least privilege, isolation, fail-closed.
5. Assume breach: log everything PII-safely, red-team regularly, and prove compliance (06/07).

15. Artifact Checklist

• A data-flow threat model (untrusted-text entry points + privileged actions).
• An OWASP LLM Top 10 mapping for the app.
• A control-per-risk table referencing docs 01–07.
• A prioritized gap list (prompt-only "controls" flagged).
• A starter pre-deployment security checklist.

Up: Phase 14 Index · Next: 01 — Prompt Injection