Enterprise Policy Engine

Phase 8 · Document 09 · LLM Gateways Prev: 08 — Admin Dashboard · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The policy engine is what makes a gateway enterprise-ready and the capstone of Phase 8: a centralized enforcement point for who can do what (authn/RBAC), what may be sent (PII redaction, content guardrails, injection defense), where data may go (residency/sovereignty), and proof of all of it (audit logs, tenant isolation). The gateway is the ideal place for this because all LLM traffic flows through it — enforce policy once, centrally, instead of in every app. This is precisely why enterprises mandate a gateway: it's the control point for SOC2/HIPAA/GDPR compliance, the thing that lets a bank or hospital use LLMs at all. This doc is the gateway-component view; the deep security treatment is Phase 14.

2. Core Concept

Plain-English primer: the gateway as the policy enforcement point

In access-control terms, the gateway is a Policy Enforcement Point (PEP): every request passes through it, so it's where you decide and enforce what's allowed — ideally evaluated by a Policy Decision Point (PDP) (rules engine) the gateway consults. Five enforcement areas, applied across the request lifecycle:

1. Authentication + Authorization (RBAC). Authn = who are you (validate the API key/JWT → identity, team, tenant). Authz/RBAC = what may you do (which models, routes, budgets, and admin actions this principal is allowed). The virtual keys carry these grants; the policy engine enforces them.

2. Input policy (pre-call). Inspect the prompt before forwarding: PII detection/redaction (strip or mask SSNs, emails, secrets — Phase 14.02), content moderation (block disallowed categories), and prompt-injection screening for agent/tool contexts (Phase 14.01, Phase 10). Decide: allow / redact / block.

3. Routing policy (data residency / sovereignty). Tag-driven, fail-closed routing so sensitive/regulated data only reaches approved backends — self-hosted/in-region/BAA-covered providers (05 data-policy routing). This is the compliance backbone: EU data stays in EU, PHI only to HIPAA-eligible endpoints.

4. Output policy (post-call). Inspect the response: moderation, PII leak checks, secret scanning, and citation/grounding requirements for some workflows (Phase 9.08). Decide: allow / redact / block.

5. Audit + tenant isolation (cross-cutting). Audit log every decision and action (who, what, model, allow/deny, redactions) — immutable, exportable to a SIEM — for compliance and incident forensics. Tenant isolation ensures one tenant's keys, data, usage, and (cached) context never bleed into another's (Phase 14.04).

Where each runs in the lifecycle

request → AUTHN (key/JWT → identity) → AUTHZ/RBAC (allowed models/routes/actions?)
        → INPUT POLICY (PII redact · moderation · injection screen)  → allow/redact/block
        → ROUTING POLICY (residency: route only to approved backend, FAIL CLOSED) [05]
        → [ provider call ]
        → OUTPUT POLICY (moderation · PII/secret leak · grounding)    → allow/redact/block
        → AUDIT LOG (decision + action, immutable) ; TENANT ISOLATION throughout

Fail closed, and the policy/decision split

• Fail closed: if policy can't be satisfied (no compliant backend, PII can't be safely handled, moderation service down for a high-risk path), reject — never silently send sensitive data onward. (Contrast "fail open," which is a breach.)
• PEP/PDP split: keep enforcement (the gateway) separate from decision (a rules engine like OPA/Rego or a config-driven ruleset), so policy is centrally managed, versioned, and testable rather than hardcoded.

Compliance is the product reason

These controls map directly to SOC2, HIPAA, GDPR/data-residency, and enterprise procurement requirements (Phase 14.07, Phase 15.08). An enterprise can't adopt LLMs without: data staying in approved boundaries, PII handled, access controlled, and everything audited. The gateway's policy engine is how a startup becomes sellable to enterprises — often the difference between a demo and a contract.

Reuse vs build

LiteLLM and productized gateways (Portkey, Kong AI Gateway) ship guardrails hooks, key/RBAC, audit, and PII/moderation integrations (02); cloud providers offer moderation/PII APIs; OPA handles authz rules. Compose these rather than building moderation/PII from scratch — build the policy orchestration + residency routing + audit that's specific to your compliance posture.

3. Mental Model

   GATEWAY = the Policy Enforcement Point (all traffic flows through it)
   AUTHN (who) → RBAC (what: models/routes/actions) →
   INPUT POLICY (PII redact · moderation · injection) → allow/redact/BLOCK →
   ROUTING POLICY (residency: approved backend only, FAIL CLOSED) [05] →
   [provider] → OUTPUT POLICY (moderation · PII/secret leak · grounding) →
   AUDIT LOG (immutable, → SIEM) ; TENANT ISOLATION end-to-end
        decision via a PDP (OPA/Rego or config) ≠ enforcement (the gateway)
   FAIL CLOSED on unsatisfiable policy.  Maps to SOC2/HIPAA/GDPR = enterprise-sellable [14,15]

Mnemonic: the gateway is the one place to enforce who/what/where/proof — authn+RBAC, input/output guardrails, fail-closed residency routing, and audit+isolation — split decision (PDP) from enforcement (PEP), and fail closed.

4. Hitchhiker's Guide

What to look for first: authn + RBAC (identity → allowed models/routes) and fail-closed residency routing (sensitive data → approved backends only). Those are the non-negotiable enterprise gates.

What to ignore at first: building moderation/PII models yourself. Use provider/cloud moderation + PII APIs and OPA for authz; orchestrate them.

What misleads beginners:

• Enforcing policy in apps, not the gateway. Scatters and weakens it; the gateway is the single PEP all traffic passes (00).
• Fail open. Letting requests through when policy can't be evaluated/satisfied is a breach — fail closed (05).
• Residency as a soft preference. Must be a hard, fail-closed filter or regulated data leaks (05).
• Logging raw prompts/PII in the audit log. The audit trail must record decisions without re-leaking the sensitive content it's protecting (Phase 14.06).
• Weak tenant isolation. Shared caches/keys/usage across tenants is a cross-tenant leak (Phase 14.04) — note: even prefix caching must not share across tenants for sensitive prompts (Phase 7.05).

How experts reason: they make the gateway the single PEP, split decision (PDP: OPA/config) from enforcement, apply input/output guardrails + fail-closed residency routing, audit every decision (without leaking PII), enforce tenant isolation end-to-end (including caches), and map controls to the specific compliance frameworks they must satisfy. They reuse moderation/PII/authz tooling and build the orchestration + residency + audit.

What matters in production: zero policy bypasses, fail-closed correctness, residency enforcement, complete + non-leaking audit, tenant isolation (incl. cache), and the latency/cost the guardrails add (moderation/PII calls aren't free).

How to debug/verify: red-team it — send PII, an injection, a sensitive-tagged request, and a non-permitted model request; confirm redaction/block/residency/deny and an audit entry for each; verify cross-tenant isolation; confirm no raw PII in logs.

Questions to ask (procurement/vendor): authn/RBAC model? input/output guardrails + PII? data-residency routing? audit log + SIEM export? tenant isolation (incl. caching)? which compliance certs? data retention? (Phase 14.07)

What silently gets expensive/unreliable: fail-open gaps, residency leaks under cost pressure, PII in audit logs, cross-tenant cache bleed, and guardrail latency/cost not budgeted.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The gateway is the single Policy Enforcement Point — enforce who/what/where/proof once, centrally.
• Five areas: authn+RBAC, input policy, residency routing, output policy, audit+isolation — across the request lifecycle.
• Fail closed on unsatisfiable policy — never silently send sensitive data onward (05).
• Data-residency routing is a hard, fail-closed filter — regulated data only to approved backends (05, Phase 14).
• Audit logs must record decisions without re-leaking PII, and export to a SIEM.
• Tenant isolation includes caches — prefix/response caches must not share across tenants for sensitive prompts (Phase 7.05).
• Split decision (PDP: OPA/config) from enforcement (gateway PEP) for manageable, testable policy.
• These controls map to SOC2/HIPAA/GDPR — the reason enterprises require a gateway (Phase 14.07, Phase 15).

9. Observations from Real Systems

• Enterprise/internal gateways exist largely for this: centralized keys, RBAC, residency routing, audit, and DLP across all teams (00).
• LiteLLM/Portkey/Kong AI Gateway ship guardrails + RBAC + audit hooks and integrate PII/moderation providers — compose these (02).
• Data-residency routing is a hard requirement for EU/regulated deployments — sensitive data never leaves the approved boundary (05).
• Compliance (SOC2/HIPAA/GDPR) is frequently the gating factor in enterprise LLM adoption — the policy engine is what unlocks the sale (Phase 15.08).
• OWASP LLM Top 10 (prompt injection, sensitive-info disclosure, etc.) is the threat catalog these controls defend against (Phase 14.01).

10. Common Misconceptions

11. Engineering Decision Framework

ENFORCE POLICY AT THE GATEWAY (single PEP; decide via PDP: OPA/config):
 1. AUTHN: validate key/JWT → identity/tenant/team.
 2. RBAC: allowed models/routes/budgets/admin actions (from virtual keys).        [06]
 3. INPUT POLICY: PII redact · moderation · injection screen → allow/redact/BLOCK. [14.01/14.02]
 4. RESIDENCY ROUTING: sensitive-tagged → approved backend ONLY, FAIL CLOSED.      [05]
 5. OUTPUT POLICY: moderation · PII/secret leak · grounding → allow/redact/BLOCK.  [9.08]
 6. AUDIT: immutable decision/action log (no raw PII) → SIEM.                      [14.06]
 7. TENANT ISOLATION: keys/data/usage/CACHES separated.                            [7.05/14.04]
 8. MAP to required frameworks (SOC2/HIPAA/GDPR); reuse moderation/PII/OPA tooling.[14.07,15.08]

12. Hands-On Lab

Goal

Add a policy engine to your gateway: RBAC, PII redaction, fail-closed residency routing, output checks, and an audit log — then red-team it.

Prerequisites

• Your gateway with routing + metering/keys; a PII/moderation API or a regex/lib for PII; (optional) OPA.

Steps

1. RBAC: give virtual keys an allowed_models + role; enforce that a key can only call its allowed models and only admins hit /admin/* (06, 08).
2. Input PII policy: detect PII (emails/SSNs/keys) in the prompt; redact (or block for a strict policy) before forwarding; log the action (Phase 14.02).
3. Residency routing (fail closed): tag a request data_class=regulated; ensure it routes only to a local/in-region backend and rejects if none is available — never to a public provider (05).
4. Output policy: scan the response for PII/secret leakage; redact/flag; log.
5. Audit log: record every decision (principal, model, allow/deny, redactions) without storing the raw PII; export a sample; confirm completeness (Phase 14.06).
6. Red-team: send (a) PII, (b) a prompt-injection attempt, (c) a regulated-tagged request, (d) a non-permitted model call; confirm redact/screen/residency/deny and an audit entry for each. Verify tenant isolation (tenant A can't see tenant B's keys/usage/cache).

Expected output

A gateway that enforces RBAC, redacts/blocks PII, fail-closed-routes regulated data, checks outputs, and produces a non-leaking audit log — with a red-team report showing each control fired.

Debugging tips

• Regulated request reached a public provider → residency was a soft preference or applied after selection (05).
• Audit log contains raw PII → you logged the prompt instead of the decision/metadata.

Extension task

Externalize decisions to OPA/Rego (PDP) so policies are versioned/testable separately from the gateway (PEP); add a policy unit-test suite.

Production extension

Integrate a real moderation/PII provider, ship audit logs to a SIEM, enforce tenant-isolated caching (Phase 7.05), and map controls to your target framework (SOC2/HIPAA/GDPR) for a compliance readiness doc (Phase 14.07, Phase 15.08).

What to measure

Policy-bypass rate (target 0), residency enforcement, redaction accuracy, audit completeness (no PII), tenant-isolation, guardrail latency/cost overhead.

Deliverables

• A gateway policy engine: RBAC + input/output guardrails + fail-closed residency + audit.
• A red-team report showing each control fired (PII/injection/residency/RBAC).
• A compliance-control map (controls → SOC2/HIPAA/GDPR requirements).

13. Verification Questions

Basic

1. What five areas does the gateway policy engine enforce?
2. What does "fail closed" mean and why is it required?
3. What's the difference between a PEP and a PDP?

Applied 4. Design fail-closed residency routing for EU-only data. 5. How do you audit policy decisions without re-leaking the PII you're protecting?

Debugging 6. Regulated data reached a public provider. Two likely policy-engine failures. 7. Tenant A sees a cached answer derived from tenant B's prompt. What isolation broke?

System design 8. Design the gateway policy engine (authn/RBAC, input/output guardrails, residency, audit, isolation) for a HIPAA workload.

Startup / product 9. Why is the policy engine often the deciding factor in enterprise sales, and which controls unlock SOC2/HIPAA/GDPR?

14. Takeaways

1. The gateway is the single Policy Enforcement Point — enforce who/what/where/proof centrally.
2. Five areas: authn+RBAC · input guardrails (PII/moderation/injection) · fail-closed residency routing · output guardrails · audit + tenant isolation.
3. Fail closed; residency is a hard filter; audit without leaking PII; isolate tenants incl. caches (Phase 7.05).
4. Split decision (PDP/OPA) from enforcement (gateway PEP); reuse moderation/PII/authz tooling.
5. These controls map to SOC2/HIPAA/GDPR — the policy engine is what makes the gateway (and your product) enterprise-sellable (Phase 14, Phase 15).

15. Artifact Checklist

• A gateway policy engine: RBAC + input/output guardrails + fail-closed residency + audit.
• A red-team report (PII/injection/residency/RBAC each fired).
• An audit log that records decisions without raw PII.
• A tenant-isolation verification (incl. caches).
• A compliance-control map (controls → SOC2/HIPAA/GDPR).

Up: Phase 8 Index · Next: Phase 9 — RAG