Policy and Guardrails

Phase 14 · Document 05 · Security, Privacy and Governance Prev: 04 — Tenant Isolation · Up: Phase 14 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The model itself is non-deterministic and only probabilistically safe — RLHF/safety training (Phase 13.03) makes it usually refuse harmful requests, but "usually" is not a control you can put in front of a regulator, a customer, or a court. Guardrails are the deterministic safety layer wrapped around the model that turns "the model probably won't" into "the system will not." They check inputs before the model sees them and outputs before anyone acts on them, and they enforce your policy (what's allowed, for whom, in what context). This is also where you operationalize everything else in the phase: PII redaction (02), injection output-scanning (01), and the access decisions that keep tenants and secrets safe (03/04). Guardrails are how governance (06/07) becomes running code.

2. Core Concept

Plain-English primer: a deterministic safety wrapper around a probabilistic model

You cannot make an LLM guaranteed safe by training or prompting alone — it's a probability machine. So you wrap it with deterministic checks you do control:

            ┌─────────── INPUT GUARDRAILS ───────────┐         ┌────────── OUTPUT GUARDRAILS ──────────┐
user input →│ moderation · PII detect · injection scan │→ MODEL →│ moderation · PII redact · schema · exfil │→ action/user
            │ policy/allow checks · jailbreak detect    │         │ scan · grounding/citation · policy        │
            └───────────────────────────────────────────┘         └────────────────────────────────────────┘
                       block / redact / rewrite / route-to-human          block / redact / regenerate / route-to-human

A guardrail is any check that can allow, block, redact, rewrite, or escalate content. Policy is the rules the guardrails enforce ("no medical advice," "no PII in output," "only employees can access HR data"). The model proposes; the guardrails + app decide (Phase 10.05).

Input guardrails (before the model)

• Content moderation — classify for hate/violence/sexual/self-harm/illegal; block disallowed requests (e.g., OpenAI Moderation API, Llama Guard, Azure Content Safety).
• PII detection/redaction — strip or tokenize personal data before it's sent to the provider/stored (02).
• Prompt-injection / jailbreak detection — heuristic/classifier screening (advisory, not a gate — 01).
• Topic / scope enforcement — keep the app on-domain ("only Acme support").
• Rate / quota / authz checks — per-tenant limits and permission checks (04).

Output guardrails (before anyone acts)

This is the more important half — never trust model output, treat it as untrusted input to the next stage:

• Moderation of generated content (the model can produce harmful text even from benign input).
• PII / secret redaction — block leakage of personal data or secrets in the response (02/03).
• Schema / format validation — structured output must validate (Pydantic/JSON Schema); reject/repair invalid output (Phase 10.02).
• Exfiltration scanning — block suspicious URLs / markdown that could leak data (the injection exfil trick, 01).
• Grounding / citation checks — for RAG, verify claims are supported by sources (faithfulness, Phase 9.08).
• Policy compliance — domain rules (no financial/medical/legal advice, required disclaimers).
• Action authorization — for agents, the app (not the model) authorizes any tool call (Phase 10.05).

Where guardrails come from (implementation)

• Hosted moderation APIs (OpenAI Moderation, Azure AI Content Safety) — easy, broad categories.
• Guardrail models — Llama Guard, NeMo Guardrails, Guardrails AI, Constitutional Classifiers — specialized small models/frameworks that classify safety or validate structure.
• Deterministic validators — regex/PII detectors, JSON-schema validators, allow/deny lists, URL scanners.
• The policy engine (PEP/PDP) — for enterprise, a centralized Policy Decision Point evaluates rules and a Policy Enforcement Point enforces them at the gateway (Phase 8.09). This separates policy (rules, centrally managed) from enforcement (in the request path).

The two cardinal rules

1. Fail closed. If a guardrail errs, times out, or is uncertain, the safe default is deny/block, not allow. A moderation API outage must not silently disable moderation. (Balance with availability for low-risk paths, but high-risk actions fail closed.)
2. Defense in depth + LLM-as-judge limits. Layer multiple guardrails; don't rely on one. And note: using an LLM as a guardrail/judge inherits LLM-judge caveats (it can be wrong, biased, or itself injected) — calibrate it and pair it with deterministic checks (Phase 12.02).

The refusal-balance problem (over- vs under-blocking)

Guardrails have two failure modes, and tuning them is a real trade-off:

• Under-blocking (false negatives) — harmful content gets through → safety incident.
• Over-blocking (false positives) — legitimate requests refused → useless, frustrating product ("I can't help with that" on benign queries).

You measure both (violation rate and over-refusal rate) and tune the threshold to your risk profile — a medical or kids' product blocks aggressively; a developer tool tolerates more. This is the same balance evaluated in safety evals (Phase 12.07): safety is a gate, but a guardrail that refuses everything is a failed product.

Guardrails vs the model's own safety

The model's RLHF safety (Phase 13.03) is the inner layer (probabilistic, can be jailbroken); guardrails are the outer deterministic layer you own and can prove. You need both — and you can't outsource your policy enforcement to the model's training.

3. Mental Model

   MODEL = probabilistically safe (RLHF [13.03], jailbreakable). GUARDRAILS = the DETERMINISTIC safety wrapper YOU control.
   "model probably won't" → "system WILL NOT". model PROPOSES, guardrails+app DECIDE [10.05].

   INPUT GUARDRAILS (pre-model): moderation · PII redact [02] · injection/jailbreak detect [01] · topic/scope · rate/authz [04]
   OUTPUT GUARDRAILS (pre-action, MORE important — never trust output):
     moderation · PII/secret redact [02/03] · SCHEMA validation [10.02] · EXFIL scan (URLs/markdown [01]) ·
     grounding/citation [9.08] · policy compliance · ACTION authorization (app, not model [10.05])
   actions a guardrail can take: ALLOW / BLOCK / REDACT / REWRITE / ESCALATE-to-human

   IMPLEMENTATION: hosted moderation (OpenAI/Azure) · guardrail models (Llama Guard, NeMo, Guardrails AI) ·
                   deterministic validators (regex/PII/JSON-schema/URL) · PEP/PDP policy engine at the gateway [8.09]

   ★ TWO CARDINAL RULES: (1) FAIL CLOSED (uncertain/error/timeout → DENY) · (2) DEFENSE IN DEPTH + LLM-judge has limits (calibrate [12.02])
   REFUSAL BALANCE: under-block (false neg → incident) vs over-block (false pos → useless). MEASURE BOTH, tune to risk profile [12.07].

Mnemonic: the model is only probably safe, so wrap it in deterministic guardrails you control — check inputs, and (more importantly) never trust outputs; allow/block/redact/rewrite/escalate; fail closed when uncertain; layer defenses; and tune the refusal balance so you neither leak harm nor refuse everything.

4. Hitchhiker's Guide

What to look for first: what must never happen (the policy: no PII leak, no harmful content, no unauthorized action, no ungrounded medical advice) and what checks the outputs before anyone acts. Output guardrails are the higher-leverage half.

What to ignore at first: building bespoke guardrail models. Start with a hosted moderation API + deterministic validators (PII, schema, URL) + fail-closed on high-risk paths; add specialized guardrails as policy matures.

What misleads beginners:

• Trusting the model's training as the safety layer. It's probabilistic and jailbreakable — you need a deterministic outer layer you own.
• Only checking inputs. Output guardrails matter more — the model can emit harm/PII/exfil even from benign input.
• Fail-open guardrails. A moderation outage that silently allows everything is a breach — fail closed on high-risk paths.
• One guardrail = safe. Layer them; LLM-as-judge guardrails can be wrong/injected (Phase 12.02).
• Tuning only violations. Over-refusal makes a useless product — measure both and tune to risk (Phase 12.07).
• Letting the model authorize actions. Authorization is deterministic app code (Phase 10.05).

How experts reason: they write the policy explicitly (what's disallowed, for whom), enforce it with layered input + output guardrails that fail closed, treat model output as untrusted (moderation + PII/secret redaction + schema + exfil scan + grounding), keep action authorization in deterministic code, centralize enterprise policy in a PEP/PDP at the gateway (Phase 8.09), and measure both violation and over-refusal rates to tune the balance (Phase 12.07).

What matters in production: output is validated before action; high-risk paths fail closed; PII/secrets/harm are blocked; structured output validates; actions are app-authorized; and the refusal balance fits the product's risk.

How to debug/verify: red-team the guardrails (can harmful in/out get through? can a benign request get over-blocked?); simulate a moderation outage (does it fail closed?); verify schema/exfil/PII checks fire; measure violation and over-refusal rates on a labeled set (Phase 12.07).

Questions to ask: what's the policy? are there input AND output guardrails? do they fail closed? is output treated as untrusted (schema/PII/exfil/grounding)? who authorizes actions — app or model? are violation and over-refusal both measured?

What silently fails you: trusting model safety, no output guardrails, fail-open on outage, single-layer guardrails, model-authorized actions, and an untuned refusal balance.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The model is only probabilistically safe (RLHF, jailbreakable) — guardrails are the deterministic safety wrapper you control (Phase 13.03).
• Guardrails check inputs (pre-model) and outputs (pre-action) and can allow/block/redact/rewrite/escalate; the model proposes, guardrails+app decide (Phase 10.05).
• Output guardrails are the more important half — never trust output: moderation, PII/secret redaction, schema validation, exfil scanning, grounding/citation, policy (01/02/Phase 10.02).
• Cardinal rule 1: fail closed — uncertain/error/timeout → deny on high-risk paths (a moderation outage must not silently allow everything).
• Cardinal rule 2: defense in depth — layer guardrails; LLM-as-judge guardrails have limits (can be wrong/injected; calibrate, Phase 12.02).
• Implementations: hosted moderation, guardrail models (Llama Guard/NeMo/Guardrails AI), deterministic validators, and a PEP/PDP policy engine at the gateway (Phase 8.09).
• Refusal balance: measure both violation (under-block) and over-refusal (over-block) rates and tune to the product's risk profile (Phase 12.07).
• Action authorization is deterministic app code, never the model (Phase 10.05).

9. Observations from Real Systems

• Output handling is its own OWASP risk (LLM02) — the real-world lesson that the model's output is untrusted input to the next stage (XSS, exfil, bad SQL, invalid JSON) drives the emphasis on output guardrails.
• Llama Guard / NeMo Guardrails / Guardrails AI became the standard open building blocks — teams compose hosted moderation + a guardrail model + deterministic validators.
• Fail-open outages are a recurring incident — guardrails wired so that a dependency timeout silently disables them; mature teams fail closed and alert.
• Over-refusal kills products — early safety-heavy assistants refused benign requests and lost users; the industry moved to measuring and tuning the refusal balance (Phase 12.07).
• Enterprise deployments centralize policy in a gateway PEP/PDP so rules are managed once and enforced consistently across apps (Phase 8.09).

10. Common Misconceptions

11. Engineering Decision Framework

GUARDRAILS FOR AN LLM APP:
 1. WRITE THE POLICY: what's disallowed (harm/PII/secrets/ungrounded claims/unauthorized actions) and for whom.
 2. INPUT guardrails: moderation + PII redact [02] + injection/jailbreak detect [01] + topic/scope + rate/authz [04].
 3. OUTPUT guardrails (priority): moderation + PII/secret redact [02/03] + SCHEMA validate [10.02] + EXFIL scan [01]
    + grounding/citation [9.08] + policy. Treat output as UNTRUSTED.
 4. ACTIONS: the APP authorizes tool calls deterministically, never the model [10.05].
 5. FAIL CLOSED on high-risk paths (uncertain/error/timeout → deny + alert); pick availability only for low-risk paths.
 6. LAYER (defense in depth); if using an LLM as a guardrail, CALIBRATE it + back with deterministic checks [12.02].
 7. ENTERPRISE: centralize policy in a PEP/PDP at the gateway [8.09].
 8. TUNE the REFUSAL BALANCE: measure violation AND over-refusal rates; set thresholds to the product's risk profile [12.07].

12. Hands-On Lab

Goal

Wrap a model with input and output guardrails that enforce a written policy, fail closed, and let you measure the refusal balance.

Prerequisites

• A simple LLM endpoint; a moderation API (or Llama Guard); a PII detector and a JSON-schema validator; a small labeled set of safe + unsafe requests.

Steps

1. Write the policy: list disallowed inputs/outputs (harmful content, PII leakage, off-topic, ungrounded claims) and the action per violation (block/redact/escalate).
2. Input guardrails: add moderation + PII redaction + a topic/scope check before the model; block/redact per policy.
3. Output guardrails: add moderation + PII/secret redaction + schema validation + an exfil URL scan on the response before returning it (01/Phase 10.02).
4. Fail closed: simulate a moderation-API timeout and confirm the system denies (not allows) on the high-risk path, and alerts.
5. Measure the balance: run the labeled set; compute violation rate (unsafe that got through) and over-refusal rate (safe that got blocked) (Phase 12.07).
6. Tune: adjust thresholds to hit your target balance for the chosen risk profile; re-measure.

Expected output

A guardrailed endpoint enforcing a written policy on inputs and outputs, failing closed on dependency failure, with measured violation and over-refusal rates and a tuned threshold — the deterministic safety layer.

Debugging tips

• If harmful output slips through → strengthen output guardrails (input-only isn't enough).
• If a timeout lets everything through → you're failing open; flip to fail-closed.

Extension task

Add a guardrail model (Llama Guard) and compare it to the hosted moderation API on the same labeled set; add a grounding check for a RAG response (Phase 9.08).

Production extension

Move policy into a PEP/PDP at the gateway (Phase 8.09); log every guardrail decision (06); wire the violation/over-refusal metrics into the safety eval gate (Phase 12.07).

What to measure

Violation rate, over-refusal rate, fail-closed behavior on outage, output-schema/exfil/PII catch rates, hosted-vs-guardrail-model comparison.

Deliverables

• A written policy + input/output guardrails enforcing it.
• Fail-closed behavior on dependency failure (+ alert).
• A violation + over-refusal measurement and a tuned threshold.

13. Verification Questions

Basic

1. Why can't the model's own safety training be your only safety layer?
2. What can a guardrail do to content (the five actions)?
3. Why are output guardrails more important than input guardrails?

Applied 4. What does "fail closed" mean and where do you apply it? 5. What is the refusal-balance problem and how do you manage it?

Debugging 6. Harmful text reaches users despite input moderation. What's missing? 7. A moderation outage let everything through. What's the fix?

System design 8. Design the guardrail layer for a healthcare assistant (policy, input/output checks, fail-closed, grounding).

Startup / product 9. How do you keep guardrails safe without over-refusing and frustrating users?

14. Takeaways

1. The model is only probabilistically safe — guardrails are the deterministic safety wrapper you own (Phase 13.03).
2. Check inputs and (more importantly) outputs — never trust output: moderation, PII/secret redaction, schema, exfil scan, grounding (01/02).
3. Fail closed on high-risk paths and layer guardrails; LLM-as-judge guardrails have limits (Phase 12.02).
4. Action authorization is deterministic app code, never the model (Phase 10.05); centralize enterprise policy in a PEP/PDP (Phase 8.09).
5. Measure both violation and over-refusal rates and tune to the product's risk — safety is a gate, but a guardrail that refuses everything is a failed product (Phase 12.07).

15. Artifact Checklist

• A written policy (disallowed inputs/outputs + action per violation).
• Input guardrails (moderation + PII + injection/scope + authz).
• Output guardrails (moderation + PII/secret + schema + exfil + grounding).
• Fail-closed behavior on dependency failure (+ alerting).
• Violation + over-refusal measurements with a tuned threshold.

Up: Phase 14 Index · Next: 06 — Audit Logs