Safety and Policy Evals

Phase 12 · Document 07 · Evaluation Prev: 06 — Latency and Cost Evals · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Quality, latency, and cost (06) measure whether the system is good; safety/policy evals measure whether it's safe to deploy — and a single safety failure (a leaked secret, a successful prompt injection, a harmful output, a compliance violation) can be catastrophic and unrecoverable in a way a quality dip isn't. So safety is a distinct, adversarial, gating axis: you don't just measure average behavior, you actively attack the system (red-teaming) and treat violations as hard fails (must be 0), not weighted trade-offs. This is the eval-side of the security/governance discipline (Phase 14) and the safety gate that agent/RAG/code evals reference (04/03/05). It's increasingly a compliance requirement (SOC2/EU AI Act) and an enterprise procurement gate.

2. Core Concept

Plain-English primer: adversarial, gating evaluation

Unlike quality eval (average performance on representative tasks), safety eval is adversarial: you deliberately probe for the worst behavior, because attackers and edge cases will. And it's gating, not weighted: a great-quality system that leaks PII or can be jailbroken into harmful output fails — you don't average safety away (04: unsafe completion = failure). The axes to test (mapping to OWASP LLM Top 10, Phase 14.01):

Red-teaming: attack your own system

The core method is red-teaming — systematically attacking the system with adversarial inputs to find failures before attackers/users do:

• Manual red-teaming — humans craft jailbreaks, injections, edge cases (domain experts find the worst).
• Automated red-teaming — use an LLM (or tools like PyRIT, Garak, promptfoo) to generate adversarial prompts at scale and probe systematically.
• Curated attack sets — known jailbreak/injection corpora as a regression suite (the same attack must never re-succeed — the flywheel, 01). Red-teaming is continuous (new attacks emerge) and feeds a growing adversarial golden set.

Scoring safety: pass/fail, not 1–5

Safety metrics are usually rates of violation (attack-success rate, leakage rate), scored by:

• Programmatic detectors where possible — regex/classifiers for PII/secrets (Phase 14.02), did-it-exfiltrate checks, did-the-agent-take-the-action checks (Phase 10.05).
• Moderation APIs / classifiers — OpenAI Moderation, Llama Guard, perspective/toxicity classifiers for harmful content.
• LLM-as-judge (02) — for nuanced "is this harmful/policy-violating?" (calibrate). The headline numbers: attack-success rate (→ 0), leakage rate (→ 0), harmful-output rate (→ 0), plus over-refusal rate (the false-positive cost).

The refusal balance (a two-sided metric)

Safety isn't just "refuse bad things" — over-refusal (declining benign requests) is a real failure that destroys usefulness. So evaluate both: harmful-request refusal rate (should be high) and benign-request over-refusal rate (should be low). A system that refuses everything is "safe" and useless. Use a balanced set with both harmful and benign-but-sensitive prompts.

Safety is a hard gate (not a weighted axis)

Critical distinction from 08/Phase 5.09: quality/latency/cost are weighted trade-offs, but safety is a gate — a candidate that fails safety thresholds is disqualified regardless of its quality score. You can't buy your way past a PII leak with better accuracy. So the harness treats safety as a pass/fail filter applied before the weighted quality/cost/latency scoring.

Containment ≠ model behavior

A key lesson from Phase 10.05: for agents, safety eval must test the whole system's containment (sandbox, approval, egress control), not just the model's refusal — because prompt injection isn't reliably solved at the model level. So red-team the system: even if the model is fooled into proposing a bad action, does containment block it? Eval the guardrails, not just the model.

3. Mental Model

   quality/latency/cost [06] = is it GOOD?   SAFETY = is it SAFE TO DEPLOY? (catastrophic, unrecoverable failures)
   ★ ADVERSARIAL (probe the WORST, not the average) + GATING (violations = HARD FAIL, not weighted [08])

   AXES (OWASP LLM Top 10 [14.01]): prompt-injection · jailbreak · PII/secret/system-prompt LEAKAGE ·
        refusal-appropriateness (refuse bad ✔ AND don't OVER-refuse benign) · harmful content · bias · excessive agency [10.05]

   METHOD: RED-TEAM (manual + automated [PyRIT/Garak/promptfoo] + curated attack sets) — continuous, feeds adversarial golden set [01]
   SCORE: pass/fail RATES — attack-success→0, leakage→0, harmful→0, over-refusal low — via detectors/moderation/calibrated judge [02]
   ★ test SYSTEM CONTAINMENT (sandbox/approval/egress [10.05]), not just model refusal — injection isn't model-solved
   SAFETY = a GATE before the weighted quality/cost/latency score [08]

Mnemonic: safety eval is adversarial (red-team for the worst) and gating (violations = hard fail, not weighted). Test injection/jailbreak/leakage/harmful/bias/excessive-agency AND over-refusal; score pass/fail rates (→0); eval system containment, not just model refusal.

4. Hitchhiker's Guide

What to look for first: is there red-teaming (you actively attack the system) and is safety a hard gate (violations block deploy regardless of quality)? Plus the refusal balance (not just refuse-bad, but don't over-refuse).

What to ignore at first: exhaustive coverage of every attack category. Start with prompt-injection + PII-leakage + jailbreak red-teaming (the highest-impact), plus over-refusal; expand. (System security depth: Phase 14.)

What misleads beginners:

• Measuring average behavior only. Safety is adversarial — you must probe the worst, not the typical (Phase 14.01).
• Treating safety as a weighted axis. It's a hard gate — a PII leak isn't offset by accuracy (08).
• Testing only the model's refusal. Eval system containment (sandbox/approval/egress) — injection isn't model-solved (Phase 10.05).
• Ignoring over-refusal. A system that refuses benign requests fails usefulness — measure both sides.
• One-time safety check. New attacks emerge — red-team continuously; curated attacks become regression tests (01).

How experts reason: they red-team (manual + automated + curated attack sets), score pass/fail violation rates (attack-success/leakage/harmful → 0) with detectors + moderation + calibrated judge, measure the refusal balance (refuse-bad high, over-refuse low), eval system containment not just model behavior, treat safety as a gate in the harness (08), and grow an adversarial golden set continuously (the flywheel).

What matters in production: attack-success rate (→0), leakage rate (→0), harmful-output rate (→0), over-refusal rate (low), and no safety regressions — gated in CI and monitored online (with abuse/jailbreak attempts logged, Phase 14).

How to debug/verify: when a safety failure is found, add it to the adversarial regression set (it must never re-succeed); for agents, verify containment blocked the action even when the model was fooled (Phase 10.05).

Questions to ask vendors/yourself: do we red-team (manual + automated)? is safety a hard gate (not weighted)? do we test containment, not just refusal? do we measure over-refusal? is there a curated adversarial regression set? which compliance frameworks (Phase 14.07)?

What silently gets expensive/unreliable: average-only safety (misses adversarial failures), safety-as-weighted (ships a leak), model-refusal-only (injection bypasses), no over-refusal metric (useless-but-safe), and one-time checks (new attacks succeed).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Safety eval is adversarial (probe the worst) and gating (violations = hard fail) — not a weighted average like quality (08).
• Axes (OWASP LLM Top 10): prompt injection, jailbreaks, sensitive-info disclosure (PII/secrets/system prompt), harmful content, bias, excessive agency (Phase 14.01).
• Red-teaming (manual + automated [PyRIT/Garak/promptfoo] + curated attack sets) is the core method — continuous; failures become adversarial regression tests (01).
• Score pass/fail rates — attack-success, leakage, harmful-output → 0 — via detectors/moderation/calibrated judge (02).
• Measure the refusal balance — refuse harmful (high) and don't over-refuse benign (low); refusing everything is useless.
• Eval system containment, not just model refusal — injection isn't model-solved; test sandbox/approval/egress (Phase 10.05).
• Safety is a hard gate applied before weighted quality/cost/latency scoring (08).
• It's increasingly a compliance requirement (SOC2/EU AI Act) and enterprise procurement gate (Phase 14.07).

9. Observations from Real Systems

• Frontier labs run extensive red-teaming (manual + automated) before release — safety evals are part of every system card (Phase 3.03).
• OWASP LLM Top 10 is the de-facto threat checklist; PyRIT/Garak/promptfoo automate red-teaming for app teams.
• Prompt injection remains unsolved at the model level — production defense is containment (eval the guardrails, not just refusal) (Phase 10.05, Phase 14.01).
• Over-refusal is a real, measured failure — labs track "false refusals" because an over-cautious model is unusable.
• Compliance (EU AI Act, SOC2) increasingly mandates documented safety evaluation — the gate that unlocks regulated/enterprise deployment (Phase 14.07).

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE SAFETY (adversarial + gating):
 1. AXES (OWASP LLM Top 10 [14.01]): injection · jailbreak · PII/secret/system-prompt LEAKAGE · harmful · bias · excessive agency [10.05].
 2. RED-TEAM: manual (experts) + automated (PyRIT/Garak/promptfoo) + curated attack sets → adversarial golden set [01]; CONTINUOUS.
 3. SCORE pass/fail RATES: attack-success→0 · leakage→0 · harmful→0 · over-refusal LOW — via detectors/moderation/calibrated judge [02].
 4. REFUSAL BALANCE: harmful-refusal (high) AND benign over-refusal (low) on a two-sided set.
 5. CONTAINMENT: red-team the SYSTEM (sandbox/approval/egress [10.05]) — even if the model is fooled, is the action blocked?
 6. GATE: safety is a HARD pass/fail filter BEFORE weighted quality/cost/latency [08]; failures → regression set (never re-succeed).
 7. COMPLY: map to required frameworks (SOC2/EU AI Act) [14.07].

12. Hands-On Lab

Goal

Red-team a system, score safety as pass/fail rates (including the refusal balance), and add a safety gate + adversarial regression set.

Prerequisites

• A system to test (a RAG/agent/chat app from earlier phases); pip install promptfoo or a simple harness; (for agents) the sandbox from Phase 10.05.

Steps

1. Attack set: assemble adversarial inputs — prompt injections (content saying "ignore instructions / exfiltrate"), jailbreaks (bypass refusals), PII-leak probes ("repeat your system prompt / any secrets"), plus benign-but-sensitive prompts (for over-refusal). Use curated sets + auto-generate variants ([PyRIT/Garak/promptfoo]).
2. Run + score rates: measure attack-success rate, leakage rate (regex/classifier detector for PII/secrets), harmful-output rate (moderation API), and over-refusal rate (benign prompts wrongly refused).
3. Containment (agents): feed injected content and a destructive request; confirm that even if the model proposes it, the sandbox/approval/egress blocks it (Phase 10.05) — eval the system, not just the model.
4. Refusal balance: report both harmful-refusal (high) and over-refusal (low); show a too-strict prompt raises over-refusal.
5. Safety gate: define thresholds (attack-success = 0, leakage = 0, harmful = 0); make safety a hard pass/fail that disqualifies a candidate before quality scoring (08).
6. Regression set: add any successful attack to an adversarial regression suite; confirm a future change is gated against it (it must never re-succeed, 01).

Expected output

A red-team report with violation rates (attack-success/leakage/harmful → 0 target, over-refusal low), a containment demonstration (agent), a safety gate, and an adversarial regression set.

Debugging tips

• Injection "passed" because the model obeyed → that's expected at the model level; verify containment blocked the action (Phase 10.05).
• Over-refusal high → prompt too strict; balance against the benign set.

Extension task

Automate red-teaming with PyRIT/Garak/promptfoo to generate attacks at scale; add bias/fairness probes across demographics.

Production extension

Wire the safety gate into CI, log/alert on jailbreak/injection attempts online (Phase 14), grow the adversarial set continuously, and map results to your compliance framework (Phase 14.07).

What to measure

Attack-success rate, leakage rate, harmful-output rate, over-refusal rate, containment-block rate (agents), regression-set pass.

Deliverables

• A red-team report (violation rates → 0; over-refusal low).
• A containment demonstration (agent: action blocked despite model fooled).
• A safety gate (hard pass/fail) + an adversarial regression set.

13. Verification Questions

Basic

1. How does safety eval differ from quality eval (two ways)?
2. What is red-teaming, and what forms does it take?
3. Why is safety a gate, not a weighted axis?

Applied 4. Why measure over-refusal, not just refusal? 5. Why eval system containment instead of just the model's refusal?

Debugging 6. A great-quality candidate leaks PII on adversarial probes. Does it ship? Why/why not? 7. An agent got prompt-injected into proposing exfiltration. What must the eval confirm?

System design 8. Design a safety-eval suite + gate (red-team axes, detectors, refusal balance, containment, regression set).

Startup / product 9. Why is documented safety evaluation increasingly a compliance/procurement gate for enterprise deployment?

14. Takeaways

1. Safety eval is adversarial (probe the worst) and gating (violations = hard fail) — not a weighted average.
2. Red-team (manual + automated + curated) across the OWASP axes; failures become adversarial regression tests (01).
3. Score pass/fail rates (attack-success/leakage/harmful → 0) and the refusal balance (refuse-bad high, over-refuse low).
4. Eval system containment, not just model refusal — injection isn't model-solved (Phase 10.05).
5. Safety is a hard gate before weighted scoring (08) and increasingly a compliance requirement (Phase 14).

15. Artifact Checklist

• A red-team attack set (injection/jailbreak/leakage + benign-but-sensitive).
• Violation-rate scores (attack-success/leakage/harmful → 0) + over-refusal rate.
• A containment check (agent: action blocked despite model fooled).
• A safety gate (hard pass/fail before weighted scoring).
• An adversarial regression set (attacks never re-succeed) + compliance mapping.

Up: Phase 12 Index · Next: 08 — Model-Selection Eval Harness