How to Read System Cards

Phase 3 · Document 01 · Model Cards and System Cards Prev: 00 — How to Read Model Cards · Next: 02 — Google DeepMind Model Card Guide

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A system card is where a lab discloses how a model behaves under stress: what it refuses, where it can be jailbroken, what dangerous-capability evaluations were run, and what deployment safeguards exist. For anyone shipping to real users — especially enterprise or regulated customers — this is the document that answers "is it safe to deploy, and what guardrails must I add?" Model cards tell you what a model can do; system cards tell you what it might do wrong and what the lab did about it. Skipping it means discovering refusal behavior, injection susceptibility, or policy gaps in production instead of in review.

2. Core Concept

Model card vs system card

• A model card (00) centers on capabilities, training, and limitations.
• A system card centers on safety: capability and danger evaluations, red-teaming, refusal/jailbreak behavior, and the deployment policy/mitigations the lab applied. OpenAI and Anthropic publish these; they often accompany a frontier release.

They overlap (both list evals and limits) but answer different questions. For a production launch you read both.

What a system card typically contains

• Capability evaluations: benchmarks for coding, math, reasoning, multimodal — the "how good" story.
• Safety/dangerous-capability evaluations: structured tests for high-risk domains (e.g. cyber, bio, persuasion, autonomy) and the lab's risk thresholds/framework.
• Red-teaming: internal and external adversarial testing; what attacks were tried and how the model held up.
• Refusal & over-refusal behavior: what it declines, and how often it wrongly refuses benign requests (a real UX/quality cost).
• Jailbreak resistance: susceptibility to prompt attacks and how mitigations changed it.
• Mitigations & deployment policy: classifiers, filters, usage policies, monitoring, and any staged-release/risk-level decisions.
• Known risks & limitations: honestly disclosed failure modes.

What you extract (the launch lens)

A system card is not academic reading — it feeds concrete decisions:

1. Refusal/over-refusal profile → will it frustrate users or block legitimate flows?
2. Injection/jailbreak susceptibility → how much guardrailing must your app add (Phase 14)?
3. Domain risks relevant to you → does the card flag weaknesses in your use domain?
4. Provider mitigations vs your responsibilities → which safety controls are the model's, and which are yours to build? (The model proposes; the application enforces — Phase 1.00, Law 5–6.)

The honest-but-self-published caveat

System cards are increasingly rigorous, but they're still authored by the lab releasing the model. Evals reflect the lab's chosen threats and thresholds; "passed our safety evals" is not "safe for your context." You still run your own safety/policy evals (Phase 13) for your data, users, and jurisdiction.

3. Mental Model

MODEL CARD  = "what it CAN do"        (capabilities, training, limits)
SYSTEM CARD = "what it might do WRONG + what we did about it"
              capability evals · DANGER evals · red-team · refusals · jailbreaks · mitigations · policy

LAUNCH LENS (read for decisions, not trivia):
  refusal/over-refusal → UX & blocked flows
  injection/jailbreak  → how much guardrailing I must add (Phase 14)
  domain risks         → relevant to MY use?
  provider mitigations → which controls are theirs vs MINE?

"Passed the lab's safety evals" ≠ "safe for MY context" → run YOUR policy evals.

4. Hitchhiker's Guide

What to read first: refusal/over-refusal behavior, jailbreak/injection findings, and the mitigations-vs-your-responsibility split. These directly shape your guardrail work.

What to ignore at first: the lab's internal risk-framework taxonomy details — note the headline risk level, skip the governance prose on a first pass.

What misleads beginners:

• Reading "passed safety evals" as "safe for me." The lab tested its threats, not yours.
• Ignoring over-refusal — a model that refuses benign requests tanks product UX.
• Assuming provider mitigations cover your app — you still own input/output validation, permissions, and audit.

How experts reason: they map each system-card finding to a concrete guardrail or product decision, and they treat the card as evidence for an enterprise-readiness assessment, not a guarantee.

What matters in production: refusal UX, injection defenses, output classification/PII handling, audit logging, and a documented division of safety responsibility (Phase 14).

How to verify: run your own red-team prompts and over-refusal probes against the model; measure violation and false-refusal rates on your content (Phase 13).

Questions to ask the provider: What's the documented over-refusal rate? What jailbreak classes remain open? Which mitigations are server-side vs my responsibility? What's the data-retention/monitoring policy? Any usage-policy restrictions for my domain?

What silently gets risky: shipping without injection defenses because "the model is safe"; ignoring over-refusal until users complain; assuming the provider logs/audits for you.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A system card focuses on safety: capability and dangerous-capability evals, red-teaming, refusals, jailbreaks, mitigations, and deployment policy.
• Model card ≠ system card — read both for a production launch.
• "Passed the lab's safety evals" ≠ "safe for your context" — run your own policy evals.
• Over-refusal is a real cost — a safe-but-frustrating model can fail your product.
• Some mitigations are server-side; many guardrails are your responsibility (input/output validation, permissions, audit).
• The application enforces safety — model safety training is soft, not a guarantee (Phase 1.00).
• System cards reflect the lab's chosen threats and thresholds, not yours.
• They're essential evidence for enterprise-readiness and compliance assessments (Phase 14).

9. Observations from Real Systems

• Anthropic system cards detail Constitutional-AI behavior, capability + safety evals, red-teaming, and ASL risk levels — Doc 03.
• OpenAI system cards pair releases with Preparedness-framework risk assessments and deployment mitigations.
• Enterprise procurement teams request system cards + data-handling docs as part of vendor review — they're a sales artifact, not just research.
• AI gateways (Phase 8) add the application-side mitigations a system card says are your responsibility (filters, audit, policy engine).
• Over-refusal complaints are a common production issue traceable to a model's documented refusal posture.

10. Common Misconceptions

11. Engineering Decision Framework

Pre-launch safety read:
  1. Refusal/over-refusal profile → acceptable for my flows? test on my content.
  2. Jailbreak/injection findings → what residual risk → what app guardrails (Phase 14)?
  3. Domain dangerous-capability notes → relevant to my use? extra controls?
  4. Mitigation split → list PROVIDER controls vs MINE; build mine.
  5. Run MY policy/safety eval (Phase 13) on my data/users/jurisdiction.
  6. Record an enterprise-readiness note for procurement.

Gate: if residual risk × my domain > tolerance AND I can't mitigate at the app layer → don't ship it here.

12. Hands-On Lab

Goal

Turn a real system card into an enterprise-readiness assessment: extract safety findings, split provider vs your responsibilities, and run a small over-refusal/injection probe.

Prerequisites

• Access to a recent system card; an API key for the model it describes.

Steps

1. Read a current Anthropic or OpenAI system card. Extract: refusal posture, notable red-team/jailbreak findings, dangerous-capability domains, and listed mitigations.
2. Build a 2-column table: provider mitigations vs your responsibilities for your use case.
3. Write 10 over-refusal probes (benign requests near a sensitive boundary) and 5 injection probes; run them against the model and record refusal/violation outcomes.
4. Produce a one-page enterprise-readiness note: residual risks, required app-side guardrails, and a go/no-go for your context.

Expected output

A responsibilities table, probe results (refusal/violation rates), and a readiness note with a decision.

Debugging tips

• All probes refused? You may be over the boundary — dial probes toward clearly-benign to measure over-refusal.
• No system card for an open model? Use its model card's safety section + your probes; note the disclosure gap.

Extension task

Add the probe set to your eval harness (Phase 1.07) so refusal/violation rates are tracked on every model/prompt change.

Production extension

Sketch the app-side guardrail layer (input sanitizer, output classifier, audit log) the assessment requires — the seed of Phase 14.

What to measure

Over-refusal rate, injection-success rate, count of guardrails that are your responsibility, residual-risk verdict.

Deliverables

• Provider-vs-you responsibilities table.
• Probe results (refusal + injection).
• A one-page enterprise-readiness note with a go/no-go.

13. Verification Questions

Basic

1. How does a system card differ from a model card?
2. What does a system card typically include beyond capabilities?
3. Why isn't "passed the lab's safety evals" sufficient for your launch?

Applied 4. Why does over-refusal matter as much as harmful-content risk for a product? 5. Give two mitigations that are the provider's job and two that are yours.

Debugging 6. Users report the assistant refuses legitimate requests. Where in the system card would you have seen this risk, and how do you fix it? 7. A prompt-injection incident occurs despite a "safe" model. What did you fail to own?

System design 8. Design an enterprise-readiness review that combines a system card with your own evals and guardrails.

Startup / product 9. An enterprise buyer asks for your AI safety posture. How do you use the provider's system card plus your own controls to answer credibly?

14. Takeaways

1. A system card discloses safety: capability + danger evals, red-teaming, refusals, jailbreaks, mitigations, policy.
2. Read it alongside the model card before any real launch.
3. "Passed their evals" ≠ "safe for you" — run your own policy/safety evals.
4. Over-refusal is a real product cost, not just a safety nicety.
5. Split provider mitigations from your responsibilities — and build yours.
6. System cards are key evidence for enterprise readiness and compliance.

15. Artifact Checklist

• Provider-vs-you responsibilities table.
• Over-refusal + injection probe set and results.
• Enterprise-readiness note with a go/no-go.
• Guardrail sketch for app-side controls (→ Phase 14).
• Eval integration: refusal/violation probes in your harness.
• Notes: model-card vs system-card distinction and the launch lens.

Next: 02 — Google DeepMind Model Card Guide