« Phase 17 · Warmup · Lab 01

Core Contributor — The Literature and the Open Implementations

Where the ideas in this phase come from, what to read, and which codebases to open when you want the real version.

Table of Contents

• 1. Composition as a safety problem
• 2. Agent frameworks, read for their seams
• 3. Prompt injection: the papers that matter
• 4. Benchmarks that attack a composition
• 5. Defence depth: where the idea comes from
• 6. Chaos engineering, and its actual literature
• 7. Policy engines you would really use
• 8. Tamper-evident logs
• 9. Standards and regulation
• 10. Determinism and reproducibility
• 11. Reading order
• 12. How to contribute

1. Composition as a safety problem

The intellectual core of this phase is fifty years old and has nothing to do with AI.

Leveson, N. — Engineering a Safer World: Systems Thinking Applied to Safety (MIT Press, 2011; free PDF from the author). STAMP's central claim: in a complex system, accidents are not caused by component failures but by unsafe interactions between components that each satisfy their requirements. Leveson's canonical example is the Mars Polar Lander — the landing-leg sensor correctly reported touchdown when the legs deployed, and the engine controller correctly shut down on touchdown. Both correct; spacecraft destroyed. Read chapters 2 and 4; the rest is a control-theoretic framework you can take or leave, but those two chapters are exactly this phase's argument.

Perrow, C. — Normal Accidents (Princeton, 1999). Interactive complexity plus tight coupling makes certain accidents normal — expected properties of the system rather than aberrations. The prescription is to reduce coupling, and it is the strongest available argument for degradation ladders and circuit breakers.

Hollnagel, E. & Woods, D. — Resilience Engineering, and Woods's later "graceful extensibility" work. The distinction between a system that is robust (survives anticipated disturbances) and one that is resilient (adapts to unanticipated ones). The degradation ladder is a robustness mechanism; the escalation path is a resilience one.

Cook, R. — "How Complex Systems Fail" (1998, four pages, https://how.complexsystems.fail/). Read it today. Point 3 — catastrophe requires multiple failures — is the defence-depth argument stated eight years before anyone wrote a policy engine.

2. Agent frameworks, read for their seams

Read these asking a single question: where does the identity chain live, and what happens to it at a delegation?

The pattern you will find: all of them model the flow and none of them model the principal. That is the gap the enterprise platform fills, and it is why this track exists. When you evaluate a framework for a bank, the first question is not "can it do multi-agent" — it is "where does the human go, and can it be lost?"

Also worth reading for the control-plane side:

• Model Context Protocol — the tool-connection standard, plus its evolving authorization spec. Read the spec's security considerations section specifically; it is unusually candid about what the protocol does not solve.
• Envoy ext_authz — the mature version of "a control plane makes an admission decision on every request", including the fail-open/fail-closed configuration flag and the arguments in its issue tracker about the default.

3. Prompt injection: the papers that matter

Greshake, K. et al. — "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (AISec 2023, arXiv:2302.12173). The paper that named indirect injection: the payload is not in the user's message, it is in the content the application retrieves. This is the threat model the taint rule addresses; every retrieval-augmented agent inherits it.

Willison, S. — the prompt-injection series (https://simonwillison.net/tags/prompt-injection/). Not academic and more useful than most of what is. Two essential pieces: the coining of the term (Sept 2022), and the dual-LLM pattern (April 2023) — a privileged model that never sees untrusted content and a quarantined model that does but cannot act. The dual-LLM pattern is the strongest structural alternative to the containment rule, and knowing it by name is a differentiator.

Wallace, E. et al. — "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions" (arXiv:2404.13208, OpenAI, 2024). The model-side mitigation. Read it for the honest evaluation section: substantial improvement, not a solution — which is precisely why the architecture must assume it fails.

Zverev, E. et al. — "Can LLMs Separate Instructions From Data?" (arXiv:2403.06833, 2024). Proposes a measure of separation and finds current models do poorly. The empirical backing for "the model has one channel".

Debenedetti, E. et al. — "Defeating Prompt Injections by Design" (CaMeL, arXiv:2503.18813, 2025). A design-level defence using a capability-based control flow that keeps untrusted data out of the code path. The most serious recent attempt at a structural rather than statistical answer, and worth reading alongside the dual-LLM pattern.

4. Benchmarks that attack a composition

AgentDojo (Debenedetti et al., NeurIPS Datasets & Benchmarks 2024, arXiv:2406.13352). 97 realistic tasks and 629 security test cases against agents with real tool calls. The important design property — and the one to steal — is that it measures utility under attack, not just attack success. A defence that stops every attack by refusing everything scores zero. That is exactly the must_deny=False control case in this lab's harness.

InjecAgent (arXiv:2403.02691). 1,054 cases across 17 tools, split into direct-harm and data-stealing. Useful taxonomy if you are building your own case list.

OWASP Top 10 for LLM Applications (2025 edition). LLM01 prompt injection through LLM10 unbounded consumption. Generate your attack cases from it — a new row becomes a missing case rather than a gap nobody noticed (Phase 11 does this explicitly).

MITRE ATLAS. ATT&CK's adversarial-ML sibling, with real case studies. The right vocabulary for talking to a bank's Cyber function, because they already speak ATT&CK.

Purple Llama / LlamaFirewall (Meta). CyberSecEval benchmarks plus a guardrail implementation. Read PromptGuard and note how narrow a well-built scanner's claims are.

5. Defence depth: where the idea comes from

Defence in depth is a military doctrine, an NSA information-assurance principle (Defense in Depth: A practical strategy, 2004), and now a checkbox in every security architecture document. The literature on measuring it is thinner than you would expect.

• NIST SP 800-53 Rev. 5 — control layering and compensating controls. The formal vocabulary for "these two controls are independent".
• Schneier, B. — "Attack Trees" (Dr. Dobb's, 1999). Model an attack as a tree of goals; the paths through it are your layers. The cleanest formal route to the depth number.
• Reason, J. — the Swiss cheese model (Human Error, 1990, and BMJ 2000). The origin of the visual everybody uses. Its key claim is the one people forget: the holes move, and layers whose holes are correlated do not compose. That is the independence argument in §2 of the Principal Deep Dive.
• Lampson, B. — "Protection" (1971) and "Computer Security in the Real World" (2004). The access matrix, and the observation that real security is about accountability and recovery at least as much as prevention.

6. Chaos engineering, and its actual literature

Basiri, A. et al. — "Chaos Engineering" (IEEE Software 33(3), 2016). The Netflix paper. The formal definition is hypothesis-first: state your steady-state hypothesis, inject, compare. The ChaosCase dataclass in this lab is that definition as a type.

Rosenthal, C. & Jones, N. — Chaos Engineering: System Resiliency in Practice (O'Reilly, 2020). The book. Chapter on "the advanced principles" is the one to read: run in production, automate, minimize blast radius.

Beyer, B. et al. — Site Reliability Engineering, ch. 22 ("Addressing Cascading Failures") and The SRE Workbook, ch. 5 (https://sre.google/books/). Cascading failure is the mechanism the degradation ladder is designed against, and ch. 22 is still the best treatment.

Tooling worth knowing: Chaos Mesh and LitmusChaos for Kubernetes fault injection; Toxiproxy for the version you can actually run in CI — latency, bandwidth and connection faults on a TCP proxy, which is the cheapest realistic upgrade from this lab's boolean switches.

7. Policy engines you would really use

Cedar deserves the closest look for this domain. Its analyzability is not a marketing claim — you can mechanically ask "does any principal have permission to do X?", which is the question a regulator asks and Rego cannot answer in general. Read the Cedar paper (Cutler et al., 2024) for how the language was deliberately restricted to keep that property.

Also read Zanzibar (Pang et al., USENIX ATC 2019) — Google's global authorization system. The paper's treatment of consistency and the "zookie" is the serious version of this lab's freshness handwave.

8. Tamper-evident logs

Haber, S. & Stornetta, W. S. — "How to Time-Stamp a Digital Document" (Journal of Cryptology 3(2), 1991). The hash chain, in its original form. Twenty-eight years before anyone said blockchain.

Crosby, S. & Wallach, D. — "Efficient Data Structures for Tamper-Evident Logging" (USENIX Security 2009). History trees; how to prove membership and consistency without rereading the whole log. The upgrade path from _chain().

Laurie, B. et al. — RFC 6962 / RFC 9162, Certificate Transparency. The best-deployed tamper-evident log in existence, with Merkle inclusion and consistency proofs. If you need to prove to a third party that an entry was in your log at time T, read this rather than inventing it.

Rescorla, E. — RFC 8785, JSON Canonicalization Scheme. Why sort_keys=True and fixed separators are correctness rather than style. A chain over a non-canonical encoding verifies only on the machine that wrote it.

Trillian is the production implementation of a verifiable log; Rekor is Sigstore's transparency log built on it and is readable in an afternoon.

9. Standards and regulation

CBUAE — the direct regulator for the JD this track answers. The Guidance on Outsourcing and Cloud Computing, the Model Management Standard, and the Consumer Protection Regulation (which is where explainability obligations bite for a customer-facing agent).

EU AI Act (Reg. 2024/1689). Read Art. 9 (risk management), Art. 12 (record-keeping), Art. 14 (human oversight) and Art. 26 (deployer obligations). Art. 12 is the clearest statutory articulation anywhere of "the output of a run is an evidence pack", and it is worth quoting in a design review.

NIST AI RMF 1.0 (Jan 2023) plus the Generative AI Profile (NIST AI 600-1, July 2024). GOVERN / MAP / MEASURE / MANAGE. The Generative AI Profile is the more useful of the two documents in practice.

ISO/IEC 42001:2023 — AI management systems. Certifiable, which is why your organization will care.

SR 11-7 (US Federal Reserve, 2011) — model risk management. Twenty pages, still the clearest statement of independent validation and the effective-challenge principle (Phase 15 is built on it).

Basel Committee — Principles for Operational Resilience (2021) and Principles for the Sound Management of Operational Risk (rev. 2021). The vocabulary — impact tolerance, critical operations, severe-but-plausible scenarios — is exactly what a chaos suite operationalizes, and using it in a board conversation lands.

10. Determinism and reproducibility

• Sculley, D. et al. — "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015). The configuration-debt and entanglement sections are the argument for the six pins.
• Mitchell, M. et al. — "Model Cards for Model Reporting" (FAT* 2019, arXiv:1810.03993).
• Gebru, T. et al. — "Datasheets for Datasets" (arXiv:1803.09010).
• Reproducible Builds — the software-supply-chain community's work on byte-identical outputs. SOURCE_DATE_EPOCH and its friends are the same problem this lab solves with an injected clock.

11. Reading order

If you have a weekend:

1. Cook, "How Complex Systems Fail" — 4 pages, today.
2. Leveson, ch. 2 and 4 — the theoretical core.
3. Greshake et al. — the threat model.
4. Willison's dual-LLM post — the structural alternative.
5. Basiri et al. — chaos, formally.
6. EU AI Act Art. 12 — evidence, statutorily.
7. AgentDojo's README and its evaluation design — how to measure a defence without breaking utility.

If you have an hour: Cook, then the dual-LLM post. They are the two that change how you read every architecture diagram afterwards.

12. How to contribute

The genuinely underserved areas, in order of how much a good contribution would be worth:

1. A composition test harness. There is no widely-used tool for "assert that identity propagates across every hop of an agent graph". Every team writes it badly, once.
2. Defence depth as a library. The dataclasses in this lab, generalized: instrument a request path, count distinct denying layers, fail CI on a regression.
3. Taint tracking that survives summarization. The hard version of Phase 11, and an open research problem with a very practical payoff.
4. Chaos cases for agent platforms. Chaos Mesh has no notion of "the model provider returns plausible nonsense" or "retrieval returns stale documents" — the failure modes specific to this architecture have no tooling.
5. An evidence-pack schema. An open, versioned schema for what an agent run must record, mapped to EU AI Act Art. 12 and SR 11-7. Every bank is inventing this privately right now.