« Phase 17 · Lab 01 · Track Overview

Warmup — Composition, from Zero to Principal

Table of Contents

• 0. Where this sits
• 1. From first principles: what a component test cannot see
• 2. The request path, drawn
• 3. The order is the architecture
• 4. A catalogue of seams
• 5. Defence depth: turning a slogan into a number
• 6. Acting is not halting
• 7. The degradation ladder, and the invariant
• 8. Fail-open, fail-shut, fail-static
• 9. Containment: what the taint rule actually buys
• 10. The output of a run is an evidence pack
• 11. End-to-end budgets
• 12. Chaos: declare, then inject
• 13. What the composition still cannot tell you
• 14. Numbers worth carrying
• 15. Whiteboarding the platform in eight minutes
• 16. Interview questions, answered
• 17. References

0. Where this sits

Seventeen phases; sixteen mechanisms. This one is the only phase that does not introduce a new mechanism, and it is the one the interview is actually about — because a candidate who can describe a guardrail chain is common, and a candidate who can say what happens to the guardrail chain when the model provider 429s during a delegated call from a suspended agent is not.

What it consumes:

1. From first principles: what a component test cannot see

Take two components, each correct.

Component A — the delegation helper. It takes a principal and an agent name and returns a new principal for the next hop. Its test suite asserts that the returned principal names the new agent, that a cycle raises, and that the tenant is preserved. Fifteen tests, all green.

Component B — the audit writer. It takes a principal and an action and writes a record. Its test suite asserts that the record names the acting agent, carries a timestamp, and hash-chains onto the previous head. Twelve tests, all green.

Now compose them. A human asks an orchestrator, which delegates to an investigator, which delegates to a compliance agent, which releases a payment. The audit record for the release names group-compliance-agent. It does not name the human. Nothing failed. No test is red.

The bug is that A's contract said "returns a new principal for the next hop" and B's contract said "names the acting agent", and neither of them is wrong. The requirement — every record must be traceable to an accountable human — is a property of the composition, and there was nowhere to write it down.

That is the whole phase, in one example. Three consequences follow:

1. Composition properties need their own home. In this lab it is handle(), RunResult, and the two harnesses. In a real platform it is an integration suite that nobody owns unless you make somebody own it.
2. Component correctness is necessary and not sufficient. A green component suite is a floor.
3. The properties are usually about propagation. Identity, tenant, classification, trace id, region, freshness. Something has to travel the whole path, and the failure is that at one hop it quietly stops.

The single most useful question a reviewer can ask about a distributed design: what travels the whole way, and where could it stop?

2. The request path, drawn

   Teams message
        │
   ┌────▼─────────────┐
   │ 1. CHANNEL       │  open the session; build the Principal ONCE
   └────┬─────────────┘  emits: session
        │
   ┌────▼─────────────┐
   │ 2. CONTROL PLANE │  registered? active? evaluation fresh?
   └────┬─────────────┘  emits: policy_decision      ← EVEN IF DENIED
        │
   ┌────▼─────────────┐
   │ 3. KNOWLEDGE     │  barrier → desk → clearance
   └────┬─────────────┘  emits: retrieval (versions + snapshot)
        │
   ┌────▼─────────────┐
   │ 4. GUARDRAILS ①  │  scan RETRIEVED CONTENT; drop; mark taint
   └────┬─────────────┘  emits: guardrail
        │
   ┌────▼─────────────┐
   │ 5. MODEL         │  classification gate → residency gate → budget gate
   └────┬─────────────┘  emits: inference (six pins), execution_step
        │
   ┌────▼─────────────┐
   │ 6. DELEGATION    │  APPEND to the chain; refuse a cycle
   └────┬─────────────┘  emits: delegation (whole chain)
        │
   ┌────▼─────────────┐
   │ 7. GUARDRAILS ②  │  scan the PROPOSED ACTION; the taint rule
   └────┬─────────────┘  emits: guardrail
        │
   ┌────▼─────────────┐
   │ 8. ACTION GATEWAY│  contract → key → dual control → breaker → execute
   └────┬─────────────┘  emits: approval, action
        │
   ┌────▼─────────────┐
   │ 9. OUTCOME       │  denied · escalated · degraded · completed
   ├────▼─────────────┤
   │10. EVIDENCE      │  complete, or NAME what is missing
   ├────▼─────────────┤
   │11. TELEMETRY     │  latency, cost, hash-chain the artifacts
   └──────────────────┘

Two things to notice before any detail.

The guardrail chain appears twice. Retrieved content and proposed tool arguments are different attacks against different targets, and a single "guardrails" box in an architecture diagram is a sign that the author has drawn the boxes and not the flow.

Every step emits. The evidence pack is not a step at the end that gathers; it is what the path leaves behind. Phase 15 makes this argument; this phase expresses it as a call order.

3. The order is the architecture

Each step's position prevents a specific bug. Move it and the bug returns.

Channel before everything. The Principal is built once, at the edge, from the human's token. Every later hop derives from it. The alternative — each layer reconstructing identity from what it was handed — is how the human disappears at hop three.

Control plane before retrieval. A suspended agent must not cause a retrieval. Not because the retrieval would leak (the barrier filter would still run) but because a denied request that performed work is a denied request that cost money, warmed a cache, and left a trail suggesting the agent was active. Cheap local refusals come first: this is the same argument as putting authn before authz before rate limiting in an ordinary API.

Retrieval before guardrails ①. Obvious, and worth stating: you cannot scan what you have not fetched. The consequence is that the fetch is inside the trust boundary and the content is not.

Guardrails ① before the model. The injected instruction must never enter the prompt. Scanning the model's output instead is the common mistake, and it is a mistake because by then the instruction has already influenced the tokens you are scanning.

Model before delegation. The delegation target is chosen by the run, so it cannot precede the inference. This is also the hop where the chain is most likely to be lost, which is why delegate_to is a method with a refusal in it rather than a replace() at the call site.

Delegation before guardrails ②. The proposed action is scanned after every party that could have influenced it has done so.

Guardrails ② before the gateway. The taint rule is a content judgment: did this action come from something an attacker controls? The gateway's checks are contract judgments: is this tool registered, is the key present, are there two approvers? They are different questions, and running the content judgment first means an obviously poisoned action never consumes an approval workflow.

The gateway's checks run even after an earlier block. This is the ordering decision that surprises people. Short-circuiting on the first denial is the natural implementation and it under-counts defence depth — you cannot report "two independent layers refused" if the second one never evaluated. Only the execution is gated. §5 is the reason.

Evidence after the outcome. The required artifact set depends on what happened: a denied run does not need an inference record; an action above the dual-control threshold needs an approval. Checking before the outcome is known means checking against the wrong contract.

4. A catalogue of seams

A seam is a place where two correct components meet and the property lives in neither. Here are the ones that recur, with the shape of each failure.

The identity seam. Something in the chain replaces instead of appends. Symptom: an audit record naming an agent. Detection: assert the whole chain at the last hop, not the acting agent.

The tenant seam. A cache key, a metric label or a log line is built before the tenant is resolved. Symptom: tenant A's answer served to tenant B, usually under load, usually months later. Detection: make the key a function that takes the resolved principal — you cannot build it early if you cannot build it without the argument.

The classification seam. Data is classified at rest and the classification does not travel with it. Symptom: a confidential document summarized into an internal-classified log line. Detection: the classification is a field on the artifact, computed from the documents actually used.

The residency seam. Every primary path is in-region and a fallback is not. Symptom: nothing — until the day the primary is unavailable, which is also the day nobody is reading the routing logs. Detection: the residency check lives inside the router, not beside it, so every route including the fallback passes through it. This is the lab's _route.

The join-key seam. Every artifact is emitted, and one of them lacks the trace id. Symptom: an evidence pack in which the approval cannot be linked to the action. Detection: emit the key in one place — the emit() closure — so no layer can forget it.

The freshness seam. A component checks that its own data is fresh; nothing checks that two components' data are fresh relative to each other. Symptom: a policy decision made against bundle v4 and an audit record labelled v5.

The ladder seam. Degradation is configured in one place and controls are implemented in another, so nothing prevents a control from becoming a rung. Symptom: none, for months. Detection: the is_control flag, checked at construction (§7).

The retry seam. A step is retried at the transport layer, and the step is not idempotent. Symptom: two payments. Detection: the idempotency key is required by the contract of any non-read tool, so a retry without one cannot be issued.

5. Defence depth: turning a slogan into a number

"Defence in depth" appears in every security architecture document ever written, and almost none of them can tell you how deep. Make it measurable:

Defence depth of a request = the number of distinct layers that acted against it.

Three design decisions inside that sentence, each of which changes what you learn:

Distinct layers, not denials. Three guardrail denials are one layer of defence. If the guardrail service is down, all three are gone together — they are correlated, and correlated defences do not compose. Counting denials flatters you; counting layers does not.

Acting, not halting. A barrier filter that removed a document defended you even though the request continued. Counting only the halting denials means a request stopped by one hard control looks identical to a request that also passed through four controls that each removed something. §6.

Per request, not per system. The number is a property of an attack against a configuration, which means it is measurable in a test suite and can regress in a pull request.

Then the policy, which is the part with teeth:

And the rule about the number 1: a depth of 1 is not a failure, it is a finding that requires an explanation. Some attacks are legitimately stopped by one control and building a second is not worth the complexity. The discipline is that a single point of failure you have named is a risk decision, and one you have not is a surprise.

What this buys you in practice, which is more than it first appears:

• It gives a security reviewer a number to argue with instead of a diagram to nod at.
• It survives refactoring. Merge the guardrail chain into the gateway for tidiness and the number drops from 2 to 1, and a test goes red at the moment the decision is made rather than at the incident.
• It makes the control suite honest, because the harness also runs cases that must not be denied. A suite of only attacks never notices the day the platform starts refusing everything.

6. Acting is not halting

This distinction is invisible inside any single component and unavoidable once you compose.

Consider two denials in the same run:

1. A deal memo behind an information barrier is removed from the retrieval set.
2. A release above the threshold has one approver instead of two.

Both are controls doing their job. They differ completely in what the user experiences: the first produces an answer built from what the user may see, and the second produces nothing. Model them the same way and one of two bad things happens — either every filtered document reports as a failed request (your availability SLI now measures your barrier policy), or you stop counting barrier removals as defence and the depth number goes quiet exactly where it should be loudest.

Hence Denial.blocking:

Two further distinctions in that table are worth naming because they change what an operator does at 3 a.m.:

Degraded vs denied. An open circuit to core banking is a dependency problem. The answer stands; the action is deferred. Reporting it as a denial tells the user their request was refused by policy, which is false, and tells the operator to look at the policy engine, which is the wrong place.

Escalated vs denied. A missing approval is not a refusal — it is a request for a human. The difference matters because "denied" ends a workflow and "escalated" opens one, and because the two have completely different SLIs.

7. The degradation ladder, and the invariant

The ladder is written in daylight, before the incident (Phase 14):

Now the invariant, which is the sentence to remember from this phase:

A control is never on the degradation ladder. Quality may degrade; safety may not.

The failure it prevents is not a bad decision. It is a gradual one, and it goes like this. During a Sev-1 the injection scan is measured at 40 ms per document. Someone adds skip-injection-scan as an emergency rung, with a comment, in a change that is reviewed and approved because the alternative that night is worse. The incident ends. Six months later, the ladder has been reorganized twice, skip-injection-scan is rung two, the comment is gone, and the platform sheds it under ordinary Tuesday load.

Nobody made a bad decision. The system had no way to remember that one rung was different.

So the flag lives on the rung — is_control: bool — and the check runs at construction:

platform = AIPlatform(...)     # LadderError: ['skip-injection-scan'] are controls

Three properties of that placement, all deliberate:

• It fails at start-up, not when the rung is engaged. Discovering it mid-incident is discovering it at the worst possible moment.
• It names the offending rung, so the reviewer of that pull request sees a red test with an answer rather than a red test to debug.
• The default is resolved at call time, not def time. Writing def validate_ladder(ladder = LADDER) freezes the tuple that existed at import, and the point is to check the ladder actually in force.

The corollary is the one people miss: if a control is genuinely too expensive to run under load, the answer is to shed the traffic, not the control. Rung 5 exists for that. Refusing new work is a worse day than skipping the scan, and it is a day you can explain to a regulator.

8. Fail-open, fail-shut, fail-static

The control plane is unreachable. Three choices:

Fail-static is the right answer and it is not free, because it has a parameter and the parameter is a policy decision:

• How stale is too stale? The lab uses 1800 ticks. A real platform states it in minutes and argues about it with Cyber, and the argument is a good one to have written down.
• What does "known-good" mean? The last bundle that validated, not the last one received.
• Who is told? A staleness alarm nobody routes is fail-open with extra steps.

The interview answer is the whole triple: "Fail static — the last known-good bundle, an alarm, and a hard stop past a defined age, because fail-open is a hole and fail-shut makes the control plane's availability the platform's availability." Note that the hard stop is what stops fail-static from degenerating into fail-open over a long outage.

9. Containment: what the taint rule actually buys

Prompt injection is not solved. Say that plainly in an interview and then say what you do about it.

The scanner in this lab is deliberately weak, and the design does not depend on it being good. What the design depends on is this rule:

An action with a side effect, derived from content that came from retrieval, requires a human approval.

Trace the strongest attacker through it. Suppose they fully control a document your platform will retrieve — an emailed invoice, an uploaded PDF, a supplier portal field. Suppose the scanner misses their payload entirely. What do they get?

The attack's ceiling is a human decision. That is not prevention, and it is not nothing: it converts an automated exploit into a social-engineering attempt against a named person who is looking at an evidence record. Meanwhile the honest path is unaffected, because a legitimate release already required approval.

Three things that make the rule work, all of which the composition must get right:

Taint propagates, it does not attach. Any output derived from tainted input is tainted. A summary of a poisoned document is poisoned.

The rule fires on the side-effect class, not the tool name. Adding a tool without a declared side effect must be impossible, which is what the contract registry is for (Phase 10).

Approval is not a checkbox. The human sees the proposed action and the source. An approval workflow that shows only "release PMT-771?" has moved the attack, not stopped it.

10. The output of a run is an evidence pack

The reframing that separates a demo from a bank platform:

A run does not return an answer. It returns an evidence pack that happens to contain an answer.

The pack answers seven questions an examiner will ask, and each is a field somebody had to remember to emit:

Two properties of the pack matter more than its contents.

Generated, not assembled. Every artifact is emitted by the step that had the information, at the moment it had it. Assembling at the end means reconstructing — and reconstruction is where fields go missing, because the assembler asks "what do I know?" instead of "what did I do?".

A missing artifact names itself. evidence_complete = False starts a hunt. missing: ('approval',) ends one. The check is cheap and the difference in an incident is enormous.

And the join key. One trace id, stamped in emit(), on every artifact, without exception — because the pack is only a pack if the pieces link, and the failure mode is exactly one artifact type missing the field.

11. End-to-end budgets

Phase 00 sets budgets per component. The capstone measures the composed path, because a per-component budget that every component meets can still compose into a path that does not. Three reasons:

1. Serial accumulation. Eight components at p95 = 200 ms each is 1.6 s, and each team is inside budget.
2. Tail amplification. The p95 of a serial path is not the sum of the p95s; it is worse, because the chance that at least one hop is slow rises with the number of hops.
3. Retry multiplication. A fallback is a second model call. The budget must hold with the fallback, or the fallback is a plan that only works when it is not needed.

Same for cost. The lab's per-request budget is 50,000 micro-USD, and the frontier route projects 27,000 — which is why the fallback fits and why a tighter budget correctly refuses to route at all rather than silently choosing the cheap model. Note the ordering: the budget check is a routing gate, not a post-hoc report. A cost report tells you what you spent; a routing gate stops you spending it.

The number to state in an interview is not the budget. It is the headroom: "the composed path runs at 12% of the latency budget and 8% of the cost budget on the happy path, and the fallback path at 16%." A budget with unstated headroom is a number somebody wrote down once.

12. Chaos: declare, then inject

The mechanical part of chaos engineering — turning things off — is easy and nearly worthless on its own. The discipline is:

Declare the expected degradation. Then inject the failure. Then compare.

A design you cannot predict is a design you do not understand, and the value is in the cases where your prediction is wrong, which is where you learn something that no amount of reading the code would have told you.

The lab's seven cases, each with its declaration:

Three assertions per case, not one. The outcome matched, the operator alarm fired, and the expected layer denied. All three, because a platform that degrades correctly and silently is a platform whose operators find out from a customer — and because "it still returned 200" is compatible with the control having quietly stopped running.

Notice what the table says about the design. Four of seven failures produce a usable answer. Two of the remaining three are not refusals — one is an escalation, one is a hard stop that a human can resolve. A platform where every dependency failure is a 500 has not been designed, it has been assembled.

13. What the composition still cannot tell you

Say these before an interviewer finds them, because each has a real mitigation and naming it is the signal:

Single failures only. Real incidents are correlated — the provider 429s because the region is degraded, which is also why retrieval is slow. The lab injects one failure at a time. The mitigation is a chaos case that declares several, and a much harder prediction.

In-process and synchronous. No partial failure mid-step, no transport retry, no clock skew, no network partition. This removes the class of failure real distributed systems spend most of their engineering on. The mitigation is that every real seam gets a timeout that is a degradation rather than an exception.

Defence depth measures the attacks you wrote. The harness is exactly as good as the case list, which is why the case list is generated from the OWASP LLM matrix rather than invented (Phase 11) — a new risk row becomes a missing case rather than a gap nobody noticed.

Evidence checks presence, not truth. An artifact can be present and wrong. Presence is mechanically checkable and truth is not; the mitigation is independent validation (Phase 15) and a human panel (Phase 16).

The idempotency store is one instance's dict. The interesting version is shared, and the interesting bug is two instances racing on the same key.

14. Numbers worth carrying

15. Whiteboarding the platform in eight minutes

The single most likely interview task. A rehearsed order:

Minute 1 — the five layers, bottom to top. Infrastructure, model, knowledge, kernel, channels; plus control plane, identity and guardrails as cross-cutting. Say "cross-cutting" out loud — it is the word that separates a platform diagram from a component diagram.

Minutes 2–4 — one request, all the way through. Use the payment investigation. Name each step and what it denies. The denial is the content; anybody can name the boxes.

Minute 5 — the seams. Pick three: identity propagation, fallback residency, the join key. Say why no component test can see them.

Minute 6 — degradation. The ladder, in order, and then the invariant, unprompted: "and no control is on it — quality may degrade, safety may not."

Minute 7 — the evidence pack. The seven questions and where each is answered. Then: "complete, or it names the missing artifact."

Minute 8 — what breaks it. One honest limitation and its mitigation. Correlated failures is the best choice, because it is real and it shows you know what the harness does not cover.

If they interrupt at any point, you are doing well — the interruption is the interview. The order matters because if you are cut off at minute 4 you have already delivered the request path, which is the thing they are actually assessing.

16. Interview questions, answered

"Walk me through a request end to end."

The eleven steps from §2, naming what each denies. Finish with the evidence pack rather than the answer — that ending is the difference between describing an application and describing a platform.

"How do you know your security is layered?"

"Because it is a number. For every case in the attack suite I count how many distinct layers denied, and I require at least two for anything irreversible. A depth of one is not automatically a failure — some attacks are properly stopped by one control — but it requires a written explanation, because a single point of failure you have named is a risk decision and one you have not is a surprise. And the suite includes legitimate requests that must not be denied, because a suite of only attacks never notices the day the platform starts refusing everything."

"An engineer proposes skipping the injection scan under extreme load. What do you say?"

"No, and here is the mechanism rather than the argument: controls carry an is_control flag and the platform refuses to start if one is on the degradation ladder. If a control is too expensive to run at peak, the answer is to shed traffic — rung 5, refuse new work — not to shed the control. Refusing new work is a worse day and it is a day I can explain to the regulator."

"Your control plane is down. What happens?"

"Fail static. Serve the last known-good bundle — the last one that validated, not the last one received — raise a staleness alarm that is actually routed, and hard-stop past a defined age. Not fail-open, which is a hole, and not fail-shut, which makes the control plane's availability the platform's availability. The hard stop is what stops fail-static from becoming fail-open over a long outage."

"A retrieved document contains an injected instruction and your scanner misses it. What happens?"

"They get a read. The containment rule is that a side-effecting action derived from retrieved content requires human approval, so the attack's ceiling is a request in front of a named person who can see the source document. That is containment, not prevention — prompt injection is not solved — and the design does not depend on the scanner being good, which is why I can tell you the scanner is deliberately weak in the reference implementation."

"How do you know an agent did what it says it did?"

"The run's output is an evidence pack, not an answer. Every step emits an artifact at the moment it has the information, every artifact carries the trace id — stamped in one place so no layer can forget it — and the pack is hash-chained. The completeness check is either complete or it names the missing artifact. Generated, not assembled: assembling at the end means reconstructing, and reconstruction is where fields go missing."

"What is the bug you would only find by composing?"

"The fallback's residency. The primary model route is in-region and correct. The fallback is cheaper, just as capable, and in West Europe. Routing is tested and correct; residency is tested and correct; nothing tests the fallback's residency, and the day you find out is the day the primary is unavailable and nobody is reading routing logs. The fix is structural — the residency gate lives inside the router, so every route including the fallback passes through it."

"Where does this design fail?"

"Correlated failures. My chaos suite injects one at a time, and real incidents do not work that way — the provider 429s because the region is degraded, which is also why retrieval is slow. Predicting a composite degradation is much harder than predicting a single one, and I would rather say that than claim the suite covers it."

17. References

Composition and system safety

• Leveson, N. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011. The argument that accidents come from unsafe interactions between components that each satisfy their own requirements — the theoretical core of this phase.
• Perrow, C. Normal Accidents. Princeton, 1999. Interactive complexity and tight coupling.
• Woods, D. & Hollnagel, E. Resilience Engineering. Ashgate, 2006. Graceful extensibility, and the difference between a system that is robust and one that degrades.

Defence in depth, made measurable

• NIST SP 800-53 Rev. 5, Security and Privacy Controls. Control layering and compensating controls.
• MITRE ATT&CK / ATLAS. ATLAS is the adversarial-ML analogue and is the right source for attack cases against an agent platform: https://atlas.mitre.org/
• OWASP Top 10 for LLM Applications, 2025. The matrix the attack suite is generated from: https://genai.owasp.org/

Chaos and operational verification

• Basiri, A. et al. "Chaos Engineering." IEEE Software, 2016. The hypothesis-first framing — declare, then inject.
• Rosenthal, C. & Jones, N. Chaos Engineering: System Resiliency in Practice. O'Reilly, 2020.
• Beyer, B. et al. Site Reliability Engineering, ch. 22 (cascading failures) and The SRE Workbook, ch. 5 (alerting on SLOs). https://sre.google/books/

Prompt injection and containment

• Greshake, K. et al. "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." AISec, 2023. arXiv:2302.12173.
• Willison, S. "The Dual LLM pattern" and the ongoing prompt-injection series: https://simonwillison.net/tags/prompt-injection/
• Debenedetti, E. et al. "AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents." NeurIPS Datasets & Benchmarks, 2024. arXiv:2406.13352.

Evidence, audit and regulation

• CBUAE, Guidance on Outsourcing and Cloud Computing and the Model Management Standard.
• EU AI Act, Art. 12 (record-keeping) and Art. 26 (deployer obligations) — the clearest statutory articulation of "the output of a run is an evidence pack."
• NIST AI RMF 1.0 (Jan 2023), MEASURE and MANAGE functions. https://www.nist.gov/itl/ai-risk-management-framework
• Basel Committee, Principles for the Sound Management of Operational Risk (rev. 2021).

Adjacent practice

• Google, Building Secure and Reliable Systems (2020), ch. 8 (design for least privilege) and ch. 19 (recovery). https://sre.google/books/building-secure-reliable-systems/
• Kleppmann, M. Designing Data-Intensive Applications, ch. 8. The failure modes the in-process version of this lab deliberately excludes.