« Phase 17 · Warmup · Lab 01

Principal Deep Dive — The Trade-offs You Own

The decisions in this document have no correct answer. They have defensible answers, and the job is to make one, write down why, and be able to revisit it when the inputs change.

Table of Contents

• 1. Where composition should live
• 2. What defence depth is worth, and what it costs
• 3. The staleness parameter
• 4. Autonomy: the decision the whole platform exists to make
• 5. Containment vs prevention, argued honestly
• 6. Escalation as a capacity decision
• 7. The ladder is a product decision wearing an SRE costume
• 8. What to measure when the answer is nondeterministic
• 9. Build, buy, or wait
• 10. Concentration risk, and the exit you will never take
• 11. Cost of controls, stated plainly
• 12. The migration nobody plans for
• 13. What you would cut
• 14. References

1. Where composition should live

The lab puts the whole path in one handle(). That is a teaching decision. In production you choose between three shapes, and the choice determines which failures are possible.

The honest assessment: orchestrator for the controls that must be ordered, library for the ones that must be everywhere, mesh for the ones that must apply to traffic you did not write. Most real platforms end up with all three and the failure is that nobody can say which control lives where. The artifact that fixes it is a one-page table — control, location, enforced-by, tested-by — and it belongs in the ORR (Phase 16).

The trap in the orchestrator shape is worth naming: it becomes the platform team's queue. Every new agent needs a change in handle(), and the team that owns handle() becomes the bottleneck for adoption — which is the metric the JD names first. The mitigation is that handle() must be policy-driven, not case-driven: new agents arrive as registry entries, not as branches.

2. What defence depth is worth, and what it costs

Requiring ≥ 2 independent layers for irreversible actions is a real constraint with a real price.

The price. Every additional layer adds latency, a failure mode, an operational surface, and — most expensively — a false positive surface. Two independent controls at 99.9% specificity refuse 0.2% of legitimate requests rather than 0.1%. At 10,000 actions a day that is ten extra people whose correct work was refused, and those ten people talk to each other.

The value. Not "more secure" in the abstract. Specifically: it means no single control failure — a bad deploy of the scanner, an expired policy bundle, a misconfigured registry — converts into an irreversible action. That is a conditional claim and it is the right one to make to a regulator, because it is verifiable.

Where the argument actually goes. The interesting question is not 2 vs 1; it is independence. Two controls that both read the same policy bundle are one control. Two controls in the same process that both die when it OOMs are one control. When you say "defence depth 2", be ready to say what makes the two layers independent — different data, different failure domain, different deploy cadence. If you cannot, the number is theatre.

My position: require 2 for irreversible, require an explanation for 1 elsewhere, and audit the independence claim annually. The explanation requirement matters more than the number, because it converts an unnoticed single point of failure into a named risk decision that somebody signed.

3. The staleness parameter

Fail-static needs a number: how old may the policy bundle be before the platform stops serving?

The trade-off is stark and neither end is safe:

   0 minutes ────────────────────────────────────────► ∞
   fail-shut                                        fail-open
   (control plane availability                     (a revoked agent
    becomes platform availability)                  keeps working)

Inputs that should move the number:

That last row is the design insight and it is worth more than the number. If the only way to stop an agent is a policy bundle refresh, staleness tolerance must be tight, and you have coupled your availability to your control plane. Add an out-of-band kill switch — a separate, dumber, more available channel whose only job is "stop" — and you can serve a stale bundle for an hour with a clear conscience, because the emergency path does not depend on it.

State it as: "30 minutes stale with an alarm at 5, and an out-of-band kill switch that does not go through the bundle at all." The pairing is the answer; the number alone is not.

4. Autonomy: the decision the whole platform exists to make

Every mechanism in this track exists so that one question can be answered responsibly: how much may an agent do without a human?

The bands, and what each actually costs:

Three things a principal should say about this that a senior engineer usually does not:

Approve-each does not scale, and its failure mode is not "slow". It is rubber-stamping. A human approving 200 actions a day approves the 201st without reading it, and you have autonomous operation with an audit trail that falsely claims human review. That is worse than autonomy, because it is autonomy you have stopped watching. If you cannot staff the review, do not choose the band.

The threshold in approve-exceptions is where the whole risk concentrates. Set it at 100,000 USD and an attacker sends 99,999. Mitigations: velocity limits (N actions per counterparty per day), aggregate limits per agent per day, and a random sample of below-threshold actions routed to review anyway. That last one is the cheapest and the most underused — it makes the threshold probabilistic.

Autonomy is earned per action class, not per agent. The same agent may be autonomous for payments.lookup, approve-exceptions for crm.append_note, and approve-each for payments.release. Granting a band to an agent as a whole is the coarse decision that people default to and it forces you to price every tool at the risk of the worst one.

5. Containment vs prevention, argued honestly

You will be asked "have you solved prompt injection?" The answer is no, and the follow-up is what separates candidates.

Why prevention fails, structurally. The model has one channel. Instructions and data arrive as the same tokens. Every proposed fix — delimiters, instruction hierarchies, system-prompt priority, detection classifiers — is a heuristic operating on a channel that cannot in principle distinguish the two. Improvements are real and the asymptote is not 100%, and a control whose asymptote is not 100% cannot be the only thing between an attacker and an irreversible action.

So the architecture assumes the scanner fails. The taint rule bounds the attacker's ceiling to a human decision. That is a structural argument rather than a statistical one, which is why it survives the next model, the next jailbreak and the next scanner.

What it costs. Every side-effecting action derived from retrieved content needs a human. If your product's value proposition is automating those actions, the containment rule caps your product. This is a real business constraint and it is where the two-in-a-box conversation happens (Phase 16): the product owner wants the automation, the engineer sees the exposure, and the honest resolution is usually to narrow the action rather than to relax the rule — a release capped at 10,000 USD to a counterparty on an established whitelist can be autonomous, because its blast radius is bounded by something other than the model's judgment.

The dual-LLM pattern is the strongest structural alternative: a privileged model that never sees untrusted content, and a quarantined model that does but cannot call tools. It works, it costs a second inference and a serialization boundary, and it is worth knowing by name because it is what you would build if the containment rule capped you somewhere unacceptable.

6. Escalation as a capacity decision

Every control that escalates creates human work, and human work has a queue, a latency and a cost. This is the operational consequence that architecture reviews consistently miss.

Do the arithmetic before you set a threshold:

    escalations/day  =  actions/day  ×  P(escalation)
    reviewer minutes =  escalations/day  ×  minutes/review
    reviewers needed =  reviewer minutes / (productive minutes per shift)

At 10,000 actions/day, 2% escalation and 4 minutes per review, that is 800 minutes — roughly two full time reviewers, at a specific salary, in a specific timezone, with a specific queue SLA. If those people do not exist, the escalation is not a control. It is a queue that grows until somebody approves in bulk, and bulk approval is the rubber-stamp failure of §4 arriving through a different door.

Two consequences a principal should own:

The escalation rate is an SLI. Track it, alarm on it, and treat a rise as a platform problem rather than a compliance success. A scanner regression shows up here before it shows up anywhere else.

Review quality decays with queue depth. If the queue is deep, reviews get faster and worse. So the queue depth is itself a safety metric, which is a genuinely non-obvious thing to instrument and one of the more impressive things to mention in an interview.

7. The ladder is a product decision wearing an SRE costume

The ladder looks like an engineering artifact. Every rung is a decision about what users lose:

Rung 3 is the one that deserves an argument. A stale answer delivered confidently is, for some questions, worse than no answer — "is this counterparty sanctioned?" answered from a six-hour-old cache is not a degraded answer, it is a wrong one with a compliance consequence. Which means the ladder may need to be per query class: cache-only is fine for "why was this held", and for sanctions status the correct rung is refuse.

That is a product decision, it must be made with the product owner, and it must be made before the incident. Which is the entire argument for writing the ladder in daylight.

The second product decision hiding in the ladder is user_visible. Rung 1 is invisible; rungs 2–5 are not. Telling users you are degraded costs trust in the moment and buys it over a year — and not telling them, when they can tell from the answer quality, costs both.

8. What to measure when the answer is nondeterministic

Classic SRE availability does not fit. The composed platform gives you better options:

time-to-revoke is the one nobody instruments and the one a regulator will eventually ask about. It is measurable — revoke a test agent in production and time it — and the number is usually much worse than the team's estimate, because it is the sum of bundle propagation, cache TTL and in-flight requests.

9. Build, buy, or wait

For each layer, the defensible position in 2026:

The rule underneath: buy the mechanism, build the policy. A vendor can evaluate a policy faster and more reliably than you can; no vendor knows that payments.release above 100,000 USD needs two approvers neither of whom is in the delegation chain.

The thing to wait on in 2026: agent-to-agent identity standards. There is real movement and no convergence. Build the seam — the chain, the verification point — and keep the wire format swappable, because you will swap it.

10. Concentration risk, and the exit you will never take

CBUAE and every other prudential regulator will ask what happens if your primary model provider becomes unavailable — commercially, technically, or geopolitically. There are three honest answers and only one of them is credible.

1. "We have a second provider configured." Credible only if you route production traffic to it. A configured-but-unused fallback is untested code on your most critical path.
2. "We could migrate in N weeks." Credible only if you have done it, at least in staging, with the eval suite. Otherwise N is a guess, and prompts do not transfer between model families cleanly.
3. "We accept the risk, here is the impact, here is who signed." Always credible. Frequently the right answer.

The uncomfortable truth is that (3) is usually correct and teams reach for (1) because it sounds better. A principal should be able to say: "We are concentrated on one provider for the frontier tier. The exit is 6–8 weeks including eval re-baselining. We run 5% of traffic on the secondary continuously so the path is warm and the evals are current. The residual risk is accepted by the Model Risk Committee and reviewed quarterly." That is a real answer; "we have a fallback" is not.

The 5% figure is the part worth arguing for internally. It costs real money to route traffic to a model you do not prefer, and it is the only thing that converts a claimed exit into a tested one.

11. Cost of controls, stated plainly

Somebody will eventually ask what the controls cost. Have the numbers.

Two observations that make this a principal-level answer rather than a table.

The cheapest controls in latency are the most expensive in humans. The taint rule costs nothing to evaluate and creates the escalation queue of §6. Latency is the wrong axis for pricing a control.

Two of the rows refuse nothing. Evidence emission and the hash chain are not controls; they are witnesses. Cutting them saves 7 ms and costs you the ability to answer any question after the fact. When a cost-reduction exercise comes for the platform, these are the first things proposed and the last things you should give up.

12. The migration nobody plans for

The platform will be built incrementally, which means there will be a period — usually eighteen months — where some agents run inside the composition and some do not. The ones that do not are the early pilots, the ones with the most business attachment, and the ones nobody wants to break.

This is the highest-risk state in the whole programme and it does not appear on any architecture diagram. Three things that help:

Make the composition the only path to the tools, not the only path to the model. Teams will find a way to call a model directly; that is survivable. A direct path to payments.release is not. Enforce at the action boundary, where enforcement is cheap and evasion is visible.

Publish the ratio. "63% of agent actions flow through the platform" is a number that moves people, and it is the number the CTTO's office will ask for. It also stops "we're migrating" from being a permanent state.

Give the stragglers a date and a reason. The reason should be a capability they want, not a policy they must — the platform's adoption argument is that it is easier than not using it. If the only argument is compliance, adoption stalls at exactly the teams with the most leverage.

13. What you would cut

The most revealing interview question in this phase is: "You have half the time. What do you cut?"

Cut first, and you will not regret it: the knowledge graph, the semantic cache, the delegation layer, multi-provider routing, the reranker. Every one of them is a capability, and a platform with fewer capabilities and intact controls is a platform.

Never cut: the action gateway's contract validation, the idempotency key requirement, dual control on irreversible actions, the taint rule, the evidence pack, default-deny admission. Those are not features; each one is the difference between a bad day and a regulatory event.

The unobvious one: do not cut the defence-depth harness, even though it is a test. Cutting it does not remove a control, it removes your ability to notice when a control disappears — and controls disappear silently, one reasonable refactor at a time. A harness that cost two days protects mechanisms that cost two quarters.

The general form: cut capabilities, keep controls, and keep the thing that tells you the controls are still there.

14. References

• Leveson, N. Engineering a Safer World. MIT Press, 2011.
• Willison, S. "The Dual LLM pattern for building AI assistants that can resist prompt injection." 2023. https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
• Greshake, K. et al. "Not What You've Signed Up For." AISec 2023, arXiv:2302.12173.
• Basel Committee on Banking Supervision, Principles for Operational Resilience, 2021.
• CBUAE, Guidance on Outsourcing and the Model Management Standard.
• EU AI Act, Art. 9 (risk management), Art. 12 (record-keeping), Art. 14 (human oversight).
• Beyer, B. et al. The Site Reliability Workbook, ch. 5 and 9. https://sre.google/books/
• Rosenthal, C. & Jones, N. Chaos Engineering. O'Reilly, 2020.
• FS-ISAC, Adversarial AI Risk in Financial Services, 2024.