« Phase 09 · Warmup · Track Overview

Principal Deep Dive — The Trade-offs You Own

The deep dive covered how the mechanisms work. This covers the decisions where there is no correct answer, only a defended one.

Table of Contents

• 1. The central tension: freshness against availability
• 2. Choosing a policy language
• 3. Where the PDP runs
• 4. How much policy belongs in policy
• 5. Who authors policy
• 6. Setting the numbers
• 7. Break-glass
• 8. Shipping a policy change without an incident
• 9. Multi-region
• 10. The blended subject, and what it costs
• 11. Migration: from service accounts to a control plane
• 12. What I would not build

1. The central tension: freshness against availability

Every decision in this phase is a position on one axis.

  FRESH                                                            AVAILABLE
    │                                                                    │
  remote PDP        short leases       long leases          replicated,
  per request       + push invalid.    + poll               fully local
    │                    │                  │                     │
  correct now       ~1s stale         ~60s stale          ~5min stale
  fails together    small coupling    no coupling         no coupling

You cannot be at both ends. What you can do is be at different points for different classes of action, and that is the principal-level move:

Stating that table in a design review is the answer. Choosing one point on the axis for everything — in either direction — is the thing that gets pushed back on, and correctly.

2. Choosing a policy language

The decision rule I would defend: if your policy fits in Cedar, use Cedar. The analyzability is worth more in a bank than the expressiveness is, because the question "can any principal ever reach this resource?" is one a regulator asks and only Cedar can answer mechanically.

Take Rego when you need policy over arbitrary JSON in domains Cedar does not model — admission control, Terraform plan validation, CI gates — or when the ecosystem is the point. Many platforms run both, and that is fine as long as the agent authorization path has one.

Hand-rolled deserves one honest mention: it is correct for a first quarter. Twenty rules in code, tested, versioned with the service. What kills it is not expressiveness — it is that policy changes now require a deploy, which means the security team files tickets against the platform team, which means policy stops changing. Migrate when that starts.

3. Where the PDP runs

For an agent's per-step authorization, sidecar is the default. The library is faster and simpler but couples policy upgrades to application releases, which is exactly the coupling you built a control plane to remove.

The remote PDP is not always wrong. It is right for low-frequency, high-value decisions — agent onboarding, a policy simulation, a break-glass approval — where the latency is irrelevant and having exactly one evaluator is valuable. What is wrong is putting it on the per-step path.

The number to bring: at 20 ms per decision and 15 decisions in an agent run, a remote PDP adds 300 ms to every task. That is usually more than the argument survives.

4. How much policy belongs in policy

The failure mode at both extremes is real.

Too little — policy says allow if authenticated, and the actual rules live in application code across nine services. Nobody can answer what the platform permits. This is the common state and it is what a control plane is for.

Too much — policy encodes business logic: fee calculations, workflow routing, which template to use. Now a policy change requires a business analyst, the bundle is 4,000 rules, evaluation is 40 ms, and nobody will approve a change because nobody understands the interactions.

The line I use: policy answers "may this happen?" — never "what should happen?"

The right-hand column is application logic that policy may reference — an obligation names the masking, the application implements it.

5. Who authors policy

Three models, and the second is a trap.

Platform team authors everything. Correct, consistent, and a bottleneck within a quarter. Every product team's onboarding waits on your review queue.

Product teams author their own. Scales beautifully and quietly deletes the control: team A writes a rule that grants team A everything it wants. This is the trap, because it looks like empowerment.

Layered. The one that works:

   BASE (platform + security, signed separately, always applied)
     ├── tenant isolation
     ├── data classification ceilings
     ├── irreversible-action requirements
     └── the panic rules
   TEAM (product teams, reviewed by platform, cannot loosen BASE)
     ├── which tools this agent may call
     ├── value limits below the base ceiling
     └── team-specific conditions

The mechanism that makes the layering real is deny-overrides plus a base layer of denies. A team rule is only ever an allow; the base denies always win. So a team can be given real authorial freedom without being able to escape the envelope — and you can say that in one sentence to a risk officer.

6. Setting the numbers

Do not copy the defaults. Derive them.

Bundle refresh. How quickly must a routine policy change take effect? Usually minutes, so 30 seconds is comfortable. Shorter costs poll traffic; ETags make it nearly free.

Staleness alarm. ~10 missed refreshes — long enough not to page on a blip, short enough that someone is working on it well before the hard stop.

Hard stop. The real question: how long can a policy change go un-applied before the risk is material? Not "how long can we tolerate an outage" — the hard stop causes the outage. For an agent fleet that can move money, 30 minutes. For a read-only fleet, hours. Get this signed off by whoever owns the risk, because it is a deliberate choice to stop the platform.

Lease TTL. This one has a clean derivation: how quickly must a revocation take effect? If the answer is 60 seconds, the TTL is 60 seconds. Then check the load: at 500 decisions/second with a 60-second TTL and reasonable locality, you are evaluating maybe 5% of requests. If that is affordable — and at sub-millisecond evaluation it is — take the shorter TTL. The TTL is a revocation SLA, not a performance knob, and treating it as the latter is how it drifts to five minutes.

Evaluation freshness. By autonomy band: 7 days read-only, 24 hours assisted, 24 hours plus a per-release gate for autonomous.

Anomaly thresholds. Only from measured distributions. A threshold picked from intuition either never fires or fires constantly, and both outcomes end with it disabled.

7. Break-glass

Every regulated platform needs an emergency path, and the design of that path is a genuine test of whether you understand controls.

The wrong version: a flag that disables policy. It will be used routinely within six months, and its use will not be visible.

The right version — break-glass is itself a policy decision:

permit (principal, action, resource)
when {
    context.break_glass_token.valid                    &&
    context.break_glass_token.approvers.size >= 2      &&
    context.break_glass_token.expires_at > context.now &&
    context.break_glass_token.reason != ""
};

Properties that make it defensible: it is a permit rule in the bundle, so it is reviewed and versioned like everything else; two approvers, neither the requester; a hard expiry in the token, typically fifteen minutes, so it cannot become permanent by inattention; a mandatory free-text reason; every action under it tagged in the decision record; and a page, not a log entry, when it is used.

The test of whether yours is right: can you produce a list of every break-glass use last quarter, with who, why, and what they did? If not, you have an off switch with a nicer name.

8. Shipping a policy change without an incident

Policy changes break production in a way code changes do not: the blast radius is every request, immediately, and the failure is a denial, which looks like an outage to everyone affected.

The pipeline that makes it safe:

1. Unit tests per rule. Each rule has at least one request it must allow and one it must deny. A rule with no test is a rule someone will change without knowing what it did.
2. Corpus replay. Keep a sample of real decision inputs (redacted). Replay the candidate bundle against them and diff the outcomes. Every difference is either intended or a bug — there is no third category, and this is the step that catches the empty-facet rule.
3. Shadow evaluation. Run the candidate alongside the active bundle in production, record disagreements, enforce nothing. A day of this on real traffic finds what the corpus missed.
4. Staged activation. One PEP, then one region, then all. Because a bundle activates atomically per PEP, staging is a distribution concern, not a policy one.
5. Fast rollback. Re-activating the previous version must be a one-command operation, and it must be tested — an untested rollback path is discovered during the incident.

Corpus replay is the highest-value step and the one most often skipped. It is also the step that makes the empty-facet allow-everything rule impossible to ship, because it lights up every request in the corpus.

9. Multi-region

Two regions, one policy source, and a partition. The questions:

Which region hard-stops first? The one that lost the source. If both distributors have the same hard stop and the source lives in region A, region B stops and region A does not — an asymmetric outage that will surprise everyone at 3 a.m. Either replicate the source per region, or set the thresholds per region and document why they differ.

Can a region activate a bundle the other has not seen? If activation is independent, yes, and for a window the two regions enforce different policy. Usually acceptable — the window is one refresh interval — but it must be stated, because "the same request was allowed in Abu Dhabi and denied in Frankfurt" is a support ticket you want a ready answer for.

Where does the kill switch land? It must reach every region, which means it cannot be a single-region service. And it must be idempotent, because it will be delivered more than once.

Where do decision records go? Under UAE data residency, decision records about UAE customers stay in the UAE (Phase 15). That constrains the audit store, which constrains the shape of "show me every denial last Tuesday" — a global query over regional stores, or a scatter-gather.

10. The blended subject, and what it costs

The model — one subject carrying both the agent and the user — is right, and it is worth being explicit about the price.

The benefit. Rules can constrain either half independently. deny-irreversible-without-user is three lines and forbids a whole class of autonomous action. deny-above-clearance reads the user's clearance, so an agent cannot exceed the person it works for. Neither is expressible if the subject is just "the agent".

Cost one: the intersection is not obvious. When an agent may call a tool and the user may not, the answer is deny — but somebody must write that rule, and the natural reading of "the payments agent may release payments" does not include it. Make the intersection explicit in the base layer.

Cost two: the empty user. An agent acting autonomously has no user, and every rule that reads subject.user_id must handle it. is_user_scoped exists precisely so that "this rule requires a human" is a positive assertion rather than an accidental null check.

Cost three: chained delegation muddies "the user". When A delegates to B which delegates to C, the user is the original human — but the agent is C and the chain is A→B→C. Policy that needs to know "did a human ask for this?" must read the chain, not just the subject (Phase 08).

Cost four: caching. The lease key includes both halves, so lease locality is worse than a per-agent cache would be. That is correct — a per-agent cache would be a per-agent authorization, which is the bug.

11. Migration: from service accounts to a control plane

The realistic starting state: forty agents, a shared service account each, permissions granted by ticket, an inventory in a spreadsheet.

The order that works, and why:

Phase 1 — inventory, no enforcement. Build the registry. Populate it from what is actually running, not from the spreadsheet. Require a human owner and a pinned model version; you will discover that a third of the agents have neither, and that discovery is itself the business case.

Phase 2 — decision records, no enforcement. Every action logs what a policy decision would have been. This is where you find out how wrong your first rule set is, at zero risk. Expect the first draft to deny 30% of legitimate traffic.

Phase 3 — enforce reads, log writes. Reads are reversible. Turn enforcement on for them and watch the denial rate. Keep writes in shadow.

Phase 4 — enforce writes, with a fast exception path. The exception path is essential: teams will hit rules nobody anticipated, and without a same-day fix route they will route around the control plane entirely — which is much worse than a loose rule.

Phase 5 — retire the service accounts. Only now, and this is the step that pays for the project.

The mistake is starting at Phase 4. A control plane that blocks legitimate work in its first week acquires a reputation it does not recover from, and the exception list becomes permanent.

12. What I would not build

A policy language. Rego and Cedar exist and are better than yours will be. The temptation is strong because both feel like overkill for twenty rules. Twenty rules becomes four hundred.

A general workflow engine in policy. Obligations that trigger obligations, rules that call rules. It always starts as "just one dependency".

An ML-based access decision. Anomaly scoring as one signal into a deterministic rule, yes. A model deciding allow/deny, no — you cannot explain it to an examiner, you cannot test it, and you cannot roll it back to a version.

A global lock for consistency. Someone will propose that all PEPs must be on the same bundle version before any of them serves. It converts your carefully decoupled distribution into a distributed transaction with the availability of its worst member.

Per-request policy compilation. Compile on activation, not on the request path. It looks like elegant dynamism and is a p99 disaster.

My own audit store. Append-only, tamper-evident, queryable, retained seven years, regionally partitioned. Buy it, or use the platform's. This is a much larger problem than it looks and it is not the interesting part of your job.