« Phase 09 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a policy request
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy the evaluator, build the model. A policy engine is a general-purpose piece of infrastructure with a research literature behind it. What agents exist, what "fit to act" means, and what must interrupt a running task are specific to your bank and nobody will ship them for you.

And a caution on the registry: it is tempting to make it a table in the platform's database. It will become the system of record for "what AI exists at this bank", which is a question Internal Audit, the CISO and the regulator all ask. Give it an API, a lifecycle and an owner.

2. A decision framework for a policy request

A team wants an agent to be allowed to do something. Seven questions, in order:

1. Is there a human in the loop? If yes, the rule can read the user's entitlements and most of the risk evaporates. If no, everything below tightens.
2. Is the action reversible? Reversible, non-idempotent, or irreversible. This single answer determines whether the decision may be leased, whether dual control applies, and whether it belongs in an autonomous band at all.
3. What is the worst case if this rule is wrong in the permissive direction? Not "what is it for" — what does it cost when it fires on a request nobody imagined.
4. What attributes does the rule need? If any is not already in the decision input, you are also asking for a new data feed with its own freshness problem. That is usually the real cost.
5. Does it belong in the base layer or the team layer? If it loosens anything in the base layer, the answer is no, and the conversation is with the security team instead.
6. What is the blast radius if it is over-broad? Which is the corpus-replay question, and it should be answered with data rather than reasoning.
7. How will we know it fired? A rule nobody monitors is a rule that will be wrong silently.

The most common outcome of running this properly is that the request narrows: what was "the agent needs to write to CRM" becomes "the agent needs to append a note to a case it already read, in its own tenant, under 4 KB". That narrowing is the work.

3. Review red flags

In a design document

• The PDP is a remote service called per request.
• No answer to "what happens when the control plane is down", or the answer is "we fail closed".
• Policy in application code, with the control plane described as "the audit layer".
• tools/list described as a database query.
• A permissions model that is a list of roles.
• Revocation described as "immediate", with no push channel.
• Decision records without a policy version.
• No hard stop, or a hard stop nobody has signed off.
• Posture as a boolean.
• An anomaly score with no stated inputs or false-positive rate.
• Product teams authoring policy with no base layer.
• A break-glass path that is a flag.
• Evaluation results going only to a dashboard.
• "We'll add the audit trail later."
• Agent records with no human owner, or an owner field that is a team name.
• An unpinned model version.

In code

# Red flag: no default deny
for rule in rules:
    if rule.matches(request) and rule.effect is ALLOW:
        return ALLOW                       # ...and if nothing matches? Undefined.

# Red flag: short-circuit on the first allow
    if rule.matches(request):
        return rule.effect                 # order now determines the answer

# Red flag: any instead of all
if any([action_ok, tenant_ok, class_ok]):  # a rule that fires far too often

# Red flag: the wildcard that eats a namespace
if action.startswith(pattern.rstrip(".*")):   # "pay.*" now matches "payments.release"

# Red flag: staleness from the last attempt
self._last_seen = now()                    # set even on a rejected bundle

# Red flag: caching a deny
self._leases[key] = Lease(decision, ...)   # unconditional — now a fix waits a TTL

# Red flag: no version check on the lease
if lease and lease.is_live(tick): return lease.decision   # new bundle ignored

# Red flag: discovery approximating policy
return [t for t in tools if t.classification <= agent.max_classification]

# Red flag: an unrecognized obligation ignored
for ob in decision.obligations:
    if ob in HANDLERS: HANDLERS[ob]()      # unknown ones silently dropped

# Red flag: signature compared with ==
if computed == provided: ...               # timing oracle

# Red flag: posture as a boolean
if not agent.healthy: return DENY          # no distinction, so it will be tuned off

In an incident review

• "We didn't know which policy version was live" → no status reporting.
• "The rule had been wrong for three weeks" → nobody monitors allow/deny rates per rule.
• "It worked in staging" → staging has a different bundle, or none.
• "We suspended it but it kept going" → no kill switch; say the number.
• "Everything was denied after the deploy" → no corpus replay.
• "Nobody noticed the bundle was stale" → alarm on last attempt, not last activation.

4. Production war stories

The empty facet. A rule intended as "allow the payments team to read payments" shipped with the actions tuple commented out during debugging. It matched everything. Nothing broke — the platform got better, tickets stopped arriving — and it ran for eleven days until a quarterly review read the bundle line by line. The remediation was corpus replay in CI, which would have caught it in ninety seconds.

The fail-shut cascade. The policy service was made a synchronous dependency because "it's only 5 ms". A bad deploy took it down for eight minutes. Every agent in the bank stopped, including the read-only ones, including the one whose job was to summarize the incident channel. The postmortem action was fail-static; the argument that had lost six months earlier was the same one.

The staleness alarm that never fired. The distributor updated its "last check" timestamp on every poll, including polls that returned 403 after a credential rotation. The dashboard was green for nine days on a bundle from the previous sprint. Discovered when someone asked why an emergency restriction added on the Monday was not being enforced.

Three and a half minutes. An agent was suspended during an investigation. It completed two more tool calls afterwards — a live lease, then a poll interval. Neither call was harmful. The finding was not about the calls; it was that nobody had been able to state the number in advance.

The cached deny. A misconfigured entitlement denied a trading desk for ten minutes. The configuration was fixed in ninety seconds. The remaining eight and a half minutes were the decision cache, and that is what the desk head escalated about — not the outage, the fact that the fix appeared not to work.

The invisible tool. A tool was removed from an agent's permitted list but left in the discovery response, because discovery read a config file and authorization read the registry. The agent kept planning around it and failing at call time, in a retry loop, for two days. The token bill was the first symptom.

Policy in the prompt. A team encoded authorization in the system prompt: "you must not access accounts outside the customer's own." It worked in testing. A prompt injection in a PDF removed it in one sentence. The lesson is not that prompts are weak; it is that a control the model can be talked out of is not a control.

The break-glass that never closed. An emergency override added during a Friday incident, with no expiry. Found fourteen months later during an access review, still enabled, used routinely by two teams who had learned it made their jobs easier.

Everything denied. A refactor changed a rule's combining semantics from all-facets-conjunctive to any-facet. Every DENY rule became far broader. Production denied 94% of requests within thirty seconds of activation. Rollback took four minutes, because rolling back a bundle had never been tested.

The anomaly score nobody trusted. Fired eleven times in the first week, all false positives, all requiring an operator to unblock an agent. Disabled on day nine "temporarily". Still disabled at the next audit, where its absence was a finding.

The agent with no owner. Built during a hackathon, promoted to production "for a demo", running for two years with write access to CRM. The owner field said ai-platform-team, which had been reorganized twice. Nobody could say what it did or whether it could be turned off.

5. The interview signal

Signal 1 — you lead with the availability argument. Not "we use OPA" but "the PDP cannot be on the synchronous request path, because its availability multiplies into ours — so bundles are pushed down and decisions are local." That sentence separates people who have run a control plane from people who have configured one.

Signal 2 — fail-static, unprompted. Naming the third posture, and explaining why fail-open and fail-shut are both wrong, is the single highest-value thing you can say in this phase. Most candidates offer two options.

Signal 3 — the hard stop, with a number and an owner. "Thirty minutes, signed off by the head of operational risk, because at that point a missing emergency rule is worse than an outage." Naming who signs it off is the staff-level detail.

Signal 4 — "discovery is authorization". With the injection argument: if the tool was never in the list, the injection has nothing to reference.

Signal 5 — you decompose revocation latency. "TTL plus refresh plus in-flight — about three and a half minutes worst case, which is fine for reads and not for payments, so there is a kill switch." Claiming instant revocation is the anti-signal.

Signal 6 — the kill switch has two halves. Dropping leases makes it fast; the revoked set stops it racing the traffic. Very few candidates get to the race.

Signal 7 — posture is graduated. And the reason: a control that downs the fleet when an eval job runs late is a control operators will disable. This shows you have operated something, not just designed it.

Signal 8 — you volunteer corpus replay. When asked how you ship a policy change safely. It is the step that catches the failure mode nobody else names — the over-broad rule that breaks nothing.

Signal 9 — the layered authorship model. Base layer of denies, team layer of allows, deny-overrides making the layering structural rather than procedural. It answers "how do you scale this to forty teams" in one diagram.

Anti-signals:

• A remote PDP on the per-request path, unremarked.
• "We fail closed" with no discussion.
• Roles instead of attributes.
• Revocation described as instant.
• Policy in the system prompt.
• Decision records without a version.
• Evaluation results that only reach a dashboard.
• A break-glass flag.
• Posture as a boolean.
• Discovery as a database query.

The question to ask them: "When you suspend an agent, how long until it stops acting — and how do you know?" The answer separates the platforms with a control plane from the platforms with a permissions table. And the follow-up, "how would you prove that number to an examiner?", separates the ones that have been examined.

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Show them two decision records. {"allowed": false} and the full record with the rule name, every matched rule, and the policy version. Ask which they would want during an incident at 3 a.m. Ten seconds, and it reframes the decision record from bookkeeping to the product.
2. Have them break their own rule set. Give them a bundle and ask them to add one rule that makes it allow everything, without it being obvious in review. Most people find the empty-facet ALLOW within a minute — and never ship one again.
3. Run the revocation clock on a whiteboard. "You clicked suspend at 14:00:00. Walk me forward second by second." The moment someone says "wait, it can still create a new lease" without prompting, they understand continuous authorization.

And the framing for the platform team: this is the phase where the cost of deferring is non-linear. Every agent built against an unfiltered tool list, a service account, or a decision-free path has to be re-plumbed. Building the registry and the decision record in the first month costs two sprints. Retrofitting them across forty agents costs a quarter, and it happens under audit pressure with an examiner waiting.

The argument that gets it funded is not policy hygiene. It is: "an examiner will ask which of our four hundred agents can move money, who owns them, and which policy allowed the last one that did. Today the answer is a spreadsheet and a grep. That is a finding, and the remediation is this."