« Phase 09 · Warmup · Track Overview

Lab 01 — The Control Plane

The problem

There are four hundred agents in production. An examiner asks four questions:

1. Which of them can move money?
2. Who owns the one that made this decision, and when was it last evaluated?
3. Which policy allowed it — and can you show me that version of the policy?
4. If you suspend one right now, how long until it stops acting?

A platform without a control plane answers the first with a spreadsheet, the second with a Slack archaeology exercise, the third with "the rules are in the prompt", and the fourth with a shrug. Those four answers are the difference between a platform and a collection of teams with API keys.

You build the layer that answers all four in milliseconds — and the one property that is harder than it sounds: it must keep answering when it is itself unreachable.

What you build

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

• All 119 tests green against your lab.py.
• No matching rule denies, and the decision still names a policy version.
• Any matching DENY beats every ALLOW, and reversing the rule order changes nothing.
• The decision lists every rule that matched, sorted — not only the winner.
• An unsigned, tampered, structurally invalid or older bundle leaves the previous one live.
• An unconditional allow-everything rule is rejected; an unconditional deny-everything is not.
• A rejected push raises an alarm and does not reset the staleness clock.
• A stale bundle keeps serving until the hard stop; past it, the evaluator refuses.
• An agent with no human owner, or no pinned model version, cannot be registered.
• draft → active is refused; retirement is terminal.
• Discovery hides a tool for five distinct reasons, and returns nothing under a categorical posture failure — but degrades to reads only under a graduated one.
• A stale evaluation blocks a high-impact action and not a read; several failures are all reported.
• A repeated read inside the TTL is served from a lease; a high-impact action never is.
• Activating a new bundle invalidates every live lease.
• A suspension alone is not seen until the lease expires — and the kill switch beats it.
• A safety failure blocks promotion even with min_score=0.0.
• Span durations are never negative, and span ids are identical across two runs.

How this maps to the real stack

Honest limits. The bundle digest covers rule structure, not condition source — because conditions here are Python callables, and a callable cannot be hashed meaningfully. That is exactly why production policy lives in Rego or Cedar text: the text is what you sign, review, diff and attest. The kill switch is in-process, so it says nothing about the genuinely hard part — propagating a revocation to fifty PEP instances across three regions faster than their lease TTL. There is no policy test framework, and a policy set without unit tests is a policy set that will be changed by someone who does not know what rule seven does. And the anomaly score arrives as a number with no provenance; in production, deciding what feeds it is a larger design than everything in this file.

Extensions

1. Swap the callables for Rego. Run a real OPA sidecar, express the same rule set in Rego, and sign the bundle with Cosign. Then hash the source and watch the digest become meaningful.
2. Policy unit tests. Give every rule a fixture set — one request it must allow, one it must deny. Then break a rule and confirm the suite catches it before the bundle ships.
3. A push-based revocation channel. Replace the in-process kill switch with a fan-out to N PEP instances. Measure the p99 propagation, then decide honestly whether your TTL can be raised.
4. Decision-log streaming. Ship every decision to an append-only store with a hash chain (Phase 10), and answer "show me every denial for agent A last Tuesday" without a grep.
5. Break-glass. Add an emergency-override path that is itself a policy decision, requires two approvers, expires in fifteen minutes, and pages someone. Emergency access nobody reviews is permanent access with a story attached.
6. Shadow evaluation. Run a candidate bundle alongside the active one, record where they disagree, and ship only when the disagreements are all intended. This is how you change policy in a bank without an incident.
7. Multi-region staleness. Two distributors, one source, a partition. Which region hard-stops first, and is that the behaviour you want?

Interview / resume bullets

• "Built the platform's control plane: a policy engine with default-deny and deny-overrides evaluating ABAC over subject, action, resource and environment, driven by signed versioned bundles with atomic activation — so every agent decision carries the policy version that produced it."
• "Designed for fail-static: when the control plane is unreachable, the data plane keeps enforcing the last known-good bundle, alarms on staleness, and hard-stops at a stated threshold — which avoided making control-plane availability a multiplier on the platform's."
• "Made capability discovery an authorization decision rather than a lookup, so an agent is never shown a tool it cannot use — removing a whole class of prompt-injection target."
• "Implemented continuous authorization with decision leases and a kill switch that beats the lease TTL, so suspending an agent stops in-flight work in under a second instead of at the next refresh."
• "Wired the evaluation pipeline into the authorization path: an agent whose safety suite is stale or failing cannot act, which turned model quality from a dashboard into a control."