d12 — Multi-Tenant Control Plane

A fully worked design. The last of the twelve, and the one that ties them together: isolation, fairness, and the fact that a control plane's failures are correlated across every tenant at once.

The interesting property here is that the control plane is the thing that fixes outages, so its own availability requirement is stricter than anything it manages.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Isolation — Noisy Neighbours and Blast Radius
• 7. Deep Dive B: Reconciliation, Not Imperative Orchestration
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We run a platform where thousands of customers each get their own isolated environment — compute, storage, config, the lot. Design the control plane: the thing that provisions, configures, scales and heals all of it. One customer's problems can't become everyone's problems, and it can't be down when we need it most."

"Can't be down when we need it most" is the constraint that makes this different. During a large incident, the control plane is what you use to fix things — scale out, fail over, roll back. So its availability requirement is stricter than the data planes it manages, and it must not share their failure modes.

The second thing to say early: control plane and data plane must be able to fail independently. A data plane that stops serving traffic when the control plane is down has turned a management outage into a customer outage.

1. Requirements and Scope

Clarifying questions asked

"If the control plane is down, do customer workloads keep running?" The fulcrum. Assumed yes, absolutely — data planes run on their last known configuration. This is static stability, and it drives the whole design.

"Are tenants isolated by construction or by policy?" Assumed tiered: large tenants get dedicated infrastructure, small tenants share pooled infrastructure with quotas. Both, because neither alone is economic.

"What's the largest tenant relative to the smallest?" Assumed 10,000×. That ratio is what makes fairness hard — anything that works for uniform tenants fails here.

"How fast must a change apply?" Assumed < 60 s p99 for a normal change; < 10 s for an emergency (a kill switch, a scale-out during an incident).

Functional

1. Provision, update and deprovision tenant environments.
2. Apply configuration changes, with staged rollout and rollback.
3. Enforce per-tenant quotas and limits.
4. Detect and remediate unhealthy resources automatically.
5. Expose tenant state and history to operators and to customers.

Non-functional

Explicitly out of scope

• The data planes themselves — this manages them.
• Billing and metering (consumes our events; separate system).
• Customer-facing UI beyond the API.

2. Scale Numbers

Resources. 10k tenants × ~100 resources = 1M managed resources. If each is reconciled every 30 s, that is 33k reconciliations/s — a serious rate, and the number that decides whether polling is viable at all (§7).

State size. 1M resources × ~10 KB of desired+observed state = 10 GB. Small — again, this is a coordination problem, not a storage problem, which is the recurring shape of control planes.

Change rate. ~1,000 tenant-initiated changes/day plus continuous automated remediation. Human changes are rare; the automated loop dominates, and it is where the risk lives.

The tenant-size ratio is the design constraint. With a 10,000× spread:

Largest tenant:  100,000 resources
Smallest tenant:        10 resources

A FIFO work queue: the large tenant's 100k reconciliations block
                   the small tenant's 10 for the entire cycle.
Round-robin per tenant: the large tenant gets 1/10,000th of capacity
                   and never converges.

Neither naive policy works, which is why §6 is a deep dive rather than "add a queue".

Availability arithmetic. 99.99% is 52 minutes/year. A control plane that shares a database with the data plane inherits the data plane's failure rate, so shared dependencies are the binding constraint — you cannot be more available than your least available dependency, and that determines the architecture more than any code does.

Blast radius. A bad config applied to all 10k tenants is a total outage. With 20 cells of 500 tenants each and staged rollout, the same bug affects 500 tenants for a few minutes — a 20× reduction from a structural choice, not from better testing.

3. API Surface

# Declarative — you state the desired end state, never the steps
PUT    /tenants/{id}/spec        {compute, storage, config, limits}  -> {version}
GET    /tenants/{id}             -> {spec, status, conditions, version}
GET    /tenants/{id}/events      -> [reconciliation history]
DELETE /tenants/{id}                                                  -> {job_id}

# Operations
POST /tenants/{id}/actions/scale      {target}
POST /tenants/{id}/actions/failover   {target_az}
POST /rollouts   {change, strategy: "canary"|"staged"|"emergency"}    -> {rollout_id}
POST /rollouts/{id}/abort

# Introspection
GET /cells                        -> [{cell, tenants, health, capacity}]
GET /rollouts/{id}                -> {stage, affected, health_by_cell}

Four choices worth defending:

• Declarative, not imperative. PUT /spec says what should be true; the system figures out how. An imperative API (POST /add-node) is not idempotent, cannot be retried safely, and has no meaningful notion of "converged". §7 is about why this matters more than it sounds.
• status and conditions separate from spec. Desired state is what the user wrote; observed state is what is true. Conflating them makes "is this converged?" unanswerable, and it is the single most common control-plane API mistake.
• version on every spec, so updates are compare-and-swap. Two operators editing one tenant is routine, not exotic.
• strategy: "emergency" exists and is audited. During an incident you need a documented way to skip staged rollout — otherwise people find an undocumented one, and that is worse.

4. Data Model

TENANT
  spec        desired state, versioned, user-authored
  status      observed state, controller-authored
  conditions  [{type: "Ready"|"Degraded"|"Provisioning", status, reason, since}]
  cell        which cell this tenant lives in — IMMUTABLE without a migration
  tier        dedicated | pooled
  quotas      {cpu, memory, storage, api_rate, ...}

RESOURCE (owned by a tenant)
  spec / status / conditions
  owner_ref -> tenant          cascading delete
  generation                   spec version
  observed_generation          the version the controller last acted on
                               → converged iff observed_generation == generation

CELL
  tenants[], capacity, health, control_plane_instance

generation vs observed_generation is the whole convergence model, and it is worth stating: the spec's generation increments on every user edit; the controller writes observed_generation after acting. Converged iff they are equal. That single comparison answers "is this done?", makes the reconciliation loop idempotent, and gives you a progress metric for free.

Cell membership is immutable without an explicit migration. If tenants could drift between cells, the blast-radius guarantee evaporates — a bug would follow tenants across cells and the containment argument is void.

Conditions rather than a single state enum. A tenant can be simultaneously Ready=true and Degraded=true (serving, but one replica down). A single enum forces you to choose which lie to tell.

5. High-Level Architecture

                       API / operators
                             │
                             ▼
            ┌────────────────────────────────┐
            │  Global API tier (thin)         │  authn, validate, route to cell
            │  routes by tenant → cell        │
            └──────┬───────────────────┬─────┘
                   │                   │
        ┌──────────▼────────┐   ┌──────▼────────────┐
        │      CELL 1       │   │      CELL 20      │   ~500 tenants each
        │ ┌───────────────┐ │   │ ┌───────────────┐ │
        │ │ Control plane │ │   │ │ Control plane │ │   FULLY INDEPENDENT
        │ │  • state store│ │   │ │  • state store│ │   own store, own
        │ │  • controllers│ │   │ │  • controllers│ │   controllers, own
        │ │  • work queue │ │   │ │  • work queue │ │   failure domain
        │ └───────┬───────┘ │   │ └───────┬───────┘ │
        │         │ reconcile │   │         │        │
        │ ┌───────▼───────┐ │   │ ┌───────▼───────┐ │
        │ │  DATA PLANE   │ │   │ │  DATA PLANE   │ │
        │ │  runs on last │ │   │ │  runs on last │ │  ← STATIC STABILITY
        │ │  known config │ │   │ │  known config │ │
        │ └───────────────┘ │   │ └───────────────┘ │
        └───────────────────┘   └───────────────────┘
                   ▲                       ▲
                   └───────────┬───────────┘
                    ┌──────────┴──────────┐
                    │  Global metadata    │  cell assignment ONLY
                    │  (small, d08-style) │  tiny, rarely written
                    └─────────────────────┘

Three structural decisions:

1. Cells are fully independent — own state store, own controllers, own work queue. A cell's failure affects 500 tenants, not 10,000. This is the blast-radius guarantee, and it only holds if the independence is real (no shared database, no shared queue).
2. The global tier is thin — authentication, validation, and routing. It holds no tenant state beyond the cell assignment, so it is cheap to make very available and it cannot become a correlated failure for tenant operations.
3. Data planes are statically stable. They hold their configuration locally and keep serving with the control plane entirely absent. This is the property that makes the whole thing safe, and it is worth restating whenever a design decision threatens it.

The two hard parts — say these at minute 10:

1. Isolation — noisy neighbours in a shared control plane, and the blast radius of the control plane's own bugs.
2. Reconciliation — why the loop must be declarative and level-triggered, and what breaks when it is not.

6. Deep Dive A: Isolation — Noisy Neighbours and Blast Radius

Two different isolation problems that get conflated. Data-plane isolation (one tenant's traffic degrading another's) is well understood — quotas, cgroups, separate pools. Control-plane isolation is the neglected one, and it is where this design earns its keep.

The control-plane noisy neighbour

One tenant runs a script that updates its spec 1,000 times/second.
The reconciliation queue fills with that tenant's work.
Every other tenant's changes wait behind it.
A one-line config change for tenant B takes 40 minutes to apply.

No data-plane quota prevents this — the tenant is using the management API, not the data path. And it is not malicious; a retry loop in someone's CI does it accidentally.

Four layers, and they answer different failure modes:

1. Per-tenant API rate limits (the d03 design). Bounds how fast a tenant can submit work.

2. Fair queueing on reconciliation, not FIFO. This is where the 10,000× size ratio bites, and neither naive policy works. The answer is weighted fair queueing with a floor:

share(tenant) = max(MIN_SHARE, size(tenant) / total_size)

The 100k-resource tenant gets a proportional share — it can converge.
The 10-resource tenant gets at least MIN_SHARE — it is never starved.

Concretely: a deficit round-robin over per-tenant queues, where each tenant's deficit is replenished by its share. Large tenants make progress proportionally; small ones are guaranteed a floor. The floor is the part people omit, and it is what makes the small tenant's experience acceptable.

3. Bounded per-tenant work in flight. A tenant with 100k resources needing reconciliation must not occupy every worker. Cap concurrent reconciliations per tenant — the same per-destination cap as the webhook design, and for exactly the same head-of-line-blocking reason.

4. Coalescing. A tenant updating its spec 1,000 times/second does not need 1,000 reconciliations — it needs one, against the latest spec. Because reconciliation is level-triggered (§7), intermediate states can be skipped entirely.

1,000 updates in 1 s → 1 reconciliation against the final state.

Coalescing is the single biggest win here and it falls out of the declarative model for free. An imperative/event-driven design cannot coalesce, because each event is a distinct instruction that must be applied — which is one of the strongest arguments for §7.

Blast radius: the control plane's own bugs

Tenant isolation does not help when the control plane itself is wrong. A bad controller deployed everywhere breaks everyone simultaneously — and correlated failure across all tenants is the worst outcome a multi-tenant platform has.

Cells are the structural answer. 20 cells of 500 tenants, each with its own control plane instance:

• A control-plane bug is deployed cell by cell, so it affects 500 tenants before it is caught.
• A cell's state store failure affects 500 tenants.
• Cells share no runtime dependency — that is the property that makes the containment real, and it is easy to erode accidentally (a shared cache, a shared queue, a shared metrics pipeline that becomes load-bearing).

Cell assignment is deliberate, not random:

Shuffle sharding is the refinement worth naming. Instead of tenant → one cell, assign each tenant a random subset of shared resources. Then two tenants rarely share their full set, so one tenant's poison affects only the small fraction that overlaps. AWS uses this for exactly this problem, and it gives far better isolation per unit of redundancy than simple sharding.

7. Deep Dive B: Reconciliation, Not Imperative Orchestration

The imperative approach, and why it fails

def scale_tenant(tenant, target):
    current = get_node_count(tenant)
    for _ in range(target - current):
        node = provision_node()          # ← crash here
        register(node)                   # ← or here
        add_to_load_balancer(node)       # ← or here

Every line is a place to fail, leaving the system in a state nobody designed. Retrying re-runs the whole sequence, so you double-provision. And after the crash, nothing knows what the target was — the intent lived only in the in-flight request.

Imperative orchestration means every partial failure is a bespoke recovery problem, and there are exponentially many partial states.

The reconciliation loop

def reconcile(tenant):
    desired = read_spec(tenant)          # what SHOULD be true
    observed = read_actual(tenant)       # what IS true
    for diff in compute_diff(desired, observed):
        apply(diff)                      # one small, idempotent step
    write_status(tenant, observed_generation=desired.generation)

Run continuously. Every invocation moves the world closer to the spec.

Four properties that fall out, and each removes a class of bug:

1. Idempotent. Running it twice is the same as once, because it acts on the difference. So retries are free and require no reasoning.
2. Crash-safe. A crash mid-reconciliation leaves a partial state; the next loop sees the remaining difference and finishes. There is no recovery code, because there is no special case.
3. Self-healing. If something is deleted out-of-band — a node dies, an operator fat-fingers a deletion — the next loop observes the difference and repairs it. Drift correction is not a feature you add; it is the same code path.
4. Coalescing. Rapid spec changes collapse into one reconciliation against the latest, which is the property §6 depends on.

Level-triggered, not edge-triggered

This is the distinction that matters and it is worth naming explicitly.

Events are an optimization for latency, never the source of truth. Use a watch to trigger a reconciliation sooner, but also resync periodically so a missed event is corrected within one resync interval rather than never. A design that trusts events alone is one dropped message away from silent, permanent divergence — and that divergence is invisible until a customer notices.

The loop's own failure modes

1. Hot loops. A resource that cannot converge — an invalid spec, a quota exhausted upstream — reconciles forever, consuming a worker. Fix: exponential backoff per resource, and after N failures mark it Degraded with a reason and stop retrying until the spec changes. A controller that retries an impossible action forever is a self-inflicted denial of service.

2. Fighting controllers. Two controllers with overlapping authority flap a resource between states, forever. Fix: strict ownership — exactly one controller owns each field, enforced by the API. This is a real and confusing production failure, and it is prevented by construction rather than by discipline.

3. Thundering herd on resync. All 1M resources resyncing at once. Fix: jittered resync intervals, so the load is spread rather than periodic spikes.

4. Reconciling against a stale view. The controller reads observed state from a cache that is behind, computes a diff against reality-as-of-a-minute-ago, and "fixes" something that is already fixed — often by creating a duplicate. Fix: read-your-writes on the controller's own actions, and make every create idempotent via a deterministic name derived from the spec.

That last one is subtle and it is the most common real bug in reconciliation systems — the controller creates a resource, does not see it in its cache, and creates it again.

8. Failure and Recovery

Static stability deserves the emphasis. The single most important property is that a data plane holds its configuration locally and keeps serving indefinitely without the control plane. That means:

• Config is pushed to data planes and cached on local disk, never pulled on demand.
• A data plane starting up with an unreachable control plane uses its last known good config, and starts.
• No data-plane request path ever calls the control plane. Not once. That is the invariant, and it must be tested — a dependency added later by an unwitting engineer is invisible until an outage reveals it.

Test it as a first-class scenario: turn the control plane off in a pre-production environment and verify every data plane keeps serving. "We believe it's statically stable" is not the same as having checked.

Deliberately accepted: during a control-plane outage, no changes apply — no scaling, no failover, no new tenants. I accept that because the alternative (a data plane that depends on the control plane to serve) converts a management outage into a customer outage, which is categorically worse. The mitigation is that the control plane is cellular, so an outage is per-cell rather than global.

9. Bottlenecks and Evolution

1. Reconciliation throughput per cell. 50k resources per cell ÷ 30 s = ~1,700 reconciliations/s. Fixes: only reconcile what changed (watch-driven, with periodic resync as the safety net) rather than sweeping everything; shard controllers by resource type so they scale independently.

2. Cell state store write rate. Every reconciliation writes status. Fix: only write when status actually changes — the common case is "nothing changed", and writing anyway multiplies load by the resync rate for no information. This is a one-line check that often cuts store load by an order of magnitude.

3. Global metadata tier. Small and rarely written, but every API request consults it for routing. Fix: cache cell assignments aggressively — they change only on tenant creation or migration, so a long TTL is safe and it removes the tier from the hot path.

4. Cell count growth. 20 cells is manageable; 200 is an operational problem — every deploy becomes 200 staged rollouts, every metric becomes 200 series. Fix: cell groups with hierarchical rollout, and tooling that treats a cell as a unit rather than a special case.

5. Tenant migration between cells. Needed for rebalancing, and it is genuinely hard: state must move, and there is a window where both cells think they own the tenant. Fix: treat it as an explicit, rare, operator-initiated operation with a fence — the same fencing primitive, so the old cell's writes are rejected after the flip. Never automate it; the containment guarantee depends on cell membership being stable.

At 100× tenants (1M): cells become the unit of everything — deployment, on-call, capacity planning, even org structure. The control plane stops being a service and becomes a fleet-management problem, and the interesting work moves to the tooling that manages cells rather than to the controllers.

10. Tradeoffs Explicitly Rejected

Rejected: a single global control plane. Simplest, one deployment, one state store. Rejected because a bug affects 100% of tenants simultaneously — correlated total failure is the worst outcome a multi-tenant platform has. Cells cost operational complexity and buy a 20× blast-radius reduction. Flip condition: below a few hundred tenants, cells are premature and a single well-tested control plane with staged rollout is better.

Rejected: imperative orchestration. More obvious to write and easier to trace. Rejected because every partial failure becomes a bespoke recovery problem, there are exponentially many partial states, and retries are not safe. Reconciliation makes crash-safety and self-healing the same code path as the normal path.

Rejected: edge-triggered (event-driven) reconciliation. Lower latency, less load. Rejected because one dropped event causes permanent silent divergence, and the divergence is invisible until a customer finds it. Events are used as a latency optimization on top of level-triggered resync.

Rejected: dedicated infrastructure for every tenant. Perfect isolation. Rejected on economics — a 10-resource tenant cannot justify a dedicated control plane. Tiering (dedicated for large, pooled with quotas for small) gets most of the isolation at a fraction of the cost. Flip condition: a regulated environment requiring physical isolation makes dedicated the only option, and the economics stop being the deciding factor.

Rejected: data planes pulling config on demand. Simpler (no push infrastructure, always current). Rejected because it destroys static stability — the control plane becomes a hard dependency of every data-plane request, so a control-plane outage becomes a customer outage. This is the single most important rejection in the design.

Rejected: automatic cell rebalancing. Would keep cells evenly loaded. Rejected because a rebalance moves tenants between failure domains — often triggered by load that is itself caused by an incident, so it moves tenants into a problem. Operator-initiated, fenced, and rare.

Rejected: a single state enum per tenant. Simpler API. Rejected because real states overlap — a tenant can be serving and degraded and mid-update. Conditions express that; an enum forces a lie.

The Hostile Critique

C1. "Static stability: data planes run on cached config indefinitely. A tenant is deprovisioned for non-payment while the control plane is down. The data plane keeps serving them on its cached config. How long, and who pays for it?"

C2. "Weighted fair queueing with a floor. A tenant with 100,000 resources and a tenant with 10. Do the arithmetic on how long the large tenant takes to converge after a change, given MIN_SHARE is guaranteeing capacity to 499 other tenants."

C3. "Cells are fully independent, no shared runtime dependencies. Name your metrics pipeline, your deploy system, your image registry, your secret store, and your DNS. Are those per-cell?"

C4. "Level-triggered reconciliation self-heals drift. An operator manually deletes a resource during an incident to stop a runaway process. Your controller helpfully recreates it 30 seconds later. What happens next?"

C5. "You said never automate cell migration. A cell's underlying AZ is being decommissioned by your cloud provider with 30 days' notice. You have 500 tenants to move manually?"

C6. "Emergency rollout skips staged rollout and is audited. It's 3am during a major incident and the emergency path has a bug — nobody has exercised it in eight months. What do you actually have?"

The Revision

R1 — Static stability needs a bounded validity, not indefinite (answers C1)

The critique is right that "indefinitely" is wrong. Serving a deprovisioned tenant is a revenue and compliance problem, and "the control plane was down" is not a defence a business will accept.

Change: cached config carries a TTL and a class.

The reasoning is that the classes have different failure costs. Serving an extra day of compute to a delinquent tenant is a small, recoverable loss. Serving a revoked API key for a day is a security incident. Treating all config identically forces you to pick one of those costs for everything.

Plus revocation gets its own path. Security-critical changes propagate through a separate, simpler channel with its own availability budget — because it must work when the main control plane does not. This is the same reasoning as d08's kill switch.

Cost: three config classes to reason about, and a data plane that can enter read-only mode — which must be tested, or it is theoretical.

R2 — Fair share must be per-change, not per-resource (answers C2)

The critique's arithmetic is damning and I had not done it. With MIN_SHARE guaranteeing capacity to 499 small tenants, the large tenant gets a fraction of the remainder:

1,700 reconciliations/s per cell.
499 small tenants × MIN_SHARE ≈ 1 rec/s each     = 499/s reserved
Large tenant gets                                  ~1,200/s
100,000 resources ÷ 1,200/s                       = 83 seconds

That is actually fine — but only because reconciliation is cheap. If each takes 2 seconds of work rather than being queue-limited, it becomes hours.

Change, three parts:

1. Fair-share on work, not on count. Weight by estimated reconciliation cost, so 100k trivial no-ops do not consume the same budget as 100 expensive provisions.
2. Prioritize by change type, not just by tenant. A tenant's 100k resources rarely all need real work — most reconciliations are no-ops confirming convergence. Split the queues:

   high:   user-initiated changes, remediation of unhealthy resources
   low:    periodic resync of healthy resources
   

   The low queue is best-effort and never blocks the high queue. This is the biggest win and it is nearly free, because it exploits the fact that the vast majority of reconciliations find nothing to do.
3. Cap what one spec change can enqueue. A tenant changing a field that touches 100k resources rolls out progressively, in batches, which is safer as well as fairer — a bad change stops after the first batch.

Cost: a cost model to maintain, and low-priority resync can lag under sustained load. Bounded by alarming on resync age, so silent drift is still detected.

R3 — Name the shared dependencies and bound each one (answers C3)

The critique is correct and the honest answer is that not everything can be per-cell — the claim of "no shared dependencies" was too strong.

Change: enumerate every shared dependency and state its containment.

The rule that makes this tractable: a shared dependency is acceptable iff its failure cannot stop a running data plane. Registry down → cannot deploy new images, existing pods run. Metrics down → blind, still working. Secret store down → cached secrets valid for their TTL.

And the discipline required: this table is only true if it is tested and enforced. Dependency injection at the boundary, plus a test that runs a data plane with every shared dependency blocked. A dependency added later by a well-meaning engineer is invisible until the outage — so it needs a check in CI, not a wiki page.

R4 — Reconciliation must be suspendable (answers C4)

The critique describes a genuinely dangerous behaviour: a controller fighting an operator during an incident, and the operator losing. That is worse than useless — it actively obstructs recovery.

Change: an explicit pause, at multiple granularities.

PUT /tenants/{id}/spec        {..., paused: true}     # this tenant
PUT /resources/{id}           {..., paused: true}     # one resource
POST /cells/{id}/pause                                # everything in a cell

Plus:

1. Deletion is respected, not reverted. A resource deleted out-of-band while paused stays deleted; when unpaused, the controller reports the divergence as a condition and waits for an explicit decision rather than silently recreating.
2. kubectl scale --replicas=0-style intent. The operator's action should be expressible as an intent change — "scale to zero" is a spec edit, which the controller then honours. Making the operator's intent expressible in the spec is better than pausing the controller, and it should be the first-choice path.
3. Pause is loud — a prominent condition, a metric, and an alarm if a pause outlives a threshold, because a forgotten pause is a resource that silently stops self-healing.

Cost: paused resources do not self-heal, which is exactly what was asked for, and a forgotten pause is a real hazard. Mitigated by the alarm and by an optional auto-expiry on the pause.

R5 — Migration must be automatable, just not automatic (answers C5)

The critique catches an overcorrection. "Never automate" was about triggers, not about tooling — and a 30-day AZ decommission with 500 tenants makes that distinction matter.

Change: cell migration is a first-class, tested, tooled operation that is operator-initiated rather than automatically triggered.

POST /migrations  {from_cell, to_cell, tenants: [...], rate: "10/hour"}
  → per-tenant: replicate state → verify → FENCE the old cell → flip routing → verify → release

• Fenced, using the d11 primitive: the source cell's writes for a migrated tenant are rejected after the flip, so the both-cells-think-they-own-it window is safe.
• Rate-limited and resumable, so 500 tenants move over days without a big-bang event.
• Per-tenant verification before releasing the source, and automatic rollback on failure.
• Exercised regularly — migrate a few tenants monthly as a drill, so the path is not first-used during the decommission.

The distinction that matters: automatic migration (triggered by load) is rejected, because load spikes are often symptoms of incidents and moving tenants into or out of a problem makes it worse. Automated migration (an operator initiates, tooling executes reliably) is essential, and conflating the two was the error.

Cost: real engineering for an operation used a few times a year. Justified: the alternative is 500 manual migrations under a deadline, which is where mistakes happen.

R6 — The emergency path must be the normal path with fewer gates (answers C6)

The critique names the thing that actually goes wrong at 3am, and it is the strongest of the six: an untested emergency path is not a safety mechanism, it is a second incident waiting for the first one.

Change: the emergency path is not a separate code path. It is the same rollout mechanism with different parameters.

normal:     stages = [1 cell, 3 cells, all], bake = 10 min each, auto-rollback on
emergency:  stages = [1 cell, all],          bake = 30 s,        auto-rollback on

Same code, same tests, same telemetry — only the timing and stage count differ. So the emergency path is exercised by every normal rollout, because it is the normal rollout.

Plus:

1. Never skip the canary entirely. Even at 30 seconds, one cell first catches the change that is wrong everywhere. The most dangerous emergency change is the one that makes the incident worse, and that is precisely when people skip verification.
2. Auto-rollback stays enabled in emergency mode. Turning off the safety net during an emergency is exactly backwards, and it is a common instinct.
3. Drill it monthly. A game day that uses the emergency path on a real (non-critical) change. "Exercised in the last 30 days" should be a dashboard item, and if it is not green the path should be assumed broken.
4. A pre-authorized break-glass with two-person approval and full audit — because there will be a case where even 30 seconds is too slow, and an undocumented workaround is worse than a documented one.

Cost: emergency changes take ~60 s instead of ~10. Worth it: the failure mode being prevented is "the emergency fix made it worse and we had no way back", which is the worst outcome available during an incident.

References

• ../WARMUP.md#410-load-control — bulkheads, cells, fair queueing
• d03-rate-limiter.md — the per-tenant API limits in §6
• d05-load-shedding.md — priority classes and reserved floors, the same pattern applied to queueing
• d08-multi-region-metadata.md — the config-distribution and static-stability model
• d11-lock-service.md — the fencing primitive used in R5
• Amazon Builders' Library. Workload isolation using shuffle-sharding. https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/
• Amazon Builders' Library. Static stability using Availability Zones. https://aws.amazon.com/builders-library/static-stability-using-availability-zones/ — the §8 principle, stated properly
• Amazon Builders' Library. Avoiding fallback in distributed systems. — why the emergency path must be the normal path
• Hightower, Burns, Beda. Kubernetes: Up and Running — the reconciliation model in §7
• Brewer, E. Kubernetes and the path to cloud native. SoCC 2015 — level-triggered vs edge-triggered
• Google SRE Book, Ch. 22 — Addressing Cascading Failures
• Hunt et al. ZooKeeper. USENIX ATC 2010 — coordination primitives underlying cell membership