m03 — GPU Cluster Scheduler (Training and Inference on One Fleet)

A fully worked design. One pool of GPUs, two workloads that want opposite things: training is gang-scheduled, throughput-critical, and checkpointable; inference is elastic, latency-critical, and not. Sharing them is where most of the money is, and where most of the failure is.

The number that decides the design: at 50% cluster occupancy, the expected number of fully free 8-GPU nodes is 0.5. A half-empty cluster cannot schedule a single tensor-parallel job. Fragmentation, not capacity, is the constraint — and almost nobody says so.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Topology Is Not a Preference, It Is a Constraint
• 7. Deep Dive B: Preemption, Gang Scheduling, and Who Yields
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We have a thousand H100s. Research wants them for training runs, product needs them for inference, and right now we split the cluster in half and both sides complain. Design a scheduler that shares them."

"Both sides complain" is the requirement, restated as a symptom. A static split guarantees both complaints simultaneously: research queues while inference GPUs idle overnight, and inference sheds traffic while a training run finishes. Each side's peak is the other's trough — which is exactly the case where sharing wins, and exactly the case people give up on because the failure mode of naive sharing is worse than the failure mode of the split.

The thing to establish in the first two minutes: a GPU is not a fungible unit of capacity.

• Eight GPUs on one node are connected by NVLink at 900 GB/s.
• Eight GPUs spread over eight nodes are connected by InfiniBand at 50 GB/s.
• That is an 18× difference, and for tensor parallelism it is the difference between a 3% and a 52% slowdown (§6).

So "we have 200 free GPUs" is not an answer to "can you run this job". The scheduler's real currency is topology-connected sets, not counts — and stating that early reframes the whole question.

1. Requirements and Scope

Clarifying questions asked

"What is the mix — how much training, how much inference?" Assumed: roughly 60% training / 40% inference by GPU-hours, but with opposite time profiles. Inference peaks during business hours in the primary market; training is 24/7 with a research cadence that spikes before deadlines. Anti-correlated demand is the premise that makes sharing worth the complexity, and if the interviewer says the demands are correlated, much of this design becomes unjustifiable — worth saying, because it shows the design has a stated precondition rather than assumed universality.

"Can training be preempted?" Assumed yes, with a checkpoint. This is the asymmetry the entire design rests on: training has a persistent, resumable state; an in-flight inference request does not (m01 §8 — the KV cache is the state and it dies with the replica). So the direction of preemption is decided by physics, not by policy: inference preempts training, never the reverse.

"What's the largest job?" Assumed 256 GPUs for a pretraining run, 8 for the largest inference replica. The ratio matters — a scheduler that handles 8-GPU allocations well can fail completely at 256, because gang scheduling at 256 requires holding resources while waiting, which is where deadlock lives.

"Is there a fairness requirement between research teams?" Assumed yes: a hierarchical quota (org → team → user). Without it, one team's 500-job sweep starves everyone, which is the most common real-world complaint about shared research clusters and is a scheduling problem, not a people problem.

"What is the SLO?" Two different ones, and they cannot be the same metric:

• Inference: capacity available within 60 s of demand (bounded by model load time).
• Training: queue time p90 < 30 min for jobs under 64 GPUs; large jobs are scheduled, not queued.

Functional

1. Submit training jobs with (gpus, topology_requirement, priority, max_runtime).
2. Inference deployments declare a desired replica count and scale within a band.
3. Gang scheduling: a training job gets all its GPUs or none.
4. Preemption with checkpoint-and-requeue.
5. Hierarchical quotas with borrowing.

Non-functional

Explicitly out of scope

• The training framework's own parallelism (FSDP/DeepSpeed config) — we schedule the shape it asks for, we do not choose it.
• Inference engine internals — m01, ../WARMUP.md.
• Multi-cluster / multi-region federation. Noted in §9.
• Spot/preemptible cloud capacity — different problem (the provider preempts you).

2. Scale Numbers

The cluster. 128 nodes × 8 H100 = 1,024 GPUs. Per node: 8× NVLink-connected GPUs, 2× 400 Gb/s InfiniBand NICs. Nodes grouped into rails/pods of 32 nodes sharing a leaf switch.

The interconnect hierarchy, which is the design's substrate:

The all-reduce arithmetic — a 70B model, TP8, batch 137, per decode step:

per all-reduce   = batch x hidden x 2 B = 137 x 8192 x 2   = 2.24 MB
ring factor      = 2(n-1)/n at n=8                          = 1.75
per step         = 2.24 MB x 1.75 x 2 per layer x 80 layers = 628 MB moved per GPU

A TP8 job placed across nodes runs at roughly two-thirds the speed of the same job on one node, using the same hardware. That is not a tuning issue. It is the scheduler's most consequential decision, and it is invisible in any metric that counts GPUs.

Fragmentation, and the number that reframes the problem. If free GPUs are distributed randomly across 128 nodes, the probability that a given node is entirely free is f^8 where f is the free fraction:

Read that last row again. With half the cluster idle — 512 GPUs doing nothing — random placement gives you an expected half of one node on which a TP8 job can be placed. You would have to empty 90%+ of the cluster before 8-GPU jobs schedule reliably by luck.

The conclusion is not "we need more GPUs." It is "placement must be topology-aware from the first allocation, because you cannot recover topology after you have destroyed it." That is deep dive A, and it is the single most valuable thing to say in this round.

Checkpoint cost, which bounds preemption (70B, FSDP-sharded across the job's GPUs):

A full training checkpoint is 980 GB, not 140 GB — 7× the model, because the optimizer state dominates. Quoting the parameter size as the checkpoint size is a common and revealing error. At cluster-FS bandwidth it is ~5 s, which makes preemption genuinely cheap; on a single writer it is 98 s, which makes it prohibitive. The storage architecture decides whether preemption is a viable scheduling primitive at all — so this is a scheduler design that depends on a storage decision, and saying so is the kind of cross-system reasoning the round is testing.

3. API Surface

POST /jobs                                        # training
  { name, image, gpus: 256, topology: "pod",      # node | pod | any
    priority: 100, max_runtime: "72h",
    checkpoint: {path, interval: "10m"},
    preemptible: true }
  -> 202 {job_id, queue_position, eta}

POST /deployments                                 # inference
  { model, replica_shape: {gpus: 4, topology: "node"},
    min_replicas: 8, max_replicas: 60,
    priority: 1000 }
  -> 200 {deployment_id}

PATCH /deployments/{id}  {desired_replicas: 34}   # the autoscaler's only verb
GET   /cluster/topology                           # free sets, not free counts

Three deliberate choices:

topology is a first-class, declared field. Not inferred, not a hint. A job that says topology: "node" is saying place all my GPUs within NVLink domains or do not place me. Making this explicit means the scheduler never has to guess, and the job never silently gets a 52% slowdown. The alternative — inferring topology needs from the framework config — fails the first time someone runs a job the inference logic does not recognize.

GET /cluster/topology returns free sets, not a free count. Because "204 GPUs free" is not actionable and, per §2, is usually a lie about what you can schedule:

{ "free_nodes_full": 3,
  "free_by_node": {"n017": 8, "n042": 8, "n091": 8, "n003": 2, ...},
  "largest_placeable": {"node": 8, "pod": 24, "any": 204} }

largest_placeable is the number an operator actually needs, and exposing it is how fragmentation stops being invisible. A cluster reporting any: 204, node: 0 is in trouble, and no GPU-count dashboard would show it.

Inference scales by desired_replicas, one field. The autoscaler's entire interface is a number, and the scheduler owns placement. This is level-triggered, not edge-triggered — the autoscaler declares a desired state and the scheduler reconciles, so a lost message costs a delay rather than a permanent divergence. Same reasoning as d12.

4. Data Model

node        (node_id, pod_id, gpu_count, gpus_free, health, drain_state)
allocation  (alloc_id, owner_kind, owner_id, node_id, gpu_mask, created_at, preemptible)
job         (job_id, team, gpus, topology, priority, state, submitted_at,
             last_checkpoint_at, preempt_count, max_runtime)
deployment  (dep_id, model, replica_shape, min, max, desired, priority)
quota       (scope, parent, guaranteed_gpus, max_gpus, borrowed)

gpu_mask is a bitmask over the node's 8 GPUs, not a count. Because which GPUs matters: on an 8-GPU H100 node, NVLink is all-to-all, but on other topologies GPUs pair through specific links, and a 4-GPU allocation of {0,1,2,3} may be materially better than {0,2,4,6}. Storing a count instead of a mask throws away the information the scheduler exists to manage — a small schema decision that determines whether topology-aware placement is possible at all.

preempt_count on the job, and it is not just telemetry. A job preempted repeatedly makes no progress while consuming scheduling effort and storage bandwidth. After N preemptions it must gain priority (aging) or the system will livelock a job forever while looking perfectly healthy on every dashboard. This is the starvation guard, and it belongs in the data model because a policy you cannot query is a policy you cannot verify.

quota is hierarchical with borrowed tracked separately from guaranteed_gpus. Guaranteed capacity is never preempted; borrowed capacity is preempted first, in reverse order of borrowing. That single rule is what makes over-subscription safe: teams can use idle capacity without the owner losing the ability to reclaim it, which is the property that makes anyone willing to share in the first place.

5. High-Level Architecture

   training submits          inference autoscaler
         │                          │ desired_replicas
   ┌─────▼──────────────────────────▼─────────────────────────┐
   │  ADMISSION:  quota check · shape validation · queue        │
   └─────┬──────────────────────────────────────────────────────┘
         │
   ┌─────▼──────────────────────────────────────────────────────┐
   │  SCHEDULER  (single writer to the allocation table)         │
   │                                                             │
   │   1. inference first (never preemptible, latency SLO)       │
   │   2. training by priority, then age                         │
   │   3. placement: best-fit over TOPOLOGY SETS, not GPUs       │
   │   4. if unplaceable: preempt lowest-priority preemptible    │
   │      allocations, newest-borrowed first                     │
   │   5. gang: reserve-and-wait with a deadline (§7)            │
   └─────┬───────────────────────────────────┬──────────────────┘
         │ bind                              │ preempt(checkpoint, deadline)
   ┌─────▼──────────────┐          ┌─────────▼──────────────────┐
   │ NODE AGENTS        │          │  running jobs               │
   │  cgroup/MIG bind   │          │  SIGTERM -> checkpoint      │
   │  health · drain    │          │  -> exit -> requeue         │
   └────────────────────┘          └─────────────────────────────┘
         │ heartbeat (state, health, ECC, NVLink errors)
   ┌─────▼──────────────────────────────────────────────────────┐
   │  RECONCILER: level-triggered; actual state -> desired       │
   └────────────────────────────────────────────────────────────┘

Five decisions:

1. One scheduler process is the single writer to allocations. At 1,024 GPUs and job arrivals measured in jobs/minute, a single writer is ample — placement decisions are microseconds and the bottleneck is the world, not the CPU. Sharding the scheduler for scale here would be solving a problem you do not have while creating one you cannot solve (two schedulers double-binding a GPU). HA is leader election with a fencing token (d11), not partitioning.

2. Inference is scheduled first and is never preemptible. Not a fairness statement — a physics one. Preempting inference means killing in-flight requests whose state cannot be checkpointed. Training loses minutes; inference loses users.

3. Placement is best-fit over topology sets. Deep dive A.

4. Preemption is cooperative with a hard deadline. SIGTERM → the job checkpoints → exits. If it has not exited within the deadline (checkpoint_size / storage_bw × 3, so ~15 s here), SIGKILL. Cooperative-only is a liveness bug — a hung job would block inference scale-up indefinitely, and inference has a 60 s SLO.

5. The reconciler is level-triggered. It compares actual node state to the allocation table continuously and fixes drift. Edge-triggered ("send a start command") loses work on any missed message; level-triggered converges from any state, including states nobody designed for. Same argument as d12, and it is the reason a scheduler that has been restarted can recover without knowing what happened while it was down.

6. Deep Dive A: Topology Is Not a Preference, It Is a Constraint

The arithmetic that makes it a constraint

From §2: a TP8 job spread across pods pays +51.8% on every decode step versus the same job on one node. That is not a tail effect or a p99 — it is the mean, on every step, forever.

Restated as capacity: 8 GPUs placed badly deliver the throughput of ~5.3 GPUs placed well. A scheduler that ignores topology has thrown away a third of the hardware while reporting 100% allocation. Your utilization dashboard will be green during this. That is the sentence to say.

Why fragmentation is the real enemy

The §2 table is the crux: at 50% free, expected fully-free nodes ≈ 0.5.

The mechanism is worth stating because it explains why the problem is self-inflicting: every allocation that takes 2 GPUs from an empty node destroys an 8-GPU placement opportunity permanently — permanently, because the other 6 cannot be recovered until that 2-GPU job exits, and its exit is not correlated with anyone's need.

Fragmentation is a ratchet. It only worsens under a topology-blind scheduler, and no amount of free capacity repairs it. That framing — a ratchet, not a fluctuation — is what justifies spending the design's complexity budget here.

The placement algorithm

Best-fit over topology sets, with tiered fallback. The rule is: consume the most fragmented resource that still satisfies the constraint.

def place(req):
    if req.topology == "node":
        # Prefer the node that will have the LEAST usable remainder.
        # Best-fit, not first-fit: leave big holes big.
        cands = [n for n in nodes if n.free >= req.gpus]
        return min(cands, key=lambda n: (n.free - req.gpus, -n.fragmentation_score))

    if req.topology == "pod":
        for pod in sorted(pods, key=lambda p: p.free_gpus):     # tightest pod that fits
            if pod.free_gpus >= req.gpus:
                return pack_within(pod, req)                    # whole nodes first
    return spread(req)                                          # topology: any

Best-fit, deliberately, and it is the opposite of the usual instinct. First-fit or worst-fit(most-free) spread small jobs across empty nodes and destroy large placements. Best-fit puts a 2-GPU job on a node that already has 6 used, preserving whole-node capacity for jobs that need it. One line of policy, and it is most of the fragmentation defence.

Two supporting rules:

• Whole-node allocation for jobs that are a multiple of 8. A 16-GPU job takes 2 whole nodes, never 3 partial ones. Slight waste when it needs 15; large gain in preserved topology.
• Segregate by shape. Reserve a set of nodes for sub-node allocations (1–4 GPUs: notebooks, small inference, dev) so their churn cannot fragment the whole-node pool. This is cell-based isolation applied to fragmentation — and calling out that it is the same primitive is worth as much as the rule.

Defragmentation, and its honest limits

Best-fit slows the ratchet; it does not reverse it. Reversing it requires moving running work, and that is where it gets expensive:

Inference replicas are the defragmentation lever, and this is the design's nicest inversion: they are the cheapest thing to move (they have no persistent state to preserve — the thing that made them un-preemptible is exactly what makes them relocatable). So:

When largest_placeable["node"] < 1 and a node-topology job is queued:
    find inference replicas occupying partial nodes
    relocate them into partial nodes elsewhere (start new, drain old)
    -> frees whole nodes without preempting any training

Cost: transient over-provisioning during the move (both replicas exist briefly), and a 60 s window per replica. Bounded by moving at most N replicas concurrently.

And the limit, stated honestly: if the cluster is genuinely full, no algorithm creates topology. Defragmentation buys placement at 70–85% occupancy; above that the answer is queueing, and the design should say so rather than implying it can always place.

7. Deep Dive B: Preemption, Gang Scheduling, and Who Yields

Gang scheduling is where the deadlock lives

A 256-GPU job needs all 256 simultaneously. Two obvious approaches, both broken:

Approach 1 — wait for 256 free, then grab. Never happens on a busy cluster: by the time the 256th frees, others have been taken. Starvation of large jobs, and the classic symptom is a big job sitting at "queued" for days on a cluster that is never full.

Approach 2 — grab GPUs as they free, hold until you have 256. Now the job holds 200 idle GPUs waiting for 56. Two such jobs deadlock, each holding what the other needs. And the cluster shows high allocation with near-zero utilization — allocated, idle, and going nowhere.

The mechanism that works: reservation with a deadline

def schedule_gang(job):
    reservation = reserve_free(job)                    # take what is free NOW
    deadline = now + RESERVATION_TIMEOUT               # e.g. 10 minutes

    while len(reservation) < job.gpus and now < deadline:
        # Actively make room rather than waiting for luck.
        victims = pick_preemptible(job.gpus - len(reservation), below=job.priority)
        if victims:
            preempt(victims); reservation += await_release(victims)
        else:
            reservation += await_natural_release(short_poll=True)

    if len(reservation) < job.gpus:
        release_all(reservation)          # <-- the anti-deadlock rule
        job.priority += AGING_BONUS       # <-- the anti-starvation rule
        requeue(job)
    else:
        bind(job, reservation)

Three rules, each fixing a specific failure:

1. A reservation has a deadline and is released whole on expiry. No indefinite holding, so no deadlock. The cost — up to 10 minutes of partially-idle GPUs — is the price of gang scheduling and should be stated as such rather than hidden.
2. Priority ages on failure. Each failed attempt raises priority, so a large job eventually outranks the stream of small ones that keeps beating it. Without aging, large jobs starve forever on a cluster that is never full, which is the single most common complaint about real research clusters.
3. Preemption is active, not passive. Waiting for natural release is unbounded; a large job makes room by preempting lower-priority preemptible work.

Backfill makes the reservation window cheap. While a 256-GPU reservation fills, run short jobs (max_runtime < time_to_deadline) on the reserved GPUs. They are guaranteed to finish before the gang needs the resources, so utilization stays high during the wait. Backfill is what makes reserve-and-wait affordable, and mentioning it unprompted is a strong signal — it is the non-obvious half of the classic HPC answer.

Who yields, and the cost of yielding

The preemption order, and the reason for each rank:

1. borrowed-over-quota, lowest priority, newest first    <- borrowed capacity is on loan
2. borrowed-over-quota, by priority then age
3. preemptible within quota, lowest priority
4. --- never below this line ---
   guaranteed-quota jobs · inference replicas · non-preemptible jobs

"Newest borrowed first" is LIFO, and LIFO is correct here even though it feels unfair. A job that has run for 10 hours has 10 hours of progress at risk beyond its last checkpoint and has built up cache and JIT state; a job that started 2 minutes ago has almost nothing to lose. FIFO preemption maximizes destroyed work; LIFO minimizes it. This is one of the places where the intuitive fairness rule is the wrong engineering rule, and knowing why is the point.

Cost of a preemption, from §2:

checkpoint write (980 GB, FSDP-sharded, cluster FS @200 GB/s)   ~5 s
process teardown + requeue + restart + reload                   ~60-120 s
LOST PROGRESS: time since last checkpoint                       up to the interval

The lost progress dominates, and it is controlled by the job's own checkpoint interval — which means the scheduler's SLO ("< 5 min lost progress p95") is only achievable if jobs checkpoint at least that often. So the scheduler must enforce it, not hope for it:

preemptible: true  REQUIRES  checkpoint.interval <= 5m

A job that will not checkpoint frequently cannot be preemptible, and therefore cannot borrow over-quota capacity. The incentive is aligned exactly right: you get access to spare capacity in exchange for being cheap to reclaim. That is the whole social contract of the cluster in one rule, and it is enforced by the API rather than by a wiki page.

Preemption must not thrash

A job preempted, requeued, rescheduled, and preempted again makes negative progress. Guards:

• Minimum runtime before preemptible. A job cannot be preempted within 10 minutes of starting. Prevents the pathological loop where a job's own restart triggers the pressure that kills it.
• preempt_count aging (§4): +priority per preemption, so a repeatedly-preempted job climbs out of the danger zone.
• A preemption budget: at most X% of running training GPU-hours preempted per hour. If demand exceeds the budget, inference scale-up is throttled instead — and an alarm fires, because that is a capacity-planning signal, not a scheduling one.

That last one is the important one and it is easy to get backwards. Without a budget, a bad inference autoscaler can preempt the entire training fleet in minutes and nothing in the system objects. The budget converts a silent catastrophe into a paged alert. Any mechanism powerful enough to reclaim the cluster needs a rate limit on its own authority.

8. Failure and Recovery

"The cluster keeps running when the scheduler is down" is the property to lead with. The scheduler binds; it does not supervise. Node agents hold their allocations and running work continues. A scheduler outage stops new placements — which is a degradation, not an outage.

This is the single most important structural property of the design, and it comes from one decision: the allocation table is the source of truth and the scheduler is a stateless function over it. It is the same shape as d12's control-plane/data-plane split. A control plane whose failure kills the data plane is not a control plane.

On flaky GPUs — worth its own sentence. The naive response to a job failing with an XID error is to requeue the job. The correct response is to quarantine the node, because the failure is a property of the hardware, not the job. Requeue-without-quarantine produces the signature pathology of badly-run GPU clusters: one bad node silently eating every job that lands on it, appearing as a mysterious cluster-wide failure rate that no job owner can reproduce.

9. Bottlenecks and Evolution

Now: the binding constraint is fragmentation, not GPU count (§2, §6). The second is checkpoint storage bandwidth, because it sets the preemption rate.

Interventions in order:

1. Expose largest_placeable as the primary dashboard metric, above utilization. Cheap, immediate, and it makes the actual constraint visible. Most clusters do not have this and it is why they are surprised.
2. Shape segregation (§6): dedicated node pools for sub-node allocations. Removes the largest source of fragmentation for the cost of some stranded capacity.
3. Inference-replica defragmentation (§6). Uses the cheapest-to-move workload to repair topology without touching training.
4. Elastic training (torchelastic-style): jobs that run at 128 or 256 GPUs and adjust. Turns a gang-scheduling problem into a scaling problem and eliminates the reservation window entirely. The highest-value change here, and the hardest — it requires the training code to cooperate, so it is an organizational change as much as a scheduling one.
5. Time-sliced sharing (MPS/MIG) for small inference and notebooks. MIG partitions an H100 into up to 7 isolated instances with hard memory isolation. Good for the long tail of 1-GPU work; useless for anything that needs full HBM bandwidth, which is all of decode (a MIG slice gets a proportional slice of bandwidth, and decode is bandwidth-bound — WARMUP §2.2). Say the limit when proposing it, or the interviewer will.
6. Multi-cluster federation. Only after single-cluster utilization is above 85%; federating two badly-scheduled clusters produces one badly-scheduled system with added latency.

10. Tradeoffs Explicitly Rejected

Rejected: static partition (the status quo). Rejected on the anti-correlation premise from §1 — each side idles while the other queues. Worth stating what would make it right: if training and inference demand were correlated, sharing gains little and the split's operational simplicity wins. The design depends on a measured property of the workload, and naming that dependency is better than defending sharing universally.

Rejected: topology-blind bin packing. §6. Reports 100% allocation while delivering ~67% of the throughput on cross-pod TP jobs, and ratchets the cluster into a state where nothing large schedules.

Rejected: first-fit or worst-fit placement. Both scatter small allocations across empty nodes. Best-fit preserves large holes; it is one line and most of the defence.

Rejected: preempting inference for training. Inference state (the KV cache) is not checkpointable and in-flight requests die. Physics, not policy.

Rejected: fully cooperative preemption (no SIGKILL). A hung job would block inference scale-up past its 60 s SLO indefinitely. Cooperative with a hard deadline.

Rejected: FIFO preemption ordering. Maximizes destroyed work. LIFO on borrowed capacity minimizes it (§7).

Rejected: sharding the scheduler. At 1,024 GPUs and jobs/minute arrival rates there is no throughput problem, and two writers to the allocation table is a double-binding bug waiting to happen. Single writer + leader election + fencing.

Rejected: MIG for the main inference fleet. Decode is bandwidth-bound; a MIG slice gets proportionally less bandwidth, so 7 slices do not serve 7× the requests. MIG is for the low-utilization tail.

Rejected: letting the autoscaler preempt without a budget. One bad scaling decision could drain the training fleet in minutes with no alarm. §7.

The Hostile Critique

C1. "Best-fit preserves whole nodes. But your inference replicas are 4 GPUs, and best-fit puts each one on a node that already has 4 used — so every node ends up half-inference, half-something-else. You've perfectly fragmented the cluster into 4-GPU chunks using the rule you introduced to prevent fragmentation. What placement do you actually get after a week?"

C2. "Reservation timeout is 10 minutes, and on expiry you release everything and requeue with an aging bonus. A 256-GPU job on a busy cluster fails this repeatedly. Each attempt idles up to 200 GPUs for 10 minutes. Ten attempts is over 300 GPU-hours burned on scheduling a job that hasn't started. How is that better than a reservation that holds?"

C3. "You require checkpoint.interval <= 5m for preemptible jobs. A 980 GB checkpoint every 5 minutes, from a job that runs for 3 days, is 864 checkpoints — 846 TB written. Multiply by the number of concurrent training jobs. Is your storage system sized for that, and what does it do to the checkpoint bandwidth you rely on for preemption?"

C4. "Defragmentation relocates inference replicas: 'start new, drain old'. You need free capacity to start the new one. You're doing this because the cluster is fragmented, which means it's full. Where does the capacity for the new replica come from?"

C5. "Node agents hold allocations so the cluster survives a scheduler outage. The scheduler comes back and rebuilds from the allocation table. In the meantime a node died and its replacement came up with the same hostname. What does the reconciler do?"

C6. "Your preemption budget throttles inference scale-up when training preemption exceeds X%/hour. So a genuine traffic spike gets throttled to protect a research job. Who signs off on that tradeoff at 3am, and what does the on-call engineer actually see?"

The Revision

R1 — Best-fit needs a shape-aligned free list, not just a "least remainder" rule (answers C1)

The critique is correct and it identifies a real emergent pathology: best-fit on a mixed workload converges to a state where every node is partially occupied by a different shape. The rule that prevents small jobs from fragmenting big holes does nothing to prevent 4-GPU jobs from fragmenting each other, and inference replicas are the highest-churn allocation in the cluster.

Change: placement is shape-aligned, and nodes carry a soft shape affinity.

# A node that already hosts 4-GPU allocations prefers more 4-GPU allocations.
# Halves pack with halves; whole nodes stay whole.
def score(node, req):
    remainder = node.free - req.gpus
    aligned   = (node.shape_affinity in (None, req.gpus))
    return (0 if remainder == 0 else 1,      # exact fill is always best
            0 if aligned else 1,             # then shape-aligned
            remainder)                        # then tightest

With 4-GPU replicas, nodes fill in pairs and reach free == 0 rather than stalling at 4. The cluster's free space stays in whole-node units because partial nodes are actively driven to full rather than left half-open.

Plus the structural fix, promoted from §9 to required: inference replicas whose shape divides a node (4 or 8) get their own node pool, sized to max_replicas × gpus_per_replica. Their churn — which is constant, because they autoscale — cannot touch the training pool at all.

Cost: stranded capacity at the pool boundary, and a pool-sizing decision that is now a capacity-planning input. Bounded by allowing the training pool to borrow from the inference pool's unused headroom as preemptible capacity — so the stranding is recovered by exactly the mechanism already built, which is the satisfying part.

And the lesson: a packing heuristic tuned against one workload mix produces a new pathology under another. The defence is not a better heuristic but segregation by shape, so each pool sees a homogeneous mix. Same conclusion as d12's cells, one level down.

R2 — Reservations must hold, with the idle time backfilled and bounded (answers C2)

The critique's arithmetic is right and the original design traded a deadlock for a livelock: 300 GPU-hours burned re-attempting a placement is strictly worse than the deadlock it was avoiding.

Change: the reservation holds across attempts, and the anti-deadlock property is provided by a different mechanism — a total order on reservations.

# Reservations are ordered by (priority, submit_time, job_id) — a total order.
# A job may only take GPUs from a reservation ranked BELOW it.
# Higher-ranked reservations are never blocked by lower-ranked ones.
# => no cycle in the wait-for graph => no deadlock, and holding is safe.

This is the classic resource-ordering solution to deadlock, and it applies exactly: deadlock requires a cycle in the wait-for graph, and a total order makes cycles impossible. With no deadlock risk, the reservation can hold indefinitely and the 10-minute release-everything rule is deleted.

And the idle time is not idle:

Reserved-but-unfilled GPUs run BACKFILL jobs with
    max_runtime < estimated_time_to_fill
Backfill is preempted the instant the reservation completes.

Backfill was in the original as a nice-to-have; the critique makes it load-bearing. With it, the cost of a slow-filling reservation drops from "200 idle GPUs" to "200 GPUs running short jobs", which is what HPC schedulers have done for thirty years.

Bounded by a starvation guard on the other side: if a reservation cannot fill within 4 hours even with active preemption, the cluster cannot host the job — alert, and surface it as a capacity-planning signal rather than leaving the job queued forever. A job that will never run should say so, not wait quietly.

Cost: the total order means a high-priority job can hold a reservation that blocks a lower-priority one indefinitely — starvation is pushed to the bottom of the priority order, where aging must handle it. Aging must therefore be strong enough to cross priority classes eventually, which is a policy parameter that needs measuring, and the design should say so rather than pretending it is solved.

R3 — Checkpoint frequency must be adaptive, and the storage math must be in the design (answers C3)

The critique is right and this was arithmetic never done — the most common defect class in the taxonomy, found here in my own design.

980 GB every 5 min = 3.27 GB/s sustained per job
x 6 concurrent training jobs                    = 19.6 GB/s sustained, forever
3-day job = 864 checkpoints x 980 GB            = 846 TB written per job

At 200 GB/s cluster-FS bandwidth, six jobs consume ~10% of it continuously just for checkpoints that will mostly never be read. And it competes with the training data read path, which is the job's actual bottleneck.

Change 1 — separate full checkpoints from preemption checkpoints.

The preemption checkpoint does not need to be periodic at all. The job is being asked to stop; it has ~15 s to write. That is the entire requirement, and it costs nothing when no preemption happens. The original design confused "recoverable from preemption" with "continuously checkpointed", and they are different requirements.

This deletes the 846 TB entirely: durable checkpoints at 45 min = 96 writes over 3 days = 94 TB, 9× less.

Change 2 — the SLO changes, honestly. "< 5 min lost progress" is no longer free. On preemption, the job writes a fresh checkpoint at that moment, so lost progress is ~0, better than before — but only if the checkpoint completes within the deadline. Hence:

preemption_deadline = checkpoint_bytes / measured_storage_bw x 3
                    = 980 GB / 200 GB/s x 3 ≈ 15 s

with the guard from §8: if measured storage bandwidth degrades, stop preempting — a preemption you cannot checkpoint is a kill.

Change 3 — the number goes in the capacity model. Checkpoint bandwidth is now a first-class cluster resource with a budget, sized as concurrent_preemptions × checkpoint_size / deadline. Which bounds how many jobs can be preempted at once — a limit the original design did not know it had, and would have discovered during an incident.

R4 — Defragmentation needs a reserved swap pool (answers C4)

The critique identifies a genuine chicken-and-egg: relocation needs free capacity, and relocation is triggered by not having free capacity in the right shape. As written, defragmentation only works when it is not needed.

Change: hold back a swap pool — one or two whole nodes, never allocated to normal work, existing only as relocation scratch.

swap_pool = max(1, ceil(0.01 x cluster_nodes))     # 128 nodes -> 2 nodes = 16 GPUs

relocate(replica):
    start replacement in swap_pool
    drain + stop the original            -> frees a partial node
    the freed GPUs join the swap pool; the pool "walks" across the cluster

The pool is a moving hole. Each relocation returns capacity to it, so a single 8-GPU pool can defragment an arbitrary number of nodes sequentially. That is the property that makes 1.5% overhead sufficient rather than needing pool-sized-to-the-problem.

Cost: 16 GPUs (1.5%) permanently unavailable for scheduling — about $350k/year of hardware held in reserve. State the number. It is justified against §6's finding that fragmentation can strand far more than 1.5% (at 50% free the cluster could not place a single TP8 job), but it is real money and pretending otherwise is how designs lose credibility.

Cheaper alternative worth naming: relocate on natural churn instead of on demand. When an inference replica restarts for any reason — deploy, autoscale-down-then-up, node drain — place it using the defragmenting choice rather than the load-balancing one. Free, slower, and for a fleet that redeploys daily it may be entirely sufficient. I would ship the free version first and measure whether largest_placeable["node"] recovers, then buy the swap pool only if it does not.

R5 — Node identity must be an epoch, not a hostname (answers C5)

The critique names a genuine correctness bug, and it is one that appears in every fleet system eventually: a hostname is not an identity. A replacement node with the same hostname inherits allocations that belong to hardware that no longer exists. The reconciler sees n042 present with alloc_id=X expected, and "reconciles" by considering it correct — binding a job to a node that never received it.

Change: identity is (node_id, boot_epoch), where boot_epoch is a monotonically increasing value the node reports (boot time, or a persisted counter).

if report.boot_epoch != table.boot_epoch[node_id]:
    # This is not the machine we allocated to. Everything on it is gone.
    invalidate_all_allocations(node_id)
    requeue_affected_jobs()
    table.boot_epoch[node_id] = report.boot_epoch

This is a fencing token (d11) in a different costume, and recognizing that is the point: the general rule is any identity that can be reused must carry an epoch, and it applies to nodes, leaders, sessions, and leases alike.

And the same bug class, one level up: the same node rebooting without replacement also invalidates its allocations, which is correct — a reboot destroyed the running work whether or not the hardware changed. The epoch check handles both cases with one rule, which is how you know it is at the right level of abstraction.

Cost: node agents must persist or derive a monotonic epoch. Boot time works if clocks are sane; a persisted counter in the agent's state directory is more robust and is what I would ship. On first contact with an unknown node, allocations are assumed absent — the safe direction.

R6 — The budget must page a human with a decision, not silently throttle (answers C6)

The critique is right that the original design buried a business decision inside a scheduler parameter. "Throttle inference scale-up to protect training" is a product decision, and at 3am the on-call engineer needs to know that is what happened, not deduce it.

Change 1 — make the tradeoff explicit and tiered, not a single threshold.

preemption_rate < 10%/hr   : preempt freely, no alarm
10-25%/hr                  : preempt, WARN, annotate the incident timeline
> 25%/hr                   : preempt only for inference below min_replicas
                             (the availability floor -- never throttled)
                             everything above the floor queues + PAGE

The floor is the key structure. Inference scaling up to min_replicas is availability and is never throttled — the product does not go down to protect a research job. Scaling above the floor is capacity optimization and can wait for a human. A tiered response distinguishes "we are losing users" from "we are losing headroom", which the original single threshold could not.

Change 2 — the page says what to do, not what happened.

PAGE: inference scale-up throttled by preemption budget
  want 44 replicas, have 31, min_replicas 24 (floor is SAFE)
  blocked: 52 GPUs behind preemption budget (28%/hr, limit 25%)
  would preempt: job-8817 (team-nlp, 128 GPU, 6h in, ckpt 4m ago)
                 job-8903 (team-rl,   64 GPU, 20m in, ckpt 20m ago)
  ACTIONS:  [raise budget to 40%]  [preempt listed jobs]  [accept degraded]
  Cost of preempting: ~26 GPU-hours of lost training progress

The page names the cost of each option in the same units. The engineer is not asked to understand the scheduler; they are asked to choose between 26 GPU-hours of training and 13 replicas of serving capacity. That is a decision a human can make at 3am; "preemption budget exceeded" is not.

And the general lesson worth stating plainly: when a system automates a tradeoff between two organizations' interests, it must be able to explain the tradeoff in the units each organization cares about. Any threshold that silently resolves a conflict between two teams will eventually be discovered during an incident, by the team that lost — and at that point the argument is about trust, not about the threshold.

References

• ../WARMUP.md#54-parallelism-tp-pp-ep — why TP is bandwidth-hungry and PP is not
• ../WARMUP.md#22-decode-is-memory-bandwidth-bound — the MIG limitation in §9
• m01-llm-api-platform.md — the inference fleet this schedules for
• m08-training-fault-tolerance.md — checkpointing in full; R3 depends on it
• ../../systems-design/designs/d11-lock-service.md — fencing tokens, the primitive behind R5
• ../../systems-design/designs/d12-multi-tenant-control-plane.md — cells, level-triggered reconciliation, control/data plane split
• Verma, A. et al. Large-scale cluster management at Google with Borg. EuroSys 2015 — quotas, priority, preemption, and the alloc model
• Ousterhout, K. et al. Sparrow: Distributed, Low Latency Scheduling. SOSP 2013 — power-of-two for schedulers
• Jeon, M. et al. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. ATC 2019 — measured fragmentation and gang-scheduling delay in a real cluster
• Weng, Q. et al. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. NSDI 2022
• Slurm documentation — backfill scheduling and reservations, the prior art for §7