m08 — Training Fault Tolerance at Scale

A fully worked design. Keeping a 30-day, thousand-GPU synchronous training run alive on hardware that fails constantly.

The number that reframes it: at 16,384 GPUs the mean time between interruptions is 3.1 hours. Not 3.1 hours between node failures — 3.1 hours between events that stop the entire job. A 30-day run takes 233 interruptions.

And the number that redirects the optimization: at that scale, restart time alone costs 5.4% of the run, no matter how often you checkpoint. Everyone tunes the checkpoint interval; the leverage is in the restart.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: The Goodput Equation
• 7. Deep Dive B: Failures That Do Not Announce Themselves
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We're going to run a 30-day pretraining job on a thousand GPUs. Make sure it finishes, and make sure the result is trustworthy."

Two requirements, and the second is the harder one. "Finishes" is availability engineering. "Trustworthy" means the model that comes out must not have been silently corrupted by hardware that computed the wrong answer without telling anyone — and at this scale that is a real, measured, routine occurrence, not a hypothetical.

Open with the failure rate, because it is the fact everything else follows from. Extrapolating from published large-run data (Llama 3 405B: 419 unexpected interruptions in 54 days on 16,384 GPUs):

MTBF halves every time the cluster doubles, because a synchronous job fails if any rank fails. Scaling up makes the job proportionally more fragile, and the mitigation must scale with it — that is the sentence that frames the whole design.

The second thing to say: this is a synchronous job. Every rank participates in an all-reduce every step, so there is no partial progress and no graceful degradation. One rank stops, all 1,024 stop. Everything below is a consequence of that coupling.

1. Requirements and Scope

Clarifying questions asked

"Is the job synchronous data-parallel, or is asynchrony acceptable?" Assumed fully synchronous (FSDP/ZeRO-3 or 3D parallelism). Asynchronous SGD would change the fault model entirely — a lost rank would only cost its own gradients — but it changes convergence behaviour, and at pretraining scale nobody accepts that. Naming the alternative and why it is rejected is worth thirty seconds because it shows the fault model is a consequence of a choice, not a given.

"What's the checkpoint size?" 70B with Adam: params (bf16, 2 B) + fp32 master (4) + m (4) + v (4) = 14 bytes/param = 980 GB. Not 140 GB — the optimizer state is 6× the model, and quoting the parameter size as the checkpoint size is a common and revealing error.

"What storage is available?" Assumed a parallel filesystem at ~200 GB/s aggregate, FSDP-sharded writes. This makes a checkpoint ~5 s, and whether it is 5 s or 100 s changes the design substantially (§6), so it is a question to ask rather than assume.

"Can the job resize?" Assumed not initially — fixed world size, elastic as a §9 improvement. This matters because it determines whether a failure means wait for a replacement or continue at reduced width.

"How would we know if the model were being corrupted?" The question behind "trustworthy", and the one nobody asks. Assumed: we would not, without a mechanism. That mechanism is deep dive B.

Functional

1. Detect a failed, hung, or degraded rank within seconds.
2. Checkpoint model + optimizer + RNG + dataloader position atomically.
3. Restart and resume with no lost or repeated data.
4. Quarantine bad hardware so the same node does not eat the next attempt.
5. Detect silent corruption before it reaches the weights.

Non-functional

Explicitly out of scope

• The training algorithm and parallelism strategy. We keep it alive; we do not choose it.
• Cluster scheduling and placement — m03.
• The data pipeline — m04, whose dataloader state we checkpoint.
• Model quality evaluation — m05.

2. Scale Numbers

Failure rate. The table in The Prompt. At 1,024 GPUs: MTBF 49.5 h, 15 interruptions over 30 days.

Checkpoint size and cost:

70B, Adam, FSDP-sharded:
  bf16 params   2 B/param   140 GB
  fp32 master   4 B/param   280 GB
  Adam m        4 B/param   280 GB
  Adam v        4 B/param   280 GB
                            ------
                            980 GB

write at 200 GB/s aggregate (sharded, all ranks write in parallel):  ~5 s

Cost of one interruption:

detect            10-90 s   (crash vs hang -- the difference matters, §7)
reschedule        60-120 s  (m03: gang scheduling, node replacement)
process start     30 s
load checkpoint   980 GB / 200 GB/s = 5 s      + framework init
NCCL re-init      30-60 s   at 1,024 ranks
first step        --
                  ~5-10 minutes total
+ LOST PROGRESS: time since the last checkpoint

Now the arithmetic that redirects the whole design. Total overhead is three terms:

\[ \text{overhead} = \underbrace{\frac{C}{T}}{\text{checkpoint}} + \underbrace{\frac{T/2}{\text{MTBF}}}{\text{lost work}} + \underbrace{\frac{R}{\text{MTBF}}}_{\text{restart}} \]

with C = checkpoint cost, T = interval, R = restart time.

At 16,384 GPUs (MTBF 3.1 h), C=30 s, R=600 s:

The restart column does not move. It is 5.39% of the entire run regardless of how you tune the checkpoint interval, because it depends only on R / MTBF.

Everyone tunes the checkpoint interval. The largest single controllable term at scale is restart time, and almost nobody optimizes it.

Halving restart from 600 s to 300 s saves 2.7% of a 30-day run — 19 hours, ~$50k at 1,024 GPUs — and no checkpoint-interval choice can do that. That is the finding to lead with, and it comes from writing down three terms instead of one.

Straggler cost, the other half of goodput:

A synchronous job runs at the speed of its slowest rank. One thermally-throttling GPU costs more per day than most engineers' salaries, and it produces no error, no alarm, and a perfectly healthy dashboard.

3. API Surface

POST /runs      { name, world_size, image, entrypoint,
                  checkpoint: {uri, interval: "auto"},     # "auto" -> Young/Daly, §6
                  health: {step_timeout: "90s", straggler_sigma: 3} }
  -> {run_id}

GET  /runs/{id}/health
  -> { step_time_p50, step_time_max, slowest_rank, goodput_7d,
       interruptions[], quarantined_nodes[], sdc_checks: {last, status} }

POST /runs/{id}/checkpoint          # force one, e.g. before a maintenance window
POST /runs/{id}/resume  {from}      # explicit resume; default is latest VALID
GET  /runs/{id}/timeline            # every interruption: cause, cost, node

Four decisions:

interval: "auto" is the default, and it derives the interval from measured MTBF. §6 gives the formula; the point is that the interval is a computed consequence of the failure rate and the checkpoint cost, not a number someone picked. A run whose MTBF degrades (a flaky rack) should checkpoint more often automatically.

goodput is the headline metric, not uptime. A job that is "up" while running 20% slow behind a straggler is not making progress at the rate it costs. Uptime is the metric that lets a straggler hide; goodput is the one that surfaces it.

/timeline with cause and cost per interruption. After 15 interruptions in 30 days you need to know whether they were 15 different nodes or the same one three times — and the second case is a quarantine bug, which is far more actionable.

resume defaults to the latest valid checkpoint, not the latest. A checkpoint written during a partial failure may be corrupt (§7), so validity is a property that must be established, not assumed.

4. Data Model

run          (run_id, world_size, state, started_at, config_digest, goodput)
checkpoint   (run_id, step, uri, shards[], digest, written_at,
              validated: bool, dataloader_state_ref)
interruption (run_id, at, detected_by, cause, node_id, lost_steps, restart_seconds)
rank_health  (run_id, rank, node_id, step_time_ewma, last_heartbeat, ecc_errors)
quarantine   (node_id, reason, since, run_ids_affected[])
sdc_check    (run_id, step, method, result, replay_rank)

checkpoint.validated is a separate field from "written". A checkpoint that exists is not a checkpoint you can resume from. Validation — all shards present, digests match, a test load succeeds — happens asynchronously after the write, and resume only considers validated ones. The alternative is discovering corruption during a recovery, which is the worst possible moment and turns one interruption into an hours-long incident.

interruption.restart_seconds is recorded per event, because §2 says it is the dominant term. You cannot optimize what you do not measure, and this is the field that makes the 5.39% visible.

quarantine.run_ids_affected links bad hardware to the jobs it killed. The signature failure of a badly-run cluster is one flaky node silently eating job after job — see m03 §8. This field turns "our jobs keep failing" into "node n0417 has killed six jobs".

dataloader_state_ref points into the same atomic write as the model shards. Per m04 §3, separating them means a crash between two writes produces correct weights with a wrong data position — silent data repetition, no error.

5. High-Level Architecture

   ┌────────────────────────────────────────────────────────────────┐
   │ SUPERVISOR (outside the job; survives it)                       │
   │   watches heartbeats · decides restart · quarantines nodes      │
   │   computes the checkpoint interval from measured MTBF           │
   └───────┬─────────────────────────────────────────┬───────────────┘
           │ heartbeat + step time per rank          │ restart / quarantine
   ┌───────▼─────────────────────────────────────────▼───────────────┐
   │ THE JOB: 1,024 ranks, synchronous                                │
   │                                                                  │
   │  ┌────────────────────────────────────────────────────────────┐ │
   │  │ per rank: step loop                                         │ │
   │  │   forward · backward · all-reduce · optimizer               │ │
   │  │   emit step_time; watchdog on collective timeout            │ │
   │  └────────────────────────────────────────────────────────────┘ │
   │                                                                  │
   │  CHECKPOINT (async, double-buffered):                            │
   │    step N: copy shard to pinned host memory   (~0.5 s, blocking) │
   │            -> background thread writes to storage (~5 s)         │
   │    training CONTINUES during the write                           │
   └──────────────────────────────────────────────────────────────────┘
           │ shards
   ┌───────▼──────────────────────────────────────────────────────────┐
   │ CHECKPOINT STORE: sharded, content-addressed                      │
   │   + async VALIDATOR: all shards? digests? test load?              │
   │   -> marks `validated`; keeps last K validated                    │
   └───────────────────────────────────────────────────────────────────┘

Five decisions:

1. The supervisor lives outside the job and outlives it. A watchdog inside the job cannot detect that the job is dead. Obvious, and skipped surprisingly often — the same control-plane/data-plane separation as d12.

2. Checkpointing is asynchronous with a short blocking copy. The GPU→pinned-host copy is fast (~0.5 s at 980 GB across 1,024 ranks: ~1 GB per rank over PCIe); the storage write happens in the background. This turns C from 5 s to ~0.5 s in the §6 formula, which shifts the optimal interval and cuts the checkpoint term by 10×. One implementation decision, a first-order effect on goodput.

3. Checkpoints are validated asynchronously, and only validated ones are resumable (§4).

4. The supervisor computes the checkpoint interval from observed MTBF rather than accepting a constant. §6.

5. Quarantine is a first-class action, not an operator's job. A node that caused an interruption is removed from the pool before the restart, so the restart does not land on it. Without this, the retry loop reschedules onto the machine that just failed — the single most common way a fault-tolerance system converts one failure into an outage.

6. Deep Dive A: The Goodput Equation

Writing it down

Goodput = fraction of wall-clock spent on useful computation. Three losses:

\[ \text{overhead} = \underbrace{\frac{C}{T}}{\text{checkpointing}} + \underbrace{\frac{T/2}{M}}{\text{lost work}} + \underbrace{\frac{R}{M}}_{\text{restart}} \]

C = checkpoint cost, T = interval, M = MTBF, R = restart time. T/2 because a failure arrives uniformly within the interval on average.

Minimize over T: differentiate, set to zero:

\[ -\frac{C}{T^2} + \frac{1}{2M} = 0 \quad\Longrightarrow\quad \boxed{T^* = \sqrt{2CM}} \]

This is Young's formula (1974), refined by Daly — fifty years old, from HPC, exactly applicable, and almost never used in ML training where intervals are chosen by feel.

Two things fall out that are worth saying:

(a) The optimum is flat. At 16,384 GPUs with C=30 s, the optimum is 13.6 min at 12.73% overhead; checkpointing every 30 min gives 15.14%. A 2× error in the interval costs 2.4 points. So the formula is worth using and is not worth agonizing over — and knowing which of those is true is more useful than the formula itself.

(b) The restart term is not in the formula at all. R/M is constant in T — no choice of interval touches it. At 16,384 GPUs it is 5.39% of the entire run, and at 32,768 it is 11%.

Therefore: optimize the restart

From §2, the ~600 s restart decomposes as:

Three interventions, largest first:

1. Hot spares. Keep 2–3% of the cluster idle as pre-warmed replacements: process started, container pulled, weights in host memory. A failure swaps in a spare instead of scheduling one. Reschedule + process start (~150 s) collapses to ~10 s.

Cost: 2–3% of the cluster idle. Against a 5.39% restart overhead, buying 3% to recover 2.5% is roughly break-even at 16,384 GPUs and clearly positive above it — and stating it that way, as a break-even rather than an obvious win, is more credible than asserting it.

2. In-memory checkpoints. Keep the most recent checkpoint in the host memory of the surviving ranks. Most failures kill one node, and the other 1,023 still hold their shards. Restore by peer-to-peer copy rather than storage read — ~1 s instead of 30 s. (This is the CheckFreq / Gemini-style approach and it is what modern frameworks are converging on.)

3. Do not tear down the world. With a framework that supports it, replace the failed rank in-place and re-form the process group rather than restarting all 1,024 processes. Saves the process-start and much of the NCCL init. This is the largest win and the hardest — it needs framework support (torchelastic, or a custom rendezvous), and it is where §9's elastic training leads.

What goodput actually is

The formula counts interruptions. Real goodput also loses to stragglers, which never interrupt anything:

\[ \text{goodput} = (1 - \text{overhead}) \times \frac{\text{step_time}{\text{ideal}}}{\text{step_time}{\text{actual}}} \]

From §2: a single 20%-slow rank multiplies the second term by 0.83 — wiping out more than the entire checkpoint-and-restart overhead, invisibly, with every health check green.

So the health system must alarm on step_time_max / step_time_p50, not on failures. That ratio is the straggler detector, and it is deep dive B.

7. Deep Dive B: Failures That Do Not Announce Themselves

Crashes are easy: the process exits, the supervisor notices in seconds. The expensive failures are the ones with no error.

Failure 1: the hang

A rank enters an all-reduce and never returns — a NIC wedge, a deadlock, a GPU that stopped responding. All 1,024 ranks sit in the collective forever. Nothing crashes. Every process is alive. Every heartbeat, if the heartbeat is a liveness ping, is green.

Detection:

# Per-rank watchdog: NOT a process liveness check -- a PROGRESS check.
if now - last_completed_step > STEP_TIMEOUT:      # 3x p99 step time, ~90 s
    dump_stacks_all_ranks()                       # the diagnostic that matters
    report_hang(rank, last_collective, peers_waiting)
    abort_job()

Two properties that make this work, and each is a mistake if omitted:

• The timeout is on progress, not on process liveness. A hung rank is perfectly alive.
• Dump stacks from every rank before aborting. With 1,024 ranks in the same collective, the one whose stack is different is the culprit. Without this dump the failure is undiagnosable and you will hit it again tomorrow, because nothing was quarantined.

NCCL's own NCCL_ASYNC_ERROR_HANDLING + TORCH_NCCL_BLOCKING_WAIT provide the timeout primitive; the design's contribution is making the abort automatic and the stack dump mandatory, so a hang costs 90 s and one diagnostic rather than hours of a human noticing that the loss curve went flat.

Failure 2: the straggler

A GPU that is slow but correct — thermal throttling, a degraded NVLink running at reduced width, a noisy neighbour on a shared NIC. The job runs, converges, and costs 20% more ($12,288/day at 1,024 GPUs).

Detection is a per-rank distribution, not a threshold:

# Every rank reports its own compute time (before the collective, so
# the measurement is not contaminated by waiting for others).
z = (rank.step_time - median_all_ranks) / mad_all_ranks
if z > 3 for 100 consecutive steps:
    flag_straggler(rank)

Measure compute time before the collective, and this is the subtle part. After the all-reduce every rank has the same elapsed time — they all waited for the slowest — so a post-collective measurement shows a uniformly slow job and identifies nobody. The signal exists only in the pre-collective window, and instrumenting the wrong side is why stragglers go undetected in practice.

Response, in order:

1. Alarm with the node ID and the measured ratio.
2. Check the obvious: nvidia-smi clocks (thermal/power throttle), NVLink width, ECC counters.
3. Evict and replace at the next checkpoint — a planned restart costs R, and if the straggler costs 20% then it pays back in R / 0.2 ≈ 50 minutes. That is a calculation the supervisor can make automatically, and expressing the eviction decision as a payback period rather than a threshold is what makes it defensible.

Failure 3: silent data corruption

A GPU that computes wrong answers without erroring. Documented in production at Meta and Google at rates of roughly one device in a thousand over its lifetime. At 1,024 GPUs over 30 days it is not a hypothetical.

Why it is the worst failure in this design: the corruption enters the gradients, the all-reduce averages it into every rank, the optimizer writes it into the weights, and the checkpoint persists it. By the time loss looks strange, every checkpoint for hours is contaminated — and if it never looks strange, you ship a model degraded in ways nobody can attribute.

Three defences, cheapest first:

(a) Cheap invariants, every step.

if not torch.isfinite(loss):                       # NaN/Inf: the loud case
    abort_and_investigate()
if grad_norm > GRAD_NORM_CEILING:                  # 10x the running p99
    skip_update(); count_anomaly()

Nearly free, and catches gross corruption. Skipping one update is harmless; the counter is the signal.

(b) Periodic deterministic replay.

Every N steps, TWO ranks compute the SAME microbatch.
Compare a checksum of their gradients.
Mismatch (beyond expected numerical tolerance) => one of them is lying.
Bisect with a third rank to find which.

Cost: one extra microbatch every N steps. At N=1,000 that is 0.1% of compute for continuous coverage of a failure mode that otherwise has none. The best cost/benefit ratio in the entire design, and it is the kind of mechanism that is obvious once stated and absent from most systems.

(c) A per-node self-test on quarantine. When a node is suspected, run a short GEMM/collective self-test with known answers before returning it to the pool. Catches the deterministic-wrong-answer case that only appears under load.

And the checkpoint-hygiene consequence: keep the last K = 5 validated checkpoints, not one. If SDC is detected at step N, you need somewhere clean to roll back to, and the last checkpoint may already be contaminated. Five checkpoints of 980 GB is 4.9 TB — cheap insurance against having to discard a month of training.

Failure 4: the corrupt checkpoint

A checkpoint written while a rank was failing may be missing shards or contain garbage.

Validation, asynchronous, after every write:
  1. all expected shards present
  2. per-shard digest matches the manifest
  3. TEST LOAD on a spare node -- actually construct the model
  4. sanity: parameter norms within the historical band
  -> mark `validated: true`

`resume` only ever considers validated checkpoints.

Step 4 is the one people skip and the one that catches the subtle case. A structurally valid checkpoint whose parameter norms have jumped 100× is corrupt in a way digests cannot see — digests confirm you read what was written, not that what was written was sane.

8. Failure and Recovery

The last row is the most important operational rule and it is worth defending. An automatic restart loop against a systemic problem — a bad image, a corrupt checkpoint, a code bug — will consume the entire remaining budget in retries while looking like it is trying to help. Three strikes, then stop and ask.

On "keep training when storage is unavailable": the alternative is halting a 1,024-GPU job because a filesystem is down, which converts a storage incident into a training incident. Checkpoint into host memory (~1 GB per rank, easily affordable), continue, and flush later. The exposure is that a failure during the storage outage costs more lost work — so the alarm is genuine, but halting would be worse.

9. Bottlenecks and Evolution

Now: restart time (5.4% at 16,384 GPUs, §6) and stragglers (up to 20% invisibly).

Interventions in order:

1. Async checkpointing with a pinned-memory staging buffer. Cuts C from 5 s to ~0.5 s, which cuts the checkpoint term 10× and lowers T*. Cheap, well-understood, and it should be table stakes.
2. Hot spares + in-memory checkpoint restore (§6). The largest attack on the restart term.
3. Straggler auto-eviction with the payback calculation (§7). Recovers a loss that is currently invisible.
4. Elastic training (torchelastic-style). Continue at reduced world size after a failure rather than halting, then re-expand. Converts an interruption into a slowdown, which is the single largest structural improvement available — and it needs the training code to handle a changing world size, including the dataloader, whose strided assignment was designed for exactly this.
5. Replay-based SDC detection (§7b). 0.1% of compute for coverage of a failure with no other detector.
6. Failure prediction. ECC error rates, thermal trends and NVLink retries often precede hard failures. Draining a node before it kills the job converts an unplanned interruption into a planned one — a 600 s restart becomes a 5 s checkpoint boundary. Speculative, and the highest ceiling on this list.

10. Tradeoffs Explicitly Rejected

Rejected: a fixed checkpoint interval chosen by intuition. §6 gives T* = sqrt(2CM). At 120 min on a large cluster the overhead is 38%.

Rejected: synchronous (blocking) checkpointing. Multiplies C by ~10 and the checkpoint term with it. Async with a staging copy.

Rejected: keeping only the latest checkpoint. SDC detected at step N needs a clean ancestor (§7). Keep 5 validated.

Rejected: resuming from an unvalidated checkpoint. Discovers corruption during recovery, the worst possible moment.

Rejected: liveness heartbeats as the hang detector. A hung rank is alive. Progress watchdog.

Rejected: measuring step time after the collective. Every rank reports the same number and no straggler is identifiable. Measure the compute phase.

Rejected: unlimited automatic restarts. A systemic failure burns the run in a retry loop. Three strikes.

Rejected: restarting onto the node that just failed. Quarantine before reschedule, or the retry loop lands on the same bad hardware.

Rejected: asynchronous SGD to avoid the coupling. It would genuinely reduce the fault impact and it changes convergence, which is not a trade the research side will make at pretraining scale. Named because rejecting it for the right reason is better than not knowing it exists.

Rejected: halting the job when checkpoint storage is unavailable. Converts a storage incident into a training incident. Buffer in host memory.

The Hostile Critique

C1. "In-memory checkpoint restore: 'the other 1,023 ranks still hold their shards'. You use FSDP, so each rank holds 1/1024 of the optimizer state. The failed rank's shard is gone. Where does the replacement rank get its 1 GB, and what have you actually saved over reading from storage?"

C2. "Straggler auto-eviction pays back in R / 0.2 ≈ 50 minutes. You evict, restart, and the replacement node is also slow — because the cause was a hot aisle, or a shared NIC, or a network path. You've now paid R twice and you're still 20% slow. What does your payback calculation say now?"

C3. "Replay-based SDC detection: two ranks compute the same microbatch and compare gradient checksums. Floating-point gradient computation isn't bitwise reproducible across ranks — different NCCL orderings, different kernel selections. What tolerance do you compare at, and what corruption is smaller than that tolerance?"

C4. "Hot spares are 2–3% of the cluster, 'pre-warmed with weights in host memory'. Weights change every step. A spare warmed at step 40,000 is stale by step 41,000. What exactly is pre-warmed, and does it help?"

C5. "'Three strikes then page a human.' Your MTBF at 16,384 GPUs is 3.1 hours — that's more than 3 failures in any 12-hour window, routinely. So you page a human roughly every 10 hours for entirely normal hardware failures. How long before they stop reading the pages?"

C6. "You keep 5 validated checkpoints for SDC rollback. SDC is detected by replay every 1,000 steps. At 2 s/step that's 33 minutes between checks, and your optimal interval at 16,384 GPUs is 13.6 minutes — so corruption can enter, be checkpointed twice, and only then be detected. Is 5 checkpoints enough, and how do you know which one is clean?"

The Revision

R1 — In-memory restore needs redundancy, which must be designed in (answers C1)

The critique is exactly right and it exposes a hole: under FSDP the failed rank's shard is the one piece of state that is not replicated, so "the survivors have it" is false for precisely the data needed.

Change: shard-level replication in host memory, at a chosen replication factor.

Each rank keeps in pinned host memory:
   its OWN latest shard              (~1 GB)
   its NEIGHBOUR's shard (rank+1)    (~1 GB)      <- replication factor 2

Rank r fails -> rank r-1 has r's shard -> peer-to-peer copy to the replacement.

Cost: 2 GB of host memory per rank and one extra shard copy per checkpoint — 980 GB of additional network traffic per checkpoint, which at ~13 min intervals over a fat interconnect is noise.

Benefit, stated honestly: restore from a peer over NVLink/IB is ~1 s versus ~30 s from storage, so this saves ~29 s of a ~600 s restart — less than 5%. The critique is right that the original framing oversold it.

The real win is elsewhere, and the critique surfaces it: in-memory checkpointing lets you checkpoint far more often (a host-memory checkpoint costs ~0.5 s, not 5 s), which lowers T* and the lost-work term. So the mechanism is valuable for a different reason than the one I gave — it attacks C, not R. Reclassifying it is the correction.

And the failure mode that must be handled: a rack-level failure takes rank r and r+1 together, so neighbour replication fails exactly when correlated failures happen. Replicate to a rank in a different failure domain — r + world_size/2 rather than r+1 — which costs nothing and is the difference between a replication scheme and a replication scheme that works.

R2 — Eviction must diagnose before it acts (answers C2)

The critique identifies a real loop: evicting a symptom whose cause is environmental just moves the symptom, and the payback calculation silently assumes the replacement is healthy.

Change: classify before evicting, and the classification is cheap.

def diagnose_straggler(rank):
    n = rank.node
    if n.gpu_temp > THERMAL_LIMIT or n.power_capped:
        # Is it just this node, or the whole aisle?
        peers = nodes_in_same_rack(n)
        if median(p.gpu_temp for p in peers) > THERMAL_LIMIT:
            return ENVIRONMENTAL      # evicting will NOT help. Alert facilities.
        return NODE_THERMAL           # evict; likely a fan or paste problem
    if n.nvlink_width < EXPECTED or n.nvlink_retries > BAND:
        return NODE_LINK              # evict
    if n.nic_shared_utilization > BAND:
        return NETWORK_CONTENTION     # evict AND change placement (m03 topology)
    return UNKNOWN                    # evict ONCE, then stop

And the payback calculation gains a confidence term:

expected_benefit = P(replacement is healthy) x straggler_cost
payback_time     = R / expected_benefit

P(healthy) is MEASURED from the eviction history:
  the fraction of past evictions that actually resolved the straggle.

If evictions have not been resolving stragglers, P falls and the system stops evicting — which is precisely the behaviour the critique's scenario demands, and it is self-correcting from data the system already records.

Plus a hard guard: at most one straggler eviction per hour per job, and never two consecutive evictions of ranks in the same rack without a human. An automatic remediation that can fire repeatedly needs a rate limit on its own authority — the same conclusion as m03's preemption budget, arriving from a different direction.

R3 — Replay must be bitwise, which means constraining the replay, not the tolerance (answers C3)

The critique is right that cross-rank bitwise comparison is not generally valid, and it is right that a loose tolerance defeats the purpose: SDC that flips a low-order mantissa bit is exactly what a tolerance would absorb, and it is exactly what accumulates.

Change: make the replay bitwise-comparable by construction, rather than comparing across naturally-differing ranks.

The replay is a SELF-CHECK, not a cross-rank check:
  every N steps, one rank recomputes ITS OWN microbatch a second time,
  same device, same kernels, same shapes, same order.
  -> deterministic; ANY bit difference is a hardware fault.

Rotate which rank self-checks, so all 1,024 are covered every 1,024 checks.

Same device, same kernel, same shapes ⟹ bitwise determinism. Then the comparison is ==, not allclose, and it detects a single flipped bit. This is a strictly better test than the cross-rank one and it costs the same — one extra microbatch.

What it does not catch, stated plainly: a deterministically wrong GPU — one that computes the same wrong answer twice — passes a self-check. That is a real class of SDC.

So the second layer matters and is now clearly motivated:

Cross-rank check, run rarely (every ~10,000 steps):
  two ranks compute the same microbatch and compare
  at a tolerance derived EMPIRICALLY from the observed
  rank-to-rank spread on known-good hardware (e.g. 5 sigma),
  not from a guess.

Loose, but it catches gross deterministic corruption; the self-check catches everything transient. Two mechanisms with different coverage, and describing what each misses is what makes the pair credible.

Cost: 0.1% for the self-check, 0.01% for the cross-check. Negligible against a failure mode whose alternative detection method is "notice the model is worse, months later".

R4 — What is pre-warmed is everything except the weights (answers C4)

The critique correctly identifies that "weights in host memory" is incoherent for a spare — they would be stale immediately. The valuable pre-warming is everything else, and enumerating it shows it is most of the cost:

PRE-WARMED on a hot spare (all of it stable across steps):
  container image pulled and started              ~60 s saved
  CUDA context + framework init                   ~20 s saved
  NCCL topology detection done                    ~10 s saved
  BASE checkpoint (step 0 / last full) resident   -- structure, not values
  pinned host buffers allocated                   ~5 s saved

NOT pre-warmed (necessarily fetched at swap-in):
  the current shard  (~1 GB)  -- from a peer replica (R1), ~1 s

~95 s of the ~150 s reschedule-and-start phase is pre-warmable, and the 1 GB that is not comes from the neighbour replica in about a second. So the spare's value is real, but it is about process and context, not weights — the critique's correction, absorbed.

And a sharper framing that follows: the spare should be running a no-op member of the process group — joined to a standby communicator, allocations made, kernels JIT'd. It is not "a machine that could join"; it is "a machine that has already joined and is idle". That is what turns swap-in from a start into a substitution.

Cost: 2–3% of the cluster idle, plus the complexity of a standby communicator that most frameworks do not expose. Honest assessment: item 1 (async checkpointing) and item 4 (elastic training) in §9 are both cheaper per point of goodput. Hot spares are the right third move, not the first — and the original ordering, which put them second, was wrong.

R5 — The restart policy must distinguish independent from correlated failures (answers C5)

The critique is right and the original rule is unusable at scale: at MTBF 3.1 h, "3 failures in 30 minutes" is rare but "3 failures in 12 hours" is the norm, and any policy that pages on normal hardware failure will be muted within a week — at which point it protects nothing.

Change: page on the pattern, not the count.

def should_stop_and_page(recent):
    # 1. Same node twice -> quarantine failed. Real bug.
    if any(count(f.node for f in recent) >= 2): return True, "quarantine ineffective"

    # 2. Failures accelerating far beyond the measured baseline.
    if observed_rate_1h > 5 * baseline_rate: return True, "failure rate anomaly"

    # 3. Failing at the SAME STEP repeatedly -> not hardware. Data or code.
    if len(set(f.step for f in recent)) == 1 and len(recent) >= 2:
        return True, "deterministic failure -- data or code, not hardware"

    # 4. No progress: restarts are consuming more time than steps.
    if goodput_1h < 0.3: return True, "goodput collapse"

    return False, None      # independent hardware failures: restart silently

Independent hardware failures are handled silently and counted; correlated ones page. That is the distinction that makes the page meaningful — and rule 3 is the one that catches the case the original policy existed for, a bad batch or a code bug that fails deterministically, which no count-based rule distinguishes from bad luck.

And the routine failures still need to be visible without being a page:

Every interruption -> the /timeline (§3) and a daily digest.
Goodput is a DASHBOARD metric with a weekly trend, not an alert.
Page only on pattern; report everything.

The general rule: alert on what a human must act on now, report everything else. A 30-day run with 233 interruptions has 233 events and perhaps two that need a person — and a system that cannot tell them apart has 233 events that need a person, which means none of them get one.

R6 — Rollback needs a clean-ancestor guarantee, not a fixed count (answers C6)

The critique's arithmetic is right: with self-checks every 1,000 steps (~33 min) and checkpoints every 13.6 min, up to three checkpoints can be written between checks. Five retained checkpoints is only ~68 minutes of history — thin, and worse, nothing establishes which of them is clean.

Change 1 — retention is defined by the check cadence, not by a constant.

retain >= 3 x (sdc_check_interval / checkpoint_interval) + 2   checkpoints
       =  3 x (33 min / 13.6 min) + 2  ≈  9

Plus a MILESTONE checkpoint every 6 hours, retained for the whole run.

The rule is derived from the detection latency, so it stays correct when either cadence changes — which a constant would not.

Change 2 — checkpoints carry a verification watermark, so "clean" is a recorded property.

checkpoint(step=N).last_verified_step = the most recent step at which
                                        an SDC check PASSED

Rollback target = the newest checkpoint whose step <= last_passing_check.

Now "which one is clean" is a lookup, not a judgement call at 3am. A checkpoint written after the last passing check is unverified, and rolling back to it would be rolling back into the suspect window.

Change 3 — tighten the check where it is cheap. The self-check (R3) is one microbatch. Running it every 100 steps instead of every 1,000 costs 1% of compute rather than 0.1%, and cuts detection latency to ~3 minutes — shorter than the checkpoint interval, so at most one checkpoint is ever unverified.

That is the right trade and the original design under-bought it: 1% of compute to guarantee that every checkpoint but the latest is verified, against the alternative of discarding hours of a run — or worse, shipping a model corrupted in a way no eval was designed to find.

And the honest limit: if SDC is detected, rollback discards everything since the last passing check, and the node must be quarantined and tested. If the corruption was deterministic and present for longer than the retention window, the run may be unrecoverable. The defence against that case is not retention, it is the periodic cross-rank check from R3 — which is exactly why both mechanisms exist rather than one.

References

• m03-gpu-cluster-scheduler.md — quarantine, gang scheduling, and where hot spares come from
• m04-training-data-pipeline.md — dataloader state, checkpointed atomically; strided assignment for elastic resize
• m05-eval-harness.md — how a corrupted checkpoint would (and would not) show up in evals
• ../../systems-design/designs/d12-multi-tenant-control-plane.md — supervisor outside the workload; control/data plane separation
• ../../systems-design/WARMUP.md#42-failure-taxonomy — crash-stop vs hang vs Byzantine, which §7 is a concrete instance of
• Young, J. W. A First Order Approximation to the Optimum Checkpoint Interval. CACM 1974 — T* = sqrt(2CM)
• Daly, J. T. A higher order estimate of the optimum checkpoint interval. FGCS 2006 — the refinement for non-negligible C
• Grattafiori, A. et al. The Llama 3 Herd of Models. 2024 — 419 interruptions in 54 days on 16,384 GPUs; the empirical basis for §2
• Dixit, H. et al. Silent Data Corruptions at Scale. Meta, 2021 — SDC rates and detection in production fleets
• Hochschild, P. et al. Cores that don't count. HotOS 2021 — Google's account of the same failure class
• Mohan, J. et al. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. FAST 2021 — async checkpointing and interval tuning
• Wang, Z. et al. GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints. SOSP 2023 — the peer-replica restore in R1