The Cross-Cutting Map

Six mechanisms account for most of what the five hands-on pages measure, and each shows up on three or more of them wearing different clothes. Recognising the same mechanism across substrates is the transferable skill; the pages are only where it was measured.


Table of Contents


The Six Recurring Mechanisms

MechanismC01C03C05C11M02
The ack positionB1–B2B5 (deadline)B2 (lease expiry)
TOCTOUB5 (dual write)B6 (GET-then-SET)B4
Fencing / epochsB6B3
The saturating signalB3B6
Reserved floorsB5 (per-key)B6
Approximate then exactB4 (Bloom)B4 (counter)B5 (tiering)

1. The Ack Position, and Why There Is No Third

The shape: an effect and a record-that-the-effect-happened are two events. One must come first. A failure between them is the only failure, and which side you lose on is decided entirely by the ordering.

PageThe two eventsBefore →After →
C01 B1–B2apply the job / ack the queue1.99% lost1.99% duplicated
C01 B5apply the effect / write the dedup keyduplicates escape at exactly the crash rate
C11 B2acquire the lease / do the worklease expires mid-worksplit brain

Why it matters: candidates try to find a third position. There is not one — this is the two-generals result, and recognising it saves the ten minutes otherwise spent inventing a protocol that cannot exist.

What to do instead: pick at-least-once and make the second application a no-op. C01 B3 measures that it works; B5 measures the one condition it requires — the dedup key must be written in the same transaction as the effect, or the window has only moved.


2. Time-of-Check to Time-of-Use

The shape: a check establishes a fact about the past; the action depends on a fact about the present. Any delay between them is a window, and the window's size is a latency you usually do not control.

PageThe checkThe gapMeasured
C03 B6GET the counternetwork RTT before the SET10 admitted against a limit of 5
C11 B4read the current fence tokenany pause before the writecorrect or wrong depending only on where the pause lands
C11 assemblyas above, swept1 ms → 1 sleaks 1.2% → 72.5%
C01 B5"have I seen this job?"the crash windowdedup catches 0 of the crash duplicates

The rule: a check and the effect it guards must be atomic, which means they must happen at the same component. Any design where X validates and Y acts has this window.

How to find it in a design: ask which component orders the operations. That is the one the check belongs in. In C03 it is Redis (INCR, not GET-then-SET); in C11 it is the resource (fence check inside the write); in C01 it is the database (dedup key in the same transaction).


3. Fencing and Epochs

The shape: any identity that can be reused, superseded or restarted must carry a monotonically increasing number, and the component that acts on the identity must reject anything below the highest it has seen.

Page / designThe identityThe epoch
C11 B3lock holderfence token from acquire
C01 B6job leasevisibility-timeout generation
d02shard ownerrebalance epoch
m03 R5a node hostnameboot epoch
Raftleaderterm
ZooKeepersessionzxid

The measured claim (C11 B3): with fencing, the stale writer's update is rejected and the correct value survives — and it is rejected without the resource talking to the lock service, which is what makes it robust to the lock service being slow, partitioned or down.

The honest limit (C11 B3, beyond the toy): fencing requires the resource to cooperate. A third-party API will not check your token. When it cannot, you do not have a safe design — you have a probabilistic one, and the correct move is to say so and reach for idempotency instead.


4. The Saturating Signal

The shape: the metric everyone reaches for is bounded above, and it saturates exactly at the point where you need it to keep moving.

PageThe tempting signalWhere it diesWhat to use
C05 B3CPU utilisationpinned at 100% from 99 rps to 120 rps, while queue depth goes 51 → 1,617queue depth, or measured wait
M02 B6GPU utilisationreads ~100% across the whole preemption cliffKV occupancy
C03 B1request count1 request/min at 128k context vs 1000/min at 200 tokenstokens, or KV·seconds

The general form: the utilisation of a resource is not the scarcity of that resource. A signal fit for shedding must be unbounded above and must lead rather than trail. Error rate fails the second test; utilisation fails the first.


5. Reserved Floors Over Strict Priority

The shape: two classes contend; strict priority protects the top class by starving the bottom one to zero, and zero is not a degradation, it is an outage for that class.

Page / designClassesStrict priority gives the bottom class
C05 B6premium / free5.8% completion — measured
d05shed classesstarvation under sustained load
m01enterprise / standard / freeand its revision: a floor must guarantee latency, not merely admission
m07hot / long-tail adaptersthe tail never enters a batch
m03quota tierslarge jobs never schedule without aging

C05 B6 measures the middle ground: a floor recovers 87% of strict priority's gain for the top class while taking the bottom class from 5.8% to 39.2%.

The question to ask whenever a design reaches for priority: what is the bottom class guaranteed? If the answer is "nothing", it will eventually get nothing — and the guarantee you can write in a contract is the floor, not the priority.


6. Approximate in Fast Memory, Exact in Slow

The shape: an exact answer is expensive; a cheap approximate test that is wrong in only one direction guards it.

PageThe approximationThe direction it is safe in
C03 B4two counters instead of a timestamp logover-estimates recent load → over-admits, bounded
C03 B5local lease, then the storeover-admits by lease × processes
C01 B4Bloom filter before the dedup tablemust answer definitely new; a false positive falls through to the exact check
M02 B5fetch a cached prefix, else recomputea miss costs recompute, never wrong output

The direction is the whole design. A Bloom filter used the wrong way round — treating "probably seen" as "seen" — skips a job, which is data loss. The same structure is safe or unsafe depending only on which way the error points, and C01 B4's beyond the toy is the worked version.


The Numbers Worth Memorising

These come up in more than one round, and every one was measured on its page.

NumberWhat it isPage
what a fixed window admits at the boundary — and the sliding counter's worst caseC03 B1, B4
8.7×p99 latency increase from 50% to 95% utilisationC05 B1
1/(1−ρ)queueing delay as a multiple of service timeC05 B1
62×goodput gain from dropping work whose deadline has passedC05 B5
320 KiBKV cache per token, 70B with GQA-8 at fp16M02 B1
23%share of a 4×H100 replica held by one 128k-context requestM02 B1
9.3 GB/sbreak-even to fetch cached KV rather than recompute, TP4M02 B5
batch increase from paged over contiguous KV allocationM02 B3
989.5 TFLOP/sH100 BF16 dense — the datasheet's 1,979 is with 2:4 sparsityM02 B5
~0cost of a fence check at the resourceC11 B3

What Every Page Ends Up Saying

Read together, the five reach the same conclusion from five directions:

Put the check where the ordering happens, and make the failure cheap rather than rare.

  • C01: the dedup key goes in the transaction that performs the effect — not in the worker, not in the queue.
  • C03: the decrement goes in the store that orders the requests — one INCR, not GET-then-SET.
  • C05: the deadline check goes at dequeue, where the capacity is about to be spent — not at enqueue, where nothing has waited yet.
  • C11: the fence check goes in the resource that orders the writes — not in the client that is about to be descheduled.
  • M02: the admission check goes on KV occupancy, the resource that actually binds — not on the utilisation metric that saturates.

And in each case the second half matters as much as the first. C01 does not prevent duplicates, it makes them no-ops. C05 does not prevent overload, it makes the failure a countable drop instead of an invisible 18-second queue. C11 does not prevent zombies, it makes their writes harmless. A design that makes the bad case cheap beats one that makes it rare, because rare failures are the ones nobody has tested.