d11 — Distributed Lock / Coordination Service

A fully worked design. Raft in anger, and the design where fencing is not a detail but the entire point.

The most common way this round is failed: designing a lock service that hands out locks correctly and never explains why that is not sufficient for correctness.

Run it first. A companion page builds this as numbered, independently runnable blocks: the zombie write, the fencing token that fixes it, and the TOCTOU window measured against the check-to-write gap: Hands-On — Locking and Fencing, Block by Block. Every number on it was produced by running the code.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Why a Correct Lock Is Not Enough
• 7. Deep Dive B: Sessions, Leases, and the Clock
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We have a bunch of services that need to coordinate — leader election, making sure only one instance runs a nightly job, protecting a resource that can't handle concurrent writers. Build the coordination service they all use."

The trap is that this sounds like a storage problem and it is a safety problem. A lock service that is fast, available and correct-looking can still allow two processes to believe they hold the same lock — not because of a bug, but because it is impossible to prevent from the lock service's side alone.

Saying that in the first two minutes, and then explaining fencing, is how this round is won. Every other part of the design is comparatively routine.

1. Requirements and Scope

Clarifying questions asked

"Is the lock for efficiency or for correctness?" The single most important question, and it changes the answer completely.

• Efficiency ("only one worker should do this expensive computation") — occasional double execution is wasteful, not wrong. A simple lock is fine.
• Correctness ("only one writer to this file") — double execution corrupts data. A lock alone cannot provide this, and §6 is about why.

Assumed: both are needed, and the API must distinguish them so callers cannot get it wrong by accident.

"What is being protected?" Assumed a mix: some resources we control (our own database), some we do not (a third-party API, a filesystem). That distinction determines whether fencing is even possible.

"How long are locks held?" Assumed seconds to hours — a nightly job may hold one for two hours, which rules out short fixed leases without renewal.

"How many locks, how often?" Assumed 100k distinct locks, 10k acquisitions/s. Low by storage standards; high by consensus standards, which is the tension.

Functional

1. Acquire/release a named lock, exclusive or shared.
2. Leader election for a group.
3. Ephemeral registration — a key that disappears when its holder dies (service discovery).
4. Watch for changes (a lock released, a leader changed).
5. Issue a fencing token with every acquisition.

Non-functional

Explicitly out of scope

• General-purpose storage — this is coordination metadata, kilobytes.
• Cross-region coordination (a cross-region lock costs 150 ms and is almost always a design smell; see §9).
• Distributed transactions across services.

2. Scale Numbers

Consensus throughput is the binding constraint. Every acquisition is a write, and every write is a Raft round trip to a majority.

5-node Raft group, same region, ~1 ms RTT to a majority.
Serial:   1,000 writes/s.
Batched:  many writes per consensus round → 20k–50k/s realistic.

Batching is what makes 10k/s feasible, and it is worth naming: a consensus round can carry hundreds of log entries, so throughput is bounded by round-trip rate, not by round-trip latency.

Storage. 100k locks × ~200 B = 20 MB. Trivially memory-resident. Again: this is not a storage problem.

Watches. 10k clients each watching a few keys = ~50k watch registrations. Each is a long-lived connection with a small amount of state — bounded, but it is the thing that scales with clients rather than with data, so it is the first thing to break (§9).

Session heartbeats. 10k clients heartbeating every 3 s = 3.3k heartbeats/s. These must not go through consensus — otherwise heartbeats alone consume a third of the write budget for no coordination value. Handling them locally on the leader with periodic batched consensus is the design (§7).

The latency floor. Acquire = one Raft write = one majority round trip ≈ 1–2 ms same-region. The 20 ms p99 is comfortable, and there is no way to go below ~1 ms without giving up linearizable safety — which is the trade that "fast" lock services silently make.

3. API Surface

# Sessions — the liveness primitive everything else is built on
create_session(ttl=30s)                    -> {session_id}
renew_session(session_id)                  -> {ok, expires_at}
close_session(session_id)                  # releases everything it holds

# Locks
acquire(key, session_id, mode="exclusive", wait=True, timeout=...)
    -> {acquired: bool, FENCE: int, holder: session_id}       ← the fence is not optional
release(key, session_id)
    -> {ok}

# Leader election
campaign(election_key, session_id, value)  -> {leader: bool, FENCE: int}
resign(election_key, session_id)
observe(election_key)                      -> stream of {leader, value, fence}

# Ephemeral keys
put_ephemeral(key, value, session_id)      # vanishes when the session dies
watch(key_or_prefix, from_revision)        -> stream of changes

Four choices worth defending:

• FENCE is returned by every acquisition and is not optional. You cannot get a lock without getting a fence, so a caller has to actively ignore it to be unsafe. Making the safe thing unavoidable is the design.
• Sessions, not per-lock leases. One heartbeat keeps all of a client's locks alive. Per-lock leases means N heartbeats for N locks, which is both wasteful and incoherent — a client could keep one lock alive while another expires, so it holds an inconsistent set.
• wait=True blocks in a queue rather than returning false. Polling for a lock is a thundering herd; a fair queue with notification is strictly better and it removes the retry storm.
• observe returns the fence, so followers know which leader epoch they are seeing. Without it, an observer cannot tell a stale leader announcement from a current one.

4. Data Model

A replicated state machine over a Raft log. The log index IS the fence.

  /locks/{key}      -> {holder_session, mode, fence, acquired_at}
  /elections/{key}  -> {leader_session, value, fence}
  /ephemeral/{key}  -> {value, session}
  /sessions/{id}    -> {ttl, last_renewed, held_keys[]}

  revision = the Raft log index — globally monotonic, totally ordered

The fence IS the Raft log index of the acquisition. This is the elegant part and it is worth stating explicitly:

• It is already globally monotonic and totally ordered — consensus produced it.
• It costs nothing extra to generate.
• It is impossible for two acquisitions to share one.
• It survives leader failover, because the log survives.

A separate counter would need its own replication and its own correctness argument. Using the log index gets it free, and recognizing that is the difference between having read about fencing and having thought about it.

Sessions own keys, and the session is the unit of liveness. When a session expires, every key it holds is released in a single Raft operation — atomically. A client cannot end up holding an inconsistent subset of its locks, which is a real failure mode of per-lock leases.

5. High-Level Architecture

   clients (10k)
        │  session heartbeats (3.3k/s — handled LOCALLY, not through consensus)
        │  acquire / release   (10k/s — through consensus, batched)
        ▼
 ┌───────────────────────────────────────────────────┐
 │  RAFT GROUP (5 nodes, same region)                 │
 │                                                    │
 │   leader:    serializes all writes, batches them   │
 │   followers: replicate; serve linearizable reads   │
 │              via ReadIndex                          │
 │                                                    │
 │   state machine: locks · elections · sessions      │
 │   fence = log index                                │
 └───────────────────┬───────────────────────────────┘
                     │ watch streams
                     ▼
              clients notified

   THE CRITICAL PATH THE LOCK SERVICE DOES NOT CONTROL:

   client ──acquire──▶ lock service ──fence=42──▶ client
                                                    │
                                                    ▼
                                        ┌────────────────────────┐
                                        │  THE RESOURCE          │
                                        │  must check the fence  │  ← DEEP DIVE A
                                        │  and reject fence < max│
                                        └────────────────────────┘

That second diagram is the design. The lock service is the easy half. The half that determines whether the system is actually safe is the resource's participation, and it is outside the service entirely.

The two hard parts — say these at minute 10:

1. A correct lock is not sufficient for correctness, and why.
2. Sessions, leases, and the clock — failure detection is a guess, and the design must be honest about it.

6. Deep Dive A: Why a Correct Lock Is Not Enough

The scenario, narrated

t=0     Client A acquires lock L. Fence = 42. Starts writing to the resource.
t=10    Client A stop-the-world GC pauses. (Or: its NIC drops. Or: the
        hypervisor deschedules it. Or: it swaps.)
t=30    A's session TTL expires. It has not renewed.
        The lock service RELEASES L — correctly, by its own rules.
t=31    Client B acquires L. Fence = 43. Starts writing.
t=45    B finishes and writes its result.
t=50    A WAKES UP. From A's perspective, NOTHING HAPPENED — it does not know
        it paused. It completes its work and writes.

        A's stale write lands AFTER B's correct one and silently overwrites it.

The lock service did nothing wrong. It expired a session that stopped renewing, which is exactly its contract. And yet two clients believed they held the lock and both wrote.

Why you cannot fix this from the lock service

You cannot distinguish a dead process from an unreachable or paused one. That is a theorem, not an implementation gap — from outside, "not responding" is the only observation, and it is consistent with both.

And "check your lease before writing" does not work either:

if lock.still_valid():     # true at this instant
    # ← the pause can land HERE
    resource.write(data)   # now stale

The check and the write are not atomic with respect to time. Making the window smaller makes it rarer, never impossible — and "rare" for a data-corruption bug means "you will find it in production, at scale, and it will be very hard to reproduce."

The fix: fencing tokens, enforced by the resource

Every acquisition carries a monotonically increasing fence. The resource rejects any write whose fence is below the highest it has seen.

t=45  B writes with fence 43.  Resource records highest_fence = 43.  ✓
t=50  A writes with fence 42.  42 < 43  →  REJECTED.                 ✗

The zombie's write is refused, and nobody detected the zombie. Correctness follows from ordering, with no liveness assumption whatsoever. That property — safety without needing to detect anything — is what makes fencing the right answer rather than a mitigation.

Where it must be checked, and this is the part people get wrong

The resource must enforce it. Not the lock service. Not the client.

• If the client checks its own token, you have gained nothing: the zombie client believes its token is current, because from inside the pause no time passed.
• If the lock service checks it, that is just the lease check again — it says nothing about the write that has already left the client.

Concretely:

-- Storage that participates:
UPDATE resource SET data = %s, fence = %s
 WHERE id = %s AND fence < %s;
-- 0 rows updated ⇒ superseded. Do NOT retry; stop.

# Object storage with preconditions:
s3.put_object(Bucket=b, Key=k, Body=data, IfMatch=expected_etag)

And when the resource cannot participate — the honest part

Some resources cannot check a fence: a third-party API with no conditional write, a legacy service, an append-only sink without preconditions. Then you cannot fence, and no lock service can make that operation safe.

The options, in order of preference:

1. Make the operation idempotent with a stable key, so a duplicate is harmless. Best fix.
2. Interpose something you control — write through a small service or a database that can check the fence, and have that be the only writer.
3. Accept at-most-once: do not re-acquire after an expiry until the previous holder is positively confirmed dead. This trades liveness for safety — the work may not happen at all.
4. Accept the risk explicitly for efficiency-class locks, and document that the operation may run twice.

Say which one applies, per resource. A design that claims a lock service provides correctness for arbitrary resources is wrong, and an interviewer who knows this will test for it.

On Redlock

Redlock attempts distributed locking across N independent Redis nodes with a majority quorum. Kleppmann's critique is that its safety argument relies on bounded clock drift and bounded process pauses, neither of which is guaranteed — so the GC-pause scenario above defeats it. Antirez's response is that with fencing tokens, or for efficiency-class locks, it is fine.

The lesson to state: the safety argument lives in fencing, not in the lock protocol. Any lock protocol plus fencing is safe; any lock protocol without it is not. That reframing is more valuable than a position on Redlock specifically.

7. Deep Dive B: Sessions, Leases, and the Clock

Failure detection is a guess, and the TTL is where you set the odds

A session has a TTL; the client renews it. If renewal stops, the session expires and its locks are released.

The TTL is a bet on the maximum tolerable pause:

Renew at TTL/3 so two consecutive missed renewals are tolerable before expiry. That is the standard ratio and it costs nothing.

Spurious expiry is not merely inconvenient — it is the cause of the zombie in §6. A shorter TTL means faster failover and more zombies. So the TTL choice and the fencing requirement are linked: fencing is what lets you choose a short TTL safely. Without fencing you are forced toward long TTLs and slow failover, which is the hidden cost of skipping it.

Whose clock decides

Not the client's. A client with a fast clock believes its lease expired when it has not, or vice versa. Skew across machines makes any cross-machine timestamp comparison unsound.

The leader's monotonic clock decides, and expiry is a write into the Raft log — so:

• Every replica agrees on exactly when a session expired, because they agree on the log.
• Expiry is ordered relative to every other operation, so there is no ambiguity about whether an acquisition happened before or after an expiry.
• A leader failover does not lose expiry state.

The subtlety worth raising: a new leader must not immediately expire every session whose renewal it has not yet seen — those renewals went to the old leader. So a new leader grants a grace period of at least one full TTL before expiring anything. Without it, every leader election causes a mass expiry of healthy sessions, which is a self-inflicted outage on top of the failover. This is a real bug in naive implementations.

Heartbeats must not go through consensus

3.3k heartbeats/s through Raft is a third of the write budget spent on liveness that carries no coordination value.

The design: the leader tracks renewals in memory and only writes to the log when a session actually expires — which is rare.

The correctness argument: in-memory renewal state is lost on leader failover, but that is exactly what the grace period covers. Sessions are conservatively kept alive across a failover (safe: a dead client's lock is released a little later), never conservatively expired (unsafe: a live client's lock is stolen while it works).

Bias every ambiguity toward keeping the session alive, because the cost of a late release is delay and the cost of an early release is a zombie.

Linearizable reads without a write

"Who holds this lock?" must not return a stale answer, but making it a Raft write would double the write load for a read-only question.

ReadIndex is the standard technique: the leader records its current commit index, confirms with a heartbeat round that it is still the leader, then serves the read once its state machine has applied up to that index. One round trip, no log entry. Followers can serve the same way by asking the leader for a read index.

And the cheaper option: lease-based reads, where a leader serves reads directly for a short lease window without confirming. Faster, and it reintroduces a clock assumption — a leader that has been partitioned but whose lease has not expired can serve a stale read. Offer it as a per-read choice, defaulting to ReadIndex, and be explicit that the fast path trades safety for latency.

8. Failure and Recovery

Two rows deserve emphasis.

Client-side self-fencing is the client's half of the contract: if a client cannot renew its session, it must stop doing the protected work immediately, before its lease expires — not after it notices. That does not make it safe by itself (the pause case defeats it), but it dramatically narrows the window, and it is free.

Forced release is safe because of fencing. Without a fence, an operator forcibly releasing a stuck lock could hand it to a second holder while the first is still working — an operator action that causes corruption. With a fence, the first holder's writes are rejected. Fencing is what makes the operational escape hatch usable.

Deliberately accepted: during a partition, the minority side cannot acquire locks and the work that depends on them does not run. I accept that because the alternative — allowing acquisition on both sides — produces two holders, which is the one thing this service exists to prevent. Coordination services must choose C over A, and a coordination service that stays available during a partition is not doing its job.

9. Bottlenecks and Evolution

1. Consensus write throughput, at ~5×. 10k acquisitions/s is fine with batching; 50k/s is at the limit of a single Raft group. Fixes: batch more aggressively (larger windows trade latency for throughput); then shard into multiple Raft groups by key hash — which works because locks are independent, so there is nothing to order across groups. Sharding coordination is much easier than sharding data, and worth noting.

2. Watch fan-out. 50k watches with a leader change means 50k notifications. Fixes: coalesce notifications within a window; a proxy tier that fans out to clients so brokers hold fewer connections. This is the thing that scales with clients rather than with data, so it breaks first as adoption grows.

3. Session heartbeat load. 3.3k/s is fine; 100k clients would be 33k/s of heartbeats, which becomes real load even handled locally. Fix: longer TTLs with proportionally longer renewal intervals, and heartbeat aggregation through the proxy tier.

4. The leader as a single point of throughput. Every write goes through one node. Reads can be served by followers via ReadIndex, which helps read-heavy workloads (leader election observers). Writes cannot be spread without sharding.

5. Cross-region — and the honest answer. A cross-region Raft group means every acquisition costs 100–150 ms. Usually the right answer is not to do it: run a coordination service per region and design so that cross-region coordination is not needed. A cross-region lock is almost always a sign that the work should have been partitioned by region. Saying that is better than designing an expensive mechanism for a requirement that should be questioned.

At 100× clients (1M): the proxy tier becomes mandatory and sessions must be hierarchical — proxies hold sessions with the core and clients hold sessions with proxies. That is a real design change and it is where this stops being a simple service.

10. Tradeoffs Explicitly Rejected

Rejected: locks without fencing tokens. Simpler API, one less thing for callers to thread through. Rejected because it is unsafe and cannot be made safe — §6. This is the one rejection that is not a tradeoff; it is a correctness requirement. The API makes the fence mandatory precisely so this cannot be chosen by accident.

Rejected: Redlock / quorum over independent Redis nodes. Faster and simpler to operate. Rejected because its safety argument depends on bounded clock drift and bounded pauses, neither of which is guaranteed. Flip condition: for efficiency-class locks where a duplicate is wasteful rather than wrong, it is genuinely fine and much cheaper — and I would use it there rather than paying for consensus.

Rejected: a database row as a lock (SELECT ... FOR UPDATE). Uses infrastructure you already have, and it is a perfectly good answer at small scale. Rejected here because it does not give ephemeral keys, watches, or leader election, and because a holder that dies holds the row lock until its connection times out — with no principled TTL. Flip condition: if the only requirement were mutual exclusion against a database you already own, this is simpler and I would use it. Say that; reaching for a coordination service when a row lock suffices is over-engineering.

Rejected: per-lock leases instead of sessions. More granular. Rejected because N locks means N heartbeats, and — worse — a client can end up holding an inconsistent subset of its locks when some expire and others do not. Sessions make liveness atomic across everything a client holds.

Rejected: availability during a partition. Rejected because a coordination service that grants locks on both sides of a partition has failed at its only job. CP, deliberately, and the degradation is "cannot acquire", not "might get two holders".

Rejected: lease-based reads as the default. Faster (no round trip). Rejected as a default because it reintroduces a clock assumption for a service whose entire value is not depending on clocks. Offered explicitly per read, so the caller chooses knowingly.

Rejected: client-side TTL enforcement. Rejected because it makes correctness depend on client clocks, which are the least trustworthy clocks in the system. The leader decides, and the decision is ordered by the log.

The Hostile Critique

C1. "Your fence is the Raft log index. Two different locks, L1 and L2, both protecting writes to the same resource — say a row and a table. L1 is acquired at index 100, L2 at index 90. The resource sees fence 100 then 90 and rejects the second. But they're different locks. What have you built?"

C2. "A new leader grants a full-TTL grace period before expiring anything. The old leader failed because the whole rack lost power, taking 3,000 client sessions with it. Those clients are definitively dead. You now hold every one of their locks for another 30 seconds. What does that do to a system doing leader election for 3,000 shards?"

C3. "Client-side self-fencing: 'stop working when you can't renew.' The client is a JVM mid-GC. It cannot execute your self-fencing code, because it cannot execute any code. What exactly is this buying you?"

C4. "Sessions own keys, and expiry releases them all in one Raft operation. A client holds 500 locks. Walk me through the size of that log entry and what it does to your replication."

C5. "You say cross-region locks are a smell and to partition by region. The thing being protected is a global uniqueness constraint — a username. Partitioning by region doesn't help. Now what?"

C6. "ReadIndex requires a heartbeat round to confirm leadership. Your read latency is now a full round trip, same as a write. So what did ReadIndex actually save?"

The Revision

R1 — Fences are per-resource, not global (answers C1)

The critique finds a genuine and serious modelling error. A monotonic global index is not a monotonic per-resource sequence, and a resource enforcing fence > highest_seen against fences from different locks will reject valid writes and accept invalid ones depending on interleaving.

Change: the fence is scoped to what it protects, and the API makes that explicit.

acquire(key, session_id) -> {fence: {resource: key, epoch: 7}}

• epoch is per-lock-key, incremented on every acquisition of that key. Monotonic within the key, which is exactly the scope in which the resource compares.
• The resource stores highest_epoch per lock key, not one global value.
• The Raft log index is still used underneath to generate epochs safely, but what the caller sees and the resource compares is (key, epoch).

And the design rule that follows, which is the real lesson: one lock per resource. If two locks protect the same resource, they do not exclude each other and the fence cannot help — that is a design error the service should make visible. So the API records which resource a lock protects, and warns when two distinct lock keys declare the same resource.

Cost: a slightly richer fence type and a registry of lock-to-resource mappings. Worth it: the critique describes a corruption bug that would be extremely hard to diagnose, and it is prevented by construction.

R2 — Positive death evidence bypasses the grace period (answers C2)

The critique is right that a blanket grace period is wrong when death is known rather than suspected. 3,000 shards leaderless for 30 seconds after a rack failure is an outage, not a safety measure.

Change: distinguish suspected from confirmed death.

Plus:

1. The grace period applies per session, not globally. A new leader expires sessions with confirmed death immediately and only grants grace to those it is genuinely unsure about.
2. Integrate with the orchestrator, which already knows. A SIGTERM handler that calls close_session() turns an ambiguous disappearance into a clean release — and it is one line in a shutdown hook.

Cost: trusting external death signals introduces a new dependency, and a wrong signal causes a premature expiry — which fencing makes safe rather than catastrophic. That is the point: fencing is what lets you be aggressive about failover. Without it you must be conservative and slow; with it you can be fast and correct.

R3 — Self-fencing is a narrowing, not a guarantee — and the honest version (answers C3)

The critique is entirely correct and I overstated the value. A paused JVM executes nothing, so self-fencing does not help in exactly the case that motivates fencing.

Change: state precisely what it does and does not buy.

What self-fencing genuinely covers — cases where the client is running but disconnected:

• Network partition between client and lock service; the client is healthy and working.
• The lock service is unavailable; the client keeps running.
• The client's renewal is failing due to a bug or misconfiguration.

These are common — arguably more common than long GC pauses — and in all of them the client can execute, so stopping is both possible and correct.

What it does not cover: any pause where the client cannot execute. Only fencing covers that, and no client-side mechanism ever can.

So the revised statement, which is the honest one:

Self-fencing narrows the window for the disconnection case and does nothing for the pause case. Fencing at the resource is the only mechanism that is sufficient. Self-fencing is defence in depth, not a substitute — and a design that relies on it is unsafe.

And a mechanism that does help the pause case: make the write deadline shorter than the session TTL. A write that was issued before the pause and arrives after it is rejected by the resource's own deadline, independent of any fence. Belt and braces, and it costs a timeout setting.

R4 — Bulk expiry must be incremental and bounded (answers C4)

The critique identifies a real operational hazard. 500 lock releases in one Raft entry is a large entry, and if several such sessions expire together — which is exactly what a rack failure produces — the log entries are enormous, replication stalls, and the service becomes unavailable during the failure it is supposed to handle.

Change:

1. Chunk expiry into bounded batches. A session releasing 500 locks becomes 10 entries of 50, applied in sequence. The session is marked expiring in the first entry — so no lock it holds can be re-acquired until the process completes — and released in the rest. Atomicity of the outcome is preserved without one giant entry.
2. Rate-limit expiry processing globally, so a mass failure drains at a bounded rate rather than saturating replication. Locks are released a little later, which is safe.
3. Cap locks per session (say 1,000), returning an error beyond it. A client holding thousands of locks is almost always a design problem — usually it wants one lock over a range, or a different partitioning — and surfacing that is better than silently supporting it.

Cost: expiry of a large session is not instantaneous. Correct: the alternative is a replication stall precisely when the cluster is already handling a failure.

R5 — Global uniqueness is not a lock problem (answers C5)

The critique is a good one because it catches me dismissing a legitimate requirement with a heuristic. Username uniqueness is genuinely global and genuinely cannot be partitioned by region.

Change: it should not be solved with a lock at all. Three better options, in order:

1. A uniqueness store with a compare-and-swap — a single globally-consistent store (which d08 already provides) holding username → user_id, written with if_not_exists. One write, atomic, no lock, no lease, no zombie. The write is the mutual exclusion.
2. Partition by the constrained value, not by region. hash(username) selects a shard whose leader may be in any region; the lock becomes local to that shard's leader. Contention on one username is inherently serialized anyway, so cross-region latency is paid only by the (rare) contended case.
3. Optimistic with reconciliation — allow regional claims and detect conflicts asynchronously. Right only when a conflict is recoverable (offer an alternative name); wrong for anything irreversible.

The general lesson worth stating: a lock is the right tool when you must exclude concurrent execution; a conditional write is the right tool when you must exclude concurrent state. Reaching for a distributed lock to enforce a uniqueness constraint is using the harder mechanism for the easier problem — and it is a very common mistake.

R6 — ReadIndex batches; the write does not (answers C6)

The critique is right that I described the mechanism without stating the benefit, which made it look pointless.

What ReadIndex actually saves:

1. No log entry. A read produces no Raft log entry, so it does not consume write throughput, does not grow the log, does not require fsync, and does not need to be replicated to disk on every follower. At 50k reads/s that is the difference between a working service and a log that grows by gigabytes an hour for read-only questions.
2. Heartbeat rounds batch across many reads. One confirmation round serves every read waiting at that moment. At high read rates the amortized cost approaches zero round trips per read, while a write cannot be amortized the same way because each needs its own log position.
3. Followers can serve reads using a read index obtained from the leader, so read capacity scales with the cluster while write capacity does not.

So the latency is similar to a write; the cost is not, and it is the cost that matters at scale. That is the correct statement and I should have made it.

And the fast path, offered explicitly: lease-based reads skip the confirmation entirely, giving sub-millisecond local reads at the price of a clock assumption. Per-read, defaulted off, documented as trading safety for latency — which is the same shape as every other choice in this design.

References

• ../WARMUP.md#chapter-4-leases-fencing-and-the-zombie — the fencing argument from zero
• ../WARMUP.md#chapter-6-consensus--raft-at-usable-depth — Raft, including both safety rules
• d01-job-scheduler.md — leases and fencing applied to job dispatch
• d08-multi-region-metadata.md — the store R5 suggests for uniqueness
• Burrows, M. The Chubby Lock Service for Loosely-Coupled Distributed Systems. OSDI 2006 — the paper for this design; sequencers are fencing tokens, and §2.4 explains why they exist
• Kleppmann, M. How to do distributed locking. https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html — the fencing argument and the Redlock critique
• Ongaro & Ousterhout. In Search of an Understandable Consensus Algorithm. USENIX ATC 2014 — §6.4 covers ReadIndex
• Hunt et al. ZooKeeper: Wait-free Coordination for Internet-scale Systems. USENIX ATC 2010 — sessions, ephemeral nodes, watches
• etcd documentation — lease, election, and concurrency APIs; a good model for the API in §3
• Fischer, Lynch, Paterson. Impossibility of Distributed Consensus with One Faulty Process. JACM 1985