d08 — Multi-Region Metadata Store

A fully worked design. Where consistency stops being free. Every other design in this set assumed a single region and said "multi-region is different"; this is the one that says how.

The physics is the constraint: light takes ~40 ms round trip across the US and ~150 ms across the Pacific, and no amount of engineering removes it.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Where the Write Goes
• 7. Deep Dive B: Reading Your Own Writes Across Regions
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We're going multi-region. We need a store for the metadata that everything else depends on — service configuration, feature flags, routing rules, tenant settings, API keys. It has to be readable everywhere with low latency, it has to be consistent enough that a config change doesn't cause an outage, and it can't be the thing that takes us down."

Three clauses in tension. Readable everywhere with low latency wants local replicas. Consistent wants coordination. Can't take us down wants no hard dependency. You cannot have all three at full strength, and the design is about which axis to bend where — which is exactly PACELC.

The reframing that unlocks it: this is a read-mostly, small-data, high-consequence store. Every one of those three properties is unusual, and each one buys you something.

1. Requirements and Scope

Clarifying questions asked

"What's the read/write ratio?" Assumed 10⁶ : 1 — config is read on every request and changed a few times a day. That ratio is the single most important fact here: it means you can pay almost anything on writes to make reads free.

"When a config change is made, how fast must it apply, and must it apply everywhere at once?" Assumed < 10 s globally, and — crucially — not atomically. A flag that is on in us-east and off in eu-west for 3 seconds is acceptable for most config, and not acceptable for some. So the API must distinguish them.

"What happens if the store is unreachable?" Assumed services must keep running on the last known config. This is the "can't take us down" clause and it drives the entire read path.

"Which is worse: a stale config or no config?" Stale, overwhelmingly. That answer selects the design.

Functional

1. Read config by key, from any region, at very low latency.
2. Write config with strong consistency — no lost updates, no split-brain.
3. Watch for changes (push, not poll).
4. Atomic multi-key updates for changes that must apply together.
5. History and rollback — what was this an hour ago, and put it back.

Non-functional

Explicitly out of scope

• User data, application state, anything high-volume. This is metadata — small, read-mostly, high-consequence.
• Secrets management (different threat model, different rotation).
• Service discovery of individual instances (churns far too fast for this store).

2. Scale Numbers

The physics first, because it bounds everything:

A 5-region Raft group with a quorum spanning continents has a write latency of at least the median RTT to a majority — 100–150 ms. No implementation removes that. It is speed of light plus routing.

Reads. 10M reads/s across the fleet. At 10 GB of data, every read can be served from process-local memory. That is the observation that makes the whole design work: 10 GB fits in RAM on every server, so the steady-state read path involves no network at all — p99 is sub-microsecond, not sub-millisecond.

Writes. ~1,000/day. That is 0.01/s. At that rate, a 150 ms write latency is completely irrelevant — nobody notices, and there is no throughput concern whatsoever. So spend everything on write correctness and nothing on write speed.

Say this arithmetic out loud; it inverts the instinct to optimize writes.

Propagation. A 10 KB config change to 10,000 servers is 100 MB of fan-out. Via a per-region hierarchy (global → region → rack → host) it is 10 KB per hop and a few hundred milliseconds. Direct fan-out from a global store to 10,000 servers would be 10,000 concurrent connections to one place — a self-inflicted thundering herd on every change.

Memory per host. 10 GB is too much for every process if a host runs 50 processes. So: a per-host agent holds the full copy; processes read from it over a unix socket (~20 µs) or mmap a shared read-only snapshot (~100 ns). The mmap option is what gets you to sub-microsecond, and it is worth naming.

3. API Surface

# Read — served locally, no network in the steady state
get(key, consistency="bounded")            -> {value, version, staleness_ms}
    consistency: "bounded"      local cache; may be up to 10 s stale (default)
                 "linearizable" round trip to the leader; ~150 ms cross-region
                 "read_my_writes" local, but blocks until version >= my last write

# Write — always through the global leader
put(key, value, if_version=None)           -> {version}          | 409
txn(writes: {...}, if_versions: {...})     -> {version}          | 409   atomic
delete(key, if_version=None)               -> {version}

# Watch — push, not poll
watch(prefix, from_version)                -> stream of {key, value, version}

# History
history(key, limit)                        -> [{version, value, actor, at}]
rollback(key, to_version)                  -> {version}

Four choices worth defending:

• Consistency is a per-read parameter, not a system property. 99.99% of reads want bounded and sub-microsecond; a few want linearizable and will pay 150 ms. Forcing one choice system-wide means either everything is slow or nothing is safe. This is the single most important API decision here.
• staleness_ms on every read. The caller can decide whether this value is fresh enough for what they are about to do. Hiding staleness is how a 10-second-stale flag causes an incident nobody can explain.
• if_version everywhere — every write is optionally a compare-and-swap, because two operators editing the same flag is a routine event, not an exotic one.
• watch streams, never poll. 10,000 hosts polling a global store every second is 10,000 req/s of pure waste for a store that changes 1,000 times a day.

4. Data Model

Logical:
  key       "/svc/{service}/config/{name}"     hierarchical, prefix-watchable
  value     opaque bytes (JSON in practice), < 1 MB
  version   globally monotonic (the Raft log index)
  metadata  actor, timestamp, comment, ttl?

Storage — a single global Raft group (5 or 7 members) holding the full keyspace.
  10 GB fits comfortably in memory on every member.

Per-region: read-only followers (learners), asynchronously replicated.
Per-host:   an agent with a full local snapshot + a watch stream.

One Raft group for everything, not sharded. At 10 GB and 0.01 writes/s there is no throughput reason to shard, and sharding would destroy the property that makes §7 work: a single global version number that totally orders every change. Sharding metadata is a classic over-engineering tell — you inherit cross-shard transaction complexity to solve a throughput problem you do not have.

Version is the Raft log index. It is already globally monotonic and totally ordered, it costs nothing, and it gives you "config as of version N" for free — which is what makes rollback and audit trivial.

Hierarchical keys so a service watches /svc/payments/ and receives only what concerns it. Watching everything means every host wakes on every change, which at 10,000 hosts is a small thundering herd on each write.

5. High-Level Architecture

                    WRITES (0.01/s)                     READS (10M/s)
                          │                                   │
                          ▼                                   │
        ┌─────────────────────────────────────┐               │
        │  GLOBAL RAFT GROUP (5 members)      │               │
        │  us-east · us-west · eu · ap · ap2  │               │
        │  leader in the region with the      │               │
        │  most write traffic                 │               │
        └───────────────┬─────────────────────┘               │
                        │ async replication (learners)        │
        ┌───────────────┼───────────────┬─────────────────┐   │
        ▼               ▼               ▼                 ▼   │
   ┌─────────┐    ┌─────────┐    ┌─────────┐        ┌─────────┐│
   │ us-east │    │ us-west │    │  eu-w   │        │  ap-se  ││
   │ replica │    │ replica │    │ replica │        │ replica ││
   └────┬────┘    └────┬────┘    └────┬────┘        └────┬────┘│
        │ watch stream, hierarchical fan-out              │    │
        ▼                                                 ▼    │
   ┌──────────────────────────────────────────────────────┐   │
   │  Per-host agent: full snapshot on local disk + mmap   │◀──┘
   │  Processes read via mmap: ~100 ns, NO NETWORK         │
   └──────────────────────────────────────────────────────┘

The whole design in one sentence: writes go through global consensus because they are rare and consequential; reads never leave the process because the data is small and stale is acceptable.

The two hard parts — say these at minute 10:

1. Where does the write go, and what does the answer cost every region that is not the leader.
2. Read-your-writes across regions — the guarantee that actually breaks in practice, and the one users notice.

6. Deep Dive A: Where the Write Goes

The problem. Linearizable writes require a majority. With members in us-east, us-west, eu-west, ap-southeast and ap-northeast, a majority is 3 of 5 — and the latency is the median RTT from the leader to that majority.

Leader placement is a latency decision, and it is asymmetric. Placing the leader in us-east gives 85 ms; in ap-southeast it gives 170 ms. That is a 2× difference from one config setting.

Option 1 — Leader in the busiest write region (chosen)

Most config changes come from where the operators are. Place the leader there; everyone else pays a cross-region hop to write, which at 0.01 writes/s nobody notices.

• ✅ Optimal for the common case, and trivially simple.
• ❌ That region's failure means a leader election (~1 s) and a latency change.
• ❌ Operators in other regions see slower writes — irrelevant at this rate.

Option 2 — Regional partitioning by key ownership (rejected)

Shard the keyspace so each region owns some keys and leads their group.

• ✅ Local writes for locally-owned keys.
• ❌ Destroys the global version ordering, which §7 depends on entirely.
• ❌ Cross-region atomic updates become 2PC.
• ❌ Solves a throughput problem that does not exist at 0.01 writes/s.

Rejected as over-engineering. Worth raising and dismissing explicitly, because it is the answer an interviewer expects you to reach for.

Option 3 — Witness replicas (the refinement)

The insight: a Raft member needs to vote and store the log, but does not need to serve reads. So place a lightweight witness in a cheap third location that is network-close to the leader.

us-east (leader) · us-west · eu-west · ap-southeast · witness in us-central

Majority = 3.  Fastest: leader + us-central (15 ms) + us-west (65 ms) → ~65 ms.

Write latency drops from 85 ms to ~65 ms, and — more importantly — the quorum no longer depends on a transatlantic hop being healthy. This is what Spanner and CockroachDB do with non-voting/witness replicas, and it is a genuinely good detail to know.

What the non-leader regions actually pay

A write from eu-west: 85 ms to reach the leader, 65 ms for the leader's quorum, 85 ms back = 235 ms. At 0.01 writes/s and a human clicking a button, that is fine — and saying "it's fine, here's the arithmetic" is much stronger than optimizing it.

Where it is not fine: an automated system doing config writes in a loop — an autoscaler updating capacity, say. That is a different access pattern and should not be in this store. Naming that boundary is the important part: this store is for human-rate, high-consequence metadata. Machine-rate state belongs elsewhere.

7. Deep Dive B: Reading Your Own Writes Across Regions

The failure that users actually hit, and the one a naive design gets wrong:

t=0     Operator in eu-west sets a flag: PUT /flags/new-checkout = true
t=235ms Write commits globally. The UI shows success.
t=236ms The UI reloads the flag from the eu-west REPLICA.
        The replica is asynchronous and hasn't received it yet.
        The UI shows: false.
t=1.2s  Replication arrives. The flag reads true.

The operator saw their own write fail. They toggle it again. Now there are two writes, and if the second raced the first's propagation, the final state may be wrong. This is a real, common, confidence-destroying bug.

The fix: version tokens

The write returns its version (the Raft log index). Subsequent reads carry it:

resp = put("/flags/new-checkout", True)      # -> version 48211
value = get("/flags/new-checkout",
            consistency="read_my_writes",
            min_version=resp.version)        # blocks until the local replica
                                             # has applied >= 48211

The local replica either has it (return immediately) or waits — bounded by the replication lag, typically well under a second. If it exceeds a timeout, it falls back to a leader read and returns the truth with a slow: true flag rather than a stale value or an error.

This is a session guarantee (read-your-writes), and it is far cheaper than linearizability: it requires no coordination, just a version comparison. Session guarantees are the ones users actually notice, and they cost almost nothing — that sentence is worth saying.

Making it automatic

Requiring every caller to thread version tokens is a footgun. The client library holds the highest version it has observed and attaches it to every subsequent read from that session. Callers get read-your-writes and monotonic reads for free, and never see the mechanism.

This is exactly how Spanner's client, DynamoDB's session tokens, and MongoDB's causal-consistency sessions work — and it generalizes: causal consistency across an entire session, from one integer.

The staleness that remains, and being honest about it

Even with session guarantees, region B does not see region A's write until replication arrives. For most config that is correct and fine. For some it is not:

Not fine:  "disable this feature globally, NOW" — a kill switch
Not fine:  two configs that must change together across regions
Fine:      "roll out to 10%"
Fine:      "update the retry timeout"

So the API must let a writer demand global visibility:

put("/flags/kill-switch", True, wait_for_global=True)   # returns when EVERY
                                                        # region has applied it

It costs the slowest region's replication lag — up to a second — and it is exactly what a kill switch needs. Anything that does not ask for it does not pay for it.

And the reader-side dual: a consumer whose correctness depends on freshness can require max_staleness_ms, failing rather than serving a value that is too old. Failing closed on staleness is right for a kill switch and wrong for a timeout tunable — which is why it is a per-read parameter, and why the default must be permissive.

8. Failure and Recovery

The single most important row is the first one. A region that loses connectivity to the rest of the world keeps serving traffic from its local snapshot. The metadata store never takes down a region, which was requirement three. It degrades to "config is frozen at the last known good value", which is almost always survivable.

And the config-change safety mechanism, which deserves its own note:

1. Write with a rollout policy:  put(key, value, rollout="canary")
2. Applies to 1% of hosts first — the agent decides by consistent hash
3. Health metrics watched for N minutes
4. Auto-promote, or auto-rollback on regression

A config store without staged rollout is a global outage waiting to happen — a config change is a deploy, and it deserves the same care. It is also the fastest possible global change, which is exactly why it is dangerous: no build, no test, no bake time, applied everywhere in 10 seconds.

Deliberately accepted: during a regional partition, that region's config is frozen and its operators cannot write. I accept that because the alternative — allowing local writes and reconciling later — means two regions can set the same flag to different values with no principled merge, and a config store with conflicting values is worse than a frozen one.

9. Bottlenecks and Evolution

1. Watch fan-out on a broad change. A change to a key that 10,000 hosts watch is a 10,000-way fan-out. The regional hierarchy handles it, but a change to a prefix everyone watches (a global default) is still a global wake-up. Fix: coalesce changes into a batched update every N seconds — since propagation SLO is 10 s, batching at 1 s costs nothing and cuts fan-out events by up to 10×.

2. Snapshot size growth. 10 GB is fine; 100 GB is not — it stops fitting comfortably on every host and the initial sync becomes slow. Fix: agents subscribe to prefixes, not everything, so a host holds only the config for the services it runs. This should be the design from the start if growth is expected.

3. History growth. Every version retained forever means the Raft log grows without bound. Fix: snapshot + log compaction (standard Raft), with the full history for the last 90 days in a separate append-only store for audit. Rollback needs recent history, not all of it.

4. Raft group membership across regions. Adding a sixth region means a membership change, and naive reconfiguration can produce two disjoint majorities. Fix: joint consensus or one-at-a-time changes, and do it during a maintenance window — it is a rare operation and does not need to be seamless.

At 100× data (1 TB): this stops being a metadata store and becomes a database, and the design does not stretch. Prefix-subscribed agents help; beyond that you need sharding, and sharding costs the global version ordering that §7 depends on. Saying "at that point it's a different system" is the honest answer, and the boundary is worth stating up front.

10. Tradeoffs Explicitly Rejected

Rejected: multi-master / active-active writes. Every region accepts writes and they reconcile. Rejected because there is no principled merge for config: if us-east sets a flag true and eu-west sets it false concurrently, last-write-wins picks by clock skew and the loser's change silently vanishes. For config, a lost change is an outage waiting to happen. Flip condition: if the data were genuinely commutative (counters, sets), a CRDT would make this correct and I would take it.

Rejected: sharding the keyspace by region. Local writes for locally-owned keys. Rejected because it destroys the single global version ordering that makes session guarantees, atomic multi-key writes and rollback simple — to solve a throughput problem that does not exist at 0.01 writes/s. Flip condition: at machine-rate writes (>1k/s), sharding becomes necessary and the design changes shape.

Rejected: reading from the leader for consistency. Always correct. Rejected on latency — 150 ms cross-region on the read path, on a store read on every request, is unusable. Offered as a per-read option for the rare caller who needs it.

Rejected: polling instead of watching. Simpler, no long-lived connections. Rejected on arithmetic: 10,000 hosts × 1/s = 10,000 req/s against a store that changes 1,000 times a day. That is a 10⁶ waste ratio. Flip condition: with tens of hosts rather than tens of thousands, polling is genuinely simpler and I would use it.

Rejected: a single global etcd/Consul cluster with direct client reads. The obvious answer. Rejected because clients reading directly makes the store a hard dependency on every request — if it is slow or unreachable, everything is. The per-host agent with a local snapshot is what turns it into a soft dependency, and that is requirement three.

Rejected: strong consistency on reads by default. Rejected on PACELC grounds: even with no partition, we choose latency over consistency for reads, because a 10-second-stale config is almost always fine and a 150 ms read is never fine. The API surfaces the choice; the default is the common case.

Rejected: using timestamps for ordering. Rejected because it would make correctness depend on clock synchronization. The Raft log index is already a total order, costs nothing, and is immune to skew.

The Hostile Critique

C1. "Your kill switch uses wait_for_global=True. A region is partitioned. The kill switch write waits for a region that will never acknowledge. What does the operator see, and what would you have them do at 3am with a live incident?"

C2. "Per-host agents hold a full snapshot and processes mmap it. You push a config change and an agent applies it while a process is mid-read. Walk me through the memory ordering."

C3. "Staged rollout by consistent hash of the hostname. The 1% canary happens to be 100 hosts all in us-east because that's where your hash landed. Your canary metrics look fine. Then it goes to 100% and eu-west falls over. What did your canary actually test?"

C4. "You said clock skew is irrelevant because ordering is the Raft index. But staleness_ms is computed from a timestamp. Which clock, and what does a reader see when it's wrong?"

C5. "10 GB in RAM on every host, mmap'd. A host runs 50 processes and you're proud that they share it. What happens during the ~1 second when the agent is writing the new snapshot and the old one is still mapped?"

C6. "A service reads config on every request from mmap. Someone deletes a key. Walk me through what that service does on its very next request."

The Revision

R1 — wait_for_global must degrade, not hang (answers C1)

The critique describes exactly the wrong behaviour at the worst moment. A kill switch that hangs during a partition is a kill switch that does not work when you need it.

Change: wait_for_global takes a timeout and a policy, and it reports partial success.

put("/flags/kill-switch", True,
    wait_for="quorum_regions",     # committed globally + applied in a majority of regions
    timeout=2.0)
-> {version: 48211,
    applied_in: ["us-east", "us-west", "eu-west"],
    pending:    ["ap-southeast"],          # partitioned
    status:     "partial"}

Three policies, and the middle one is the right default for a kill switch:

The key insight: the write is already durably committed by Raft the moment the majority acknowledges. A partitioned region will apply it the instant connectivity returns. So the operator's kill switch has taken effect everywhere reachable, and the unreachable region is — by definition — not serving traffic that the operator can reach either.

What the operator sees: a clear "applied in 3 of 4 regions; ap-southeast is partitioned and will apply on reconnect." That is actionable at 3am. A spinner is not.

R2 — Snapshot swap must be atomic, not in-place (answers C2)

The critique names a genuine data race that I had glossed. Writing into a mapped region while readers are reading it is undefined behaviour, and the failure would be rare, non-deterministic, and horrifying to debug.

Change: never mutate a mapped snapshot. Swap atomically instead.

1. Agent writes the new snapshot to a NEW file:  config.48211.snap
2. fsync it.
3. Atomically update a small pointer file (or a symlink) via os.replace.
4. Readers detect the version change on their next read boundary,
   mmap the new file, and drop the old mapping.
5. The old file is unlinked once its refcount reaches zero — the OS keeps the
   pages alive for anyone still mapping it, which is exactly the semantics we want.

Within a read, the process holds a stable mapping for the whole operation, so it sees a consistent snapshot — never a torn one. Consistency is per read, which is the right granularity: a request handler that reads five keys sees all five from the same version.

And the version is checked cheaply: a single atomic load of a version counter in a tiny shared page, so the common case (no change) is one cache-line read.

Cost: briefly two snapshots on disk and in page cache — 20 GB instead of 10 for a second. Fine, and it is the standard copy-on-write swap that every configuration system converges on.

R3 — Canary must be stratified, not hashed (answers C3)

The critique is right and the flaw is a real one: a hash-based 1% is a random 1%, and random sampling of a heterogeneous population does not test the population.

Change: rollout stages are explicit and stratified, not percentage-based.

stage 1:  one host per region              (~5 hosts)   2 min
stage 2:  one AZ per region                (~5%)        5 min
stage 3:  one full region (the smallest)   (~15%)      10 min
stage 4:  all remaining regions            (100%)

Every stage covers every region, so a region-specific failure surfaces at stage 1 rather than at 100%. And the stages are ordered by blast radius, not by percentage.

Plus the guard that matters more than the schedule: auto-rollback on per-region health regression, not aggregate. An aggregate metric across four healthy regions and one broken one looks fine — which is precisely how the critique's failure happens, and it happens with any aggregate-metric canary.

Cost: slower rollouts (about 17 minutes to full). Correct for config, which is the fastest and therefore most dangerous change mechanism you have. An emergency override exists for kill switches, and it is the one thing that skips staging — deliberately, and with an audit record.

R4 — Staleness must be measured in versions, not seconds (answers C4)

The critique catches a genuine inconsistency: I removed clocks from the correctness path and then put one back in the observability path, where readers make decisions with it.

Change: the primary staleness measure is version lag, which needs no clock.

staleness = {
  "version_lag": 3,            # versions behind the leader's last known commit
  "applied_version": 48208,
  "leader_version": 48211,     # from the replication stream, not a clock
  "approx_seconds": 1.4        # derived, ADVISORY ONLY
}

• max_staleness policies are expressed in versions, which is exact and skew-immune.
• approx_seconds is computed from the replica's own monotonic clock measuring how long since it last received an update — a local duration, not a cross-machine timestamp comparison. It is immune to skew because no two clocks are compared.
• It is labelled advisory, and the API documents that any correctness decision must use version_lag.

Cost: version lag is less intuitive to a human ("3 versions behind" vs "1.4 seconds"). Solved by showing both and being explicit about which one is load-bearing.

The general lesson worth stating: if you have removed clocks from your correctness path, check whether you have quietly reintroduced them through a metric that someone will make decisions with.

R5 — Bound the swap cost with delta application (answers C5)

The critique is right that a full 10 GB rewrite per change is absurd for a change that touches a few kilobytes — 1,000 changes/day × 10 GB is 10 TB/day of pointless disk writes, plus a page cache that doubles during every swap.

Change: apply deltas to the snapshot; rewrite the full snapshot rarely.

config.base.snap          full snapshot, rewritten daily (or after N deltas)
config.48209.delta        small, append-only
config.48210.delta
config.48211.delta

Reader maps the base + the deltas, applying them in order on load.
When deltas exceed a threshold (size or count), a new base is written in the
background and the deltas are dropped.

• A single-key change writes a few hundred bytes, not 10 GB.
• Readers mmap the base once (shared, stable) plus small deltas — page-cache pressure is negligible.
• The atomic swap from R2 now applies only to the tiny pointer file listing the active base and deltas, so the swap itself is one os.replace of a few dozen bytes.

Cost: read setup is slightly more work (apply deltas at load), and a delta chain must be bounded or reads get slow. Bounded by the compaction threshold, which is exactly the LSM base-plus-deltas pattern — and it is nice that it recurs here.

R6 — Deletion must be a tombstone with an explicit contract (answers C6)

The critique exposes an unspecified behaviour, and unspecified behaviour in a config store read on every request is a latent outage.

Change, three parts:

1. Deletion is a tombstone, replicated like any other change, so every replica converges on "absent" rather than one replica having the old value and another having nothing.
2. The read API forces the caller to handle absence:

   get(key)                     # raises KeyNotFound — no silent None
   get(key, default=...)        # explicit
   

   A silent None for a missing config key is how a service ends up with a timeout of None and fails in a way nobody can trace back to a deletion.
3. Deletion is guarded by usage. The store tracks which services have read each key in the last 7 days (a cheap sampled counter from the agents). Deleting a key that is being actively read requires force=true and emits a warning naming the readers.

And the strong recommendation, stated as such: for config, deprecate rather than delete — mark it deprecated, alarm on continued reads, and delete only after reads reach zero. Deletion of a live config key is one of the few operations in this system that can cause an immediate global outage, and the design should make it hard rather than easy.

Cost: deleted keys linger as tombstones, and a usage index to maintain. Both trivial at 10 GB and 1,000 changes/day, and they buy a whole class of outage prevented.

References

• ../WARMUP.md#chapter-3-time-and-why-you-cannot-trust-it · #chapter-6-consensus--raft-at-usable-depth · #chapter-7-consistency-models
• d02-distributed-kv.md — the same consistency questions at a very different read/write ratio, and why the answers differ
• d11-lock-service.md — consensus used for coordination rather than for storage
• Burrows, M. The Chubby Lock Service for Loosely-Coupled Distributed Systems. OSDI 2006 — the canonical "small, consistent, read-mostly, everything depends on it" store, including the client-cache design in §5
• Corbett et al. Spanner. OSDI 2012 — witness replicas, leader placement, and read-only replicas
• Ongaro & Ousterhout. In Search of an Understandable Consensus Algorithm. USENIX ATC 2014 — including membership change
• Abadi, D. Consistency Tradeoffs in Modern Distributed Database System Design. IEEE Computer 2012 — PACELC, which is the frame for §1
• Terry et al. Session Guarantees for Weakly Consistent Replicated Data. PDIS 1994 — read-your-writes and monotonic reads, the mechanism in §7
• etcd documentation — watch semantics, revisions, and compaction