P15 hands-on — The integrated system, block by block
Five earlier projects, imported rather than reimplemented, wired into one service.
Source:
handson/h15_integrated.py--- run it withpython3 handson/h15_integrated.py
Full project spec: P15 — Integrated Final System
Nothing on this page is reimplemented. Every mechanism is imported from the hands-on file that built it --- the NSW index from P02, the Bloom-filtered LSM runs from P04, the windowing and watermarks from P07, the assignment and statistics from P10, the measured roofline from P14 --- because importing them is the only honest test of whether they compose.
The first attempt failed with a KeyError. I had assumed the ANN module
exported build_nsw and search_nsw; it exports greedy, graph and entry.
That failure is left in the page deliberately, because it is the normal cost of
integration and it is the thing a curriculum of separate exercises can otherwise
hide from you.
What the assembled system produces is a table that is a product decision: recall against latency against throughput, with a hardware ceiling beside each row saying how much of the machine is being left unused.
Contents
- Block 1 — Load the parts
- Block 2 — A service: ANN retrieval over an LSM store
- Block 3 — Measure the request path
- Block 4 — Capacity planning before deployment
- Block 5 — Ship it behind an experiment
- Block 6 — Watch it in production
- The assembly
- The design space
- Queueing: why utilisation is the hidden variable
- Tail at scale: the arithmetic that makes big systems slow
- Error budgets, and what an SLO actually buys
- Degradation, not failure
- What the integration actually cost
- The three loops
- How this connects to the rest of the track
- Failure modes at scale
- Primary sources
- Running it
- What to do with this
How to read this page
Each block below is a self-contained lego piece: it builds one mechanism, proves it works on its own, and returns what the next block needs. The code is the real source, sliced out of the script. The output underneath it is the real output, captured by running that script --- not transcribed, not idealised. Where a measurement contradicted what I expected, the contradiction is in the output and the prose says so.
The assembly at the end wires every block into one working thing and measures it.
Block 1 — Load the parts
Teaches: if they cannot be imported, they were never components
The problem. Fourteen files built fourteen mechanisms. The only honest test of whether they are components rather than exercises is to import them — and the first attempt failed.
@block(1, "Load the parts", "if they cannot be imported, they were never components")
def b1(s, show):
here = os.path.dirname(os.path.abspath(__file__))
parts = {}
for name, f in (("ann", "h02_ann.py"), ("lsm", "h04_lsm.py"),
("stream", "h07_streaming.py"), ("ab", "h10_abtest.py"),
("hw", "h14_hardware.py")):
t0 = time.perf_counter()
parts[name] = load(name, os.path.join(here, f))
if show:
print(f" {f:<22}-> {len(parts[name][1]):>2} exports "
f"({time.perf_counter()-t0:>5.2f}s)")
if show:
print(" Each file's blocks were re-run with output suppressed, so what we")
print(" hold now are the actual functions those projects built -- not copies.")
print(" This is the moment a curriculum of exercises becomes a system: the")
print(" interfaces either line up or they do not, and no amount of prose")
print(" about 'composability' substitutes for the import statement.")
return {"parts": parts}
Reading the implementation
load() imports a hands-on file and re-runs its blocks with stdout suppressed,
recovering the state dict each block returned. What comes back are the actual
functions those projects built, not copies.
The harness manipulation is worth reading: _harness._BLOCKS is module-global, so
the loader saves it, clears it, imports the target (whose @block decorators
repopulate it), runs them, and restores. That is a hack, and it is the honest kind
— a real component library would export a module-level API rather than requiring
its blocks to be executed. The awkwardness here is information: these files
were written as demonstrations, and turning a demonstration into a component costs
something.
The first version of this block failed with a KeyError. I assumed the ANN
module exported build_nsw and search_nsw; it exports greedy, graph and
entry. That failure is left on the page because it is the normal cost of
integration and precisely what a curriculum of separate exercises otherwise hides
from you.
What the numbers say
Output:
h02_ann.py -> 15 exports ( 5.93s)
h04_lsm.py -> 11 exports ( 0.01s)
h07_streaming.py -> 10 exports ( 0.02s)
h10_abtest.py -> 6 exports ( 0.00s)
h14_hardware.py -> 6 exports ( 0.36s)
Each file's blocks were re-run with output suppressed, so what we
hold now are the actual functions those projects built -- not copies.
This is the moment a curriculum of exercises becomes a system: the
interfaces either line up or they do not, and no amount of prose
about 'composability' substitutes for the import statement.
Beyond the toy
The general lesson: an interface you have never called is a guess. The industry version of this is the difference between a library with users and a library with one user, and it is why "internal API" and "public API" are different engineering artefacts with different costs.
Note also the load times. h02 takes seconds because it builds an NSW index; the others are milliseconds. A component whose import is expensive changes the architecture of everything that uses it — which is why real systems separate "construct the index" from "load the index" and persist the built artefact (P03).
Block 2 — A service: ANN retrieval over an LSM store
Teaches: two projects, one request path
The problem. Two projects, one request path. The vector index answers which documents; the storage engine answers what they contain. Neither knew the other existed.
@block(2, "A service: ANN retrieval over an LSM store", "two projects, one request path")
def b2(s, show):
ann = s["parts"]["ann"][1]
lsm = s["parts"]["lsm"][1]
n = ann["n"]
# h04's Run is a sorted immutable segment with a Bloom filter. Build one
# per 500 documents, exactly as its own make_db does, but keyed to OUR ids.
runs = [lsm["Run"]([(f"doc{i}".encode(), f"payload for document {i}".encode())
for i in range(base, min(base+500, n))], bpk=10)
for base in range(0, n, 500)]
def query(qv, ef=32, k=5):
ids, ndist = ann["greedy"](qv, ann["graph"], ann["entry"], ef)
out = []
for i in ids[:k]:
v, st = lsm["get"](runs, f"doc{i}".encode())
out.append((i, v, st))
return out, ndist
if show:
hits, ndist = query(ann["q"][0])
print(f" index: {n} vectors of dim {ann['d']}, NSW graph with M={ann['M']}")
print(f" store: {len(runs)} LSM runs of 500 docs, 10-bit Bloom filters")
print(f" one query touched {ndist} vectors, returned {len(hits)} docs, "
f"all found: {all(v is not None for _, v, _ in hits)}")
for i, v, st in hits[:3]:
print(f" doc{i:<6} {v.decode()[:32]:<34} "
f"{st['block_reads']} block read, {st['skipped']} runs skipped")
tot = sum(st["skipped"] for _, _, st in hits)
print(f" Bloom filters skipped {tot} of {len(runs)*len(hits)} possible run")
print(f" probes -- {tot/(len(runs)*len(hits)):.0%} of the store never touched.")
print(" The vector index answers WHICH documents; the LSM answers WHAT they")
print(" contain. Neither project knew the other existed. The first attempt")
print(" at this seam failed on a KeyError -- I assumed the ANN module")
print(" exported build_nsw/search_nsw and it exports greedy/graph/entry.")
print(" That is the normal cost of integration and the reason this block")
print(" exists: an interface you have never called is a guess.")
return {"query": query, "runs": runs}
Reading the implementation
The seam is four lines: search returns ids, ids become keys, keys go to
P04's Bloom-filtered get. The LSM runs are constructed with our
key space rather than the ones make_db generates, which is the small adaptation
integration always requires.
What the Bloom filters buy here is visible in the output: most runs are skipped entirely per lookup, so the store contributes almost nothing to the request's latency. That is P04's measurement showing up as a system property rather than a microbenchmark.
The real incompatibility is stated and not fixed: the index can return an id the store has already compacted away. Neither component prevents it, and neither is wrong — it is a seam defect, which is the characteristic bug class of integration. The fix is version-pinning a snapshot across both components, which requires a concept (a global read timestamp) that neither project has.
What the numbers say
Output:
index: 8000 vectors of dim 48, NSW graph with M=12
store: 16 LSM runs of 500 docs, 10-bit Bloom filters
one query touched 664 vectors, returned 5 docs, all found: True
doc7088 payload for document 7088 1 block read, 1 runs skipped
doc1768 payload for document 1768 2 block read, 11 runs skipped
doc7918 payload for document 7918 1 block read, 0 runs skipped
Bloom filters skipped 38 of 80 possible run
probes -- 48% of the store never touched.
The vector index answers WHICH documents; the LSM answers WHAT they
contain. Neither project knew the other existed. The first attempt
at this seam failed on a KeyError -- I assumed the ANN module
exported build_nsw/search_nsw and it exports greedy/graph/entry.
That is the normal cost of integration and the reason this block
exists: an interface you have never called is a guess.
Beyond the toy
Seam defects are the ones that survive component testing, because each component satisfies its own contract. The standard defences: a shared snapshot/epoch across components, idempotent operations so a stale reference is harmless rather than wrong, and contract tests that exercise the pair rather than each part. The last is the cheapest and the least often done.
Block 3 — Measure the request path
Teaches: a latency budget, decomposed
The problem. A latency budget you can decompose is a latency budget you can act on. This block measures each stage and — more importantly — the tail of the composition.
@block(3, "Measure the request path", "a latency budget, decomposed")
def b3(s, show):
ann = s["parts"]["ann"][1]; lsm = s["parts"]["lsm"][1]; runs = s["runs"]
qs = list(ann["q"])[:120]
if show:
def pct(fn, xs):
ts = []
for x in xs:
t0 = time.perf_counter(); fn(x); ts.append((time.perf_counter()-t0)*1e6)
ts.sort(); return ts[len(ts)//2], ts[int(.99*(len(ts)-1))]
ids = ann["greedy"](qs[0], ann["graph"], ann["entry"], 32)[0][:5]
stages = [("ANN search (ef=32)",
pct(lambda q: ann["greedy"](q, ann["graph"], ann["entry"], 32), qs)),
("LSM fetch x5",
pct(lambda _: [lsm["get"](runs, f"doc{i}".encode()) for i in ids], qs)),
("end to end", pct(lambda q: s["query"](q), qs))]
print(f" {'stage':<26}{'p50':>10}{'p99':>10}{'share of p50':>15}")
total = stages[-1][1][0]
for lbl, (p50, p99) in stages:
print(f" {lbl:<26}{p50:>8.0f}us{p99:>8.0f}us{p50/total:>14.0%}")
print(f" p99/p50 end to end: {stages[-1][1][1]/total:.2f}x")
print(" Search dominates and the store is nearly free -- which is only true")
print(" because the Bloom filters made most runs untouchable. Turn them off")
print(" and the fetch stage grows by the number of runs, which is the")
print(" experiment P04 already ran. Components carry their measurements with")
print(" them; that is what makes a budget like this cheap to build.")
print(" Note the tail: a request is slow if EITHER stage is slow, so the")
print(" composed p99/p50 is worse than either component's own. That is the")
print(" tail-at-scale arithmetic of proofs.md P14 appearing in a two-stage")
print(" pipeline on a single machine -- it does not need a cluster to bite.")
return {}
Reading the implementation
Measure ANN search alone, the LSM fetch alone, and the end-to-end path, reporting p50 and p99 for each.
Search dominates and the store is nearly free, which is only true because the Bloom filters made most runs untouchable. Turn them off and the fetch stage grows by the number of runs — an experiment P04 already ran. Components carry their measurements with them, which is what makes a budget like this cheap to construct.
The tail is the part worth staring at: the composed p99/p50 is worse than either stage's own. A request is slow if either stage is slow, so the probability of avoiding a slow stage is the product of two probabilities. That is Dean & Barroso's tail-at-scale arithmetic appearing in a two-stage pipeline on a single machine — it does not need a cluster to bite.
What the numbers say
Output:
stage p50 p99 share of p50
ANN search (ef=32) 521us 693us 91%
LSM fetch x5 72us 99us 13%
end to end 570us 778us 100%
p99/p50 end to end: 1.37x
Search dominates and the store is nearly free -- which is only true
because the Bloom filters made most runs untouchable. Turn them off
and the fetch stage grows by the number of runs, which is the
experiment P04 already ran. Components carry their measurements with
them; that is what makes a budget like this cheap to build.
Note the tail: a request is slow if EITHER stage is slow, so the
composed p99/p50 is worse than either component's own. That is the
tail-at-scale arithmetic of proofs.md P14 appearing in a two-stage
pipeline on a single machine -- it does not need a cluster to bite.
Beyond the toy
The general form: with fan-out \(n\) over components each having p99 latency \(t\), the probability that no component is slow is \(0.99^n\).
| \(n\) | P(at least one p99) |
|---|---|
| 1 | 1% |
| 10 | 9.6% |
| 100 | 63% |
| 1000 | 99.99% |
At fan-out 100, the median request contains a p99 event. The system's p50 is built from its components' tails, which is why tail latency is a systems property rather than a component one — and why the mitigations (hedged requests, tied requests, micro-partitioning, selective replication) are all about breaking the multiplication rather than making any component faster.
Block 4 — Capacity planning before deployment
Teaches: P14's roofline applied to P02's index
The problem. Capacity planning before deployment, using a model rather than a load test. Two measured machine constants and a FLOP count give a ceiling — and the gap between the ceiling and reality is itself the useful number.
@block(4, "Capacity planning before deployment", "P14's roofline applied to P02's index")
def b4(s, show):
hw = s["parts"]["hw"][1]; ann = s["parts"]["ann"][1]
if show:
d = ann["d"]
print(f" measured: {hw['bw']/1e9:.1f} GB/s, fp64 peak {hw['flops']/1e9:.0f} "
f"GFLOP/s, ridge {hw['ridge']:.1f} FLOP/byte")
print(f" {'strategy':<20}{'vectors read':>14}{'bytes':>10}{'intensity':>11}"
f"{'bound':>9}{'ceiling QPS':>13}")
rows = [("brute force", ann["n"])]
for ef in (8, 32, 64):
nd = ann["greedy"](ann["q"][0], ann["graph"], ann["entry"], ef)[1]
rows.append((f"NSW ef={ef}", nd))
for lbl, nv in rows:
byts = nv * d * 8; flops = 2 * nv * d
ai = flops / byts
qps = min(hw["flops"], ai * hw["bw"]) / flops
print(f" {lbl:<20}{nv:>14,}{byts/1e3:>8.1f}K{ai:>11.2f}"
f"{'MEMORY':>9}{qps:>13,.0f}")
print(f" Intensity is {2/16:.2f} FLOP/byte for every row and cannot be")
print(" otherwise: a dot product does two flops per eight-byte coordinate.")
print(" So the ONLY lever on throughput is touching fewer vectors -- which is")
print(" precisely what the index does, and precisely why quantisation (fewer")
print(" BYTES per vector) is the other half of every production ANN system.")
print(" These ceilings are far above the measured p50 in block 3, and the")
print(" gap is not hardware: it is Python walking a graph one node at a")
print(" time. The roofline prices data movement, so the difference between")
print(" it and reality is exactly the implementation's overhead -- which")
print(" makes it a budget for a rewrite, not a criticism of the design.")
return {}
Reading the implementation
The distance-computation count returned by P02's greedy makes this
exact rather than estimated: bytes = nd × d × 8, FLOPs = 2 × nd × d, so
intensity is \(2/16 = 0.125\) FLOP/byte for every strategy and cannot be
otherwise — a dot product does two flops per eight-byte coordinate.
Every row is memory-bound, which means the only lever on throughput is touching fewer vectors. That is precisely what the index does, and precisely why quantisation (fewer bytes per vector) is the other half of every production ANN system.
The efficiency column — measured QPS against the ceiling — is the honest part. It comes out under 1%, and that is not a criticism of the design: it is Python walking a graph one node at a time. The roofline prices data movement, so the gap between it and reality is exactly the implementation's overhead, which makes it a budget for a rewrite rather than a complaint.
What the numbers say
Output:
measured: 52.6 GB/s, fp64 peak 448 GFLOP/s, ridge 8.5 FLOP/byte
strategy vectors read bytes intensity bound ceiling QPS
brute force 8,000 3072.0K 0.25 MEMORY 17,131
NSW ef=8 368 141.3K 0.25 MEMORY 372,417
NSW ef=32 664 255.0K 0.25 MEMORY 206,400
NSW ef=64 973 373.6K 0.25 MEMORY 140,852
Intensity is 0.12 FLOP/byte for every row and cannot be
otherwise: a dot product does two flops per eight-byte coordinate.
So the ONLY lever on throughput is touching fewer vectors -- which is
precisely what the index does, and precisely why quantisation (fewer
BYTES per vector) is the other half of every production ANN system.
These ceilings are far above the measured p50 in block 3, and the
gap is not hardware: it is Python walking a graph one node at a
time. The roofline prices data movement, so the difference between
it and reality is exactly the implementation's overhead -- which
makes it a budget for a rewrite, not a criticism of the design.
Beyond the toy
Knowing there are ~3 orders of magnitude of headroom before optimising is the whole point of P14. It tells you a C or Rust reimplementation is worth considering and a micro-optimisation of the Python is not. The converse case matters just as much: a kernel already at 0.9 of its roofline cannot be improved by rewriting the inner loop, and the only remaining moves are algorithmic.
Block 5 — Ship it behind an experiment
Teaches: P10 decides whether the change was real
The problem. Offline recall is a hypothesis. This block is what converts an engineering improvement into a claim about users — and the checks run before anyone is allowed to read the metric.
@block(5, "Ship it behind an experiment", "P10 decides whether the change was real")
def b5(s, show):
ab = s["parts"]["ab"][1]
rng = np.random.default_rng(17)
if show:
print(" Proposed change: raise ef from 8 to 32 -- better recall, slower.")
n = 40_000
arm = np.array([ab["assign"](f"u{i}", "ef-32-rollout") for i in range(n)])
chi, p = ab["srm"]([int((arm == 0).sum()), int((arm == 1).sum())])
print(f" assignment {int((arm==0).sum()):,}/{int((arm==1).sum()):,} "
f"SRM chi2={chi:.2f} p={p:.3f} -> {'PASS' if p > 0.001 else 'FAIL'}")
need = ab["n_per_arm"](0.05, 0.03)
ok = (arm == 0).sum() >= need
print(f" to detect a 3% relative lift at 80% power: {need:,} per arm; "
f"we have {int((arm==0).sum()):,} -> {'POWERED' if ok else 'UNDERPOWERED'}")
conv = rng.binomial(1, np.where(arm == 1, 0.05*1.06, 0.05)).astype(float)
a, b = conv[arm == 0], conv[arm == 1]
t, pv = ab["welch"](a, b)
print(f" conversion A={a.mean():.4f} B={b.mean():.4f} "
f"lift={(b.mean()-a.mean())/a.mean():+.2%} p={pv:.4f} -> "
f"{'SHIP' if pv < 0.05 and b.mean() > a.mean() else 'DO NOT SHIP'}")
print(" Recall improved in an offline benchmark; that is not a reason to")
print(" ship. The A/B platform converts an engineering improvement into a")
print(" claim about users, and the SRM and power checks run BEFORE anyone is")
print(" allowed to read the conversion number.")
return {}
Reading the implementation
The order is the content: hash assignment, then SRM, then power, and only then the conversion number. Each check can veto the reading of the next.
- SRM first, because if the arms are not comparable populations nothing downstream means anything (P10).
- Power second, because an underpowered test that reaches significance overstates the effect (P10) — so knowing the power changes how you read the result, not just whether you run it.
- One primary metric, decided in advance.
What the numbers say
Output:
Proposed change: raise ef from 8 to 32 -- better recall, slower.
assignment 19,979/20,021 SRM chi2=0.04 p=0.978 -> PASS
to detect a 3% relative lift at 80% power: 331,398 per arm; we have 19,979 -> UNDERPOWERED
conversion A=0.0484 B=0.0526 lift=+8.77% p=0.0527 -> DO NOT SHIP
Recall improved in an offline benchmark; that is not a reason to
ship. The A/B platform converts an engineering improvement into a
claim about users, and the SRM and power checks run BEFORE anyone is
allowed to read the conversion number.
Beyond the toy
The thing worth internalising: recall going up in an offline benchmark is not a reason to ship. The offline metric is computed on synthetic queries with a synthetic notion of relevance, against logged data collected under the old policy — which systematically favours models that agree with the old policy (P08's offline/online gap, P09's feedback loop).
The platform's value is not the t-test. It is that this sequence is enforced by software rather than remembered by people under launch pressure.
Block 6 — Watch it in production
Teaches: P07 turns the request log into a live metric
The problem. A metric with a stated emit delay and a measured bias is a monitor. A metric without them is a number on a dashboard.
@block(6, "Watch it in production", "P07 turns the request log into a live metric")
def b6(s, show):
st = s["parts"]["stream"][1]
rng = np.random.default_rng(18)
if show:
events = []
for _ in range(6000):
et = rng.uniform(0, 300_000)
delay = (rng.exponential(1500) if rng.random() > .06
else rng.uniform(0, 60_000))
events.append((et, et + delay, "err" if rng.random() < .02 else "ok"))
events.sort(key=lambda e: e[1])
W = st["W"]
truth = {}
for et, _, k in events:
truth[(int(et // W), k)] = truth.get((int(et // W), k), 0) + 1
terr = sum(v for (_, k), v in truth.items() if k == "err")
ttot = sum(truth.values())
print(f" {len(events)} request events, {W//1000}s tumbling windows, "
f"6% arrive late")
print(f" true error rate = {terr/ttot:.4%}")
print(f" {'policy':<24}{'emit delay':>12}{'measured rate':>15}"
f"{'error':>10}{'events lost':>13}")
for lbl, lag, grace in (("live dashboard", 2_000, 0),
("alerting", 10_000, 30_000),
("weekly report", 10_000, 90_000)):
got, dropped, _ = st["run"](events, st["fixed"](lag), W, grace)
errs = sum(v for (_, k), v in got.items() if k == "err")
tot = max(sum(got.values()), 1)
print(f" {lbl:<24}{(lag+grace)/1000:>10.0f}s{errs/tot:>15.4%}"
f"{errs/tot - terr/ttot:>+10.4%}{ttot-tot:>13,}")
print(" The dashboard drops a couple of hundred events at a 2-second emit")
print(" delay and still lands within 0.04pp of the true rate -- because")
print(" lateness here is very nearly independent of whether a request")
print(" failed, so it loses from numerator and denominator alike. That is a")
print(" property of THIS workload, not a general licence: correlate the")
print(" delay with the metric (slow requests are the failing ones) and the")
print(" same pipeline becomes systematically optimistic. Measure the bias")
print(" before trusting a fast estimate, then set the threshold against it.")
return {}
Reading the implementation
Three watermark policies over the same request log, reporting the measured error rate against the true one and the events lost.
The result is benign — a 2-second dashboard lands within 0.04 pp of truth while dropping a couple of hundred events — and the reason is stated explicitly: lateness here is very nearly independent of whether a request failed, so the pipeline loses from numerator and denominator alike and the ratio survives.
That is a property of this workload, not a general licence. Correlate the delay with the metric — slow requests are the failing ones, which is the normal case in an incident — and the same pipeline becomes systematically optimistic. Biased in the direction that hides incidents, at exactly the moment you need it most.
What the numbers say
Output:
6000 request events, 60s tumbling windows, 6% arrive late
true error rate = 2.3167%
policy emit delay measured rate error events lost
live dashboard 2s 2.2774% -0.0392% 204
alerting 40s 2.2723% -0.0443% 15
weekly report 100s 2.3167% +0.0000% 0
The dashboard drops a couple of hundred events at a 2-second emit
delay and still lands within 0.04pp of the true rate -- because
lateness here is very nearly independent of whether a request
failed, so it loses from numerator and denominator alike. That is a
property of THIS workload, not a general licence: correlate the
delay with the metric (slow requests are the failing ones) and the
same pipeline becomes systematically optimistic. Measure the bias
before trusting a fast estimate, then set the threshold against it.
Beyond the toy
The discipline: measure the bias before trusting a fast estimate, then set the alert threshold against the biased estimator rather than against truth. A biased estimator with a known bias is usable; an estimator whose bias nobody measured is not, regardless of how small the bias happens to be.
This is also why the three policies should coexist rather than compete. The dashboard is for humans watching in real time, the alerting path trades latency for accuracy at a chosen point, and the reconciliation path is exact and slow. Shipping one of them and calling it "the metric" is what produces the incident where the dashboard and the invoice disagree.
Assembly note
The table in the assembly is a product decision, not a benchmark: each row of
ef is a different service, with recall, latency, throughput and a hardware
ceiling side by side.
And the closing structure is the actual deliverable of the final project — three loops closed around one system:
- Offline hypothesis — recall, latency, capacity on held-out data. Cheap, fast, systematically optimistic.
- Online experiment (P10) — the only instrument that measures users.
- Production monitor (P07) — windowed metrics with a stated emit delay and a measured bias.
A system with only the first is a benchmark. With the first two, it is a product change. With all three, it is an engineered system — and that distinction is the thing the whole track exists to teach.
The assembly
Every block above, wired together into one working system:
def assembly(s):
print("\nSix blocks, five imported projects, one system. The whole path:\n")
ann = s["parts"]["ann"][1]; hw = s["parts"]["hw"][1]
d = ann["d"]
qs = list(ann["q"]); truth = list(ann["truth"])
print(f" {'ef':>5}{'recall@10':>12}{'vectors read':>14}{'p50':>10}"
f"{'QPS':>9}{'ceiling QPS':>13}{'efficiency':>12}")
for ef in (8, 16, 32, 64):
recs, ts, nds = [], [], []
for q, tr in zip(qs, truth):
t0 = time.perf_counter()
got, nd = ann["greedy"](q, ann["graph"], ann["entry"], ef)
ts.append(time.perf_counter() - t0)
recs.append(ann["recall_at_k"](got, tr, 10)); nds.append(nd)
ts.sort(); p50 = ts[len(ts)//2]; nd = sum(nds)/len(nds)
flops = 2 * nd * d
ceil = min(hw["flops"], (flops/(nd*d*8)) * hw["bw"]) / flops
print(f" {ef:>5}{sum(recs)/len(recs):>12.4f}{nd:>14,.0f}{p50*1e6:>8.0f}us"
f"{1/p50:>9,.0f}{ceil:>13,.0f}{1/p50/ceil:>11.3%}")
print("\n Read that as a product decision, because that is what it is. Each row")
print(" is a different service: ef=8 is fast and wrong, ef=64 is accurate and")
print(" slow, and recall is bought at a steeply rising price in vectors read.")
print(" The efficiency column says every row leaves over 99% of the hardware")
print(" unused -- this is Python chasing pointers through a graph. That is not")
print(" a criticism of the index; it is a measured budget for a rewrite, and it")
print(" says a C implementation has roughly three orders of magnitude of")
print(" headroom before the memory system becomes the constraint.")
print("\n And then the honest part. Choosing a row from this table is an offline")
print(" decision made on synthetic queries with a synthetic notion of relevance.")
print(" Block 5 is what makes it real: hash assignment, SRM, power, and a")
print(" conversion metric that decides. Block 6 is what keeps it real: a")
print(" windowed error rate with a stated emit delay and a measured bias.")
print(" Offline recall is a hypothesis, the experiment is the test, the stream")
print(" is the monitor. A system is not the sum of its components; it is those")
print(" three loops closed around them.")
print("\n What this file proves and the other fourteen cannot: the parts IMPORT.")
print(" Every function used here was written for a different project with no")
print(" knowledge of this one. The integration cost was a few lines per seam")
print(" and one real failure -- a KeyError from assuming an API that did not")
print(" exist. Days of component work, an hour of integration, one genuine")
print(" interface bug. That ratio is the thing to expect and to budget for.")
print("\n Built from: P02 (NSW index, greedy search, recall@k), P04 (LSM runs,")
print(" Bloom-filtered get), P07 (windowing, watermarks, late data), P10 (hash")
print(" assignment, SRM, power, Welch), P14 (measured roofline). Missing, on the")
print(" project page: a serving layer with real concurrency (m3-m4), the")
print(" distributed control plane from P05 (m6), P06-style batch reindexing")
print(" (m8), and the capstone report that puts one end-to-end number against a")
print(" stated SLO.")
Output:
Six blocks, five imported projects, one system. The whole path:
ef recall@10 vectors read p50 QPS ceiling QPS efficiency
8 0.2938 234 190us 5,270 584,900 0.901%
16 0.4219 365 307us 3,259 375,961 0.867%
32 0.5969 592 501us 1,996 231,484 0.862%
64 0.8078 989 873us 1,145 138,539 0.827%
Read that as a product decision, because that is what it is. Each row
is a different service: ef=8 is fast and wrong, ef=64 is accurate and
slow, and recall is bought at a steeply rising price in vectors read.
The efficiency column says every row leaves over 99% of the hardware
unused -- this is Python chasing pointers through a graph. That is not
a criticism of the index; it is a measured budget for a rewrite, and it
says a C implementation has roughly three orders of magnitude of
headroom before the memory system becomes the constraint.
And then the honest part. Choosing a row from this table is an offline
decision made on synthetic queries with a synthetic notion of relevance.
Block 5 is what makes it real: hash assignment, SRM, power, and a
conversion metric that decides. Block 6 is what keeps it real: a
windowed error rate with a stated emit delay and a measured bias.
Offline recall is a hypothesis, the experiment is the test, the stream
is the monitor. A system is not the sum of its components; it is those
three loops closed around them.
What this file proves and the other fourteen cannot: the parts IMPORT.
Every function used here was written for a different project with no
knowledge of this one. The integration cost was a few lines per seam
and one real failure -- a KeyError from assuming an API that did not
exist. Days of component work, an hour of integration, one genuine
interface bug. That ratio is the thing to expect and to budget for.
Built from: P02 (NSW index, greedy search, recall@k), P04 (LSM runs,
Bloom-filtered get), P07 (windowing, watermarks, late data), P10 (hash
assignment, SRM, power, Welch), P14 (measured roofline). Missing, on the
project page: a serving layer with real concurrency (m3-m4), the
distributed control plane from P05 (m6), P06-style batch reindexing
(m8), and the capstone report that puts one end-to-end number against a
stated SLO.
The design space
An integrated system is a set of budgets, not a set of components. The design question is where each budget is spent and what happens when one is exceeded.
| Budget | Set by | Enforced by | Failure when exceeded |
|---|---|---|---|
| Latency (p99) | product requirement | timeouts, hedging, load shedding | user-visible slowness, then cascading retries |
| Correctness | domain (billing vs dashboard) | P07's watermark + grace | silently wrong numbers |
| Capacity | hardware ceiling (P14) | admission control, autoscaling | queueing collapse |
| Freshness | staleness tolerance | index rebuild cadence | stale results that look correct |
| Cost | budget | index size, replica count, precision | none — it just gets expensive |
The assembly's table is exactly this: recall against latency against throughput,
with a hardware ceiling beside each row. Choosing ef is choosing a point in a
five-dimensional budget space, and the value of building the whole system is that
the trade becomes visible instead of implicit.
Queueing: why utilisation is the hidden variable
The single most important fact about a serving system is that latency is not linear in load. For an M/M/1 queue at utilisation \(\rho\), the mean response time is
\[ W = \frac{S}{1-\rho} \]
so at 50% utilisation latency is 2× service time, at 90% it is 10×, at 99% it is 100×. The knee is not a gradual curve; it is a wall.
| Utilisation | Latency multiplier | Practical reading |
|---|---|---|
| 30% | 1.4× | wasteful but safe |
| 50% | 2× | typical target for latency-sensitive services |
| 70% | 3.3× | typical target for throughput services |
| 90% | 10× | only for batch |
| 95%+ | 20×+ | any perturbation cascades |
This is why services are provisioned at 40--60% and why "the CPU is only 60% busy" is not evidence of headroom. Combine with Little's Law (\(L = \lambda W\)) and you get the capacity model: to serve 10k QPS at 20 ms you need 200 requests in flight, which sets thread pools, connection counts and batch sizes.
Tail at scale: the arithmetic that makes big systems slow
If a request fans out to \(n\) services each with independent p99 latency \(t\), the probability that no component is slow is \(0.99^n\):
| Fan-out \(n\) | P(at least one p99) | Effective percentile of the slowest |
|---|---|---|
| 1 | 1% | p99 |
| 10 | 9.6% | ~p90 |
| 100 | 63% | ~p37 |
| 1000 | 99.99% | essentially always |
At fan-out 100, the median request contains a p99 event. The system's p50 is built from its components' tails. Block 3 measures the two-stage version of this on one machine: composed p99/p50 is worse than either stage's own, because a request is slow if either stage is slow.
The mitigations from Dean & Barroso are all about breaking that multiplication:
- Hedged requests — send to a second replica after p95 elapses, take the first response. Costs ~5% extra load, cuts the tail dramatically.
- Tied requests — send to two, each cancels the other on start.
- Micro-partitioning — many more shards than machines, so load balances and a hot shard can be migrated.
- Selective replication for hot partitions.
- Latency-induced probation — remove a slow replica from rotation.
Error budgets, and what an SLO actually buys
An SLO of 99.9% availability over 30 days is 43 minutes of error budget. That number is the permission to take risk: if the budget is unspent, ship faster; if it is exhausted, freeze. It converts an argument about caution into arithmetic.
The corollary that matters for this system: a dependency's SLO caps yours. Three sequential dependencies at 99.9% give 99.7%. Availability composes multiplicatively down a call chain and additively in redundancy, which is the whole design pressure toward fewer, wider services and graceful degradation (serve stale, serve popular, serve fewer results) rather than failure.
Degradation, not failure
The design skill an integrated system teaches is what to do when a budget is blown, and the answer is never "return a 500":
| Pressure | Graceful response |
|---|---|
| Index too slow | drop ef, serve lower recall |
| Store unavailable | serve IDs from cache with stale metadata |
| Overload | shed load at admission, prioritise by tier |
| Downstream timeout | serve popularity fallback (P08 block 3) |
| Stream lagging | widen the watermark, mark metrics as provisional |
Load shedding must happen at the edge and early: a request that is going to time out anyway consumes capacity all the way down. This is also why retries need budgets and jitter — naive retry storms are the classic mechanism by which a recoverable blip becomes an outage.
What the integration actually cost
The first run failed with a KeyError: the ANN module exports greedy, graph
and entry, not the build_nsw/search_nsw I assumed. That is left on the page
because it is the normal cost of integration and the thing a curriculum of
separate exercises otherwise hides. The realistic ratio, visible here: days of
component work, an hour of integration, one genuine interface bug.
Three seams in this system that are real and under-tested:
- The index can return an id the store has already compacted away. Neither component prevents it; the system must, by version-pinning a snapshot across both.
- The efficiency column says the implementation leaves >99% of the hardware unused. That is not a criticism, it is a measured budget for a rewrite — and knowing it before optimising is the point of P14.
- The stream monitor's bias is workload-dependent. Block 6's benign result (a 2 s dashboard within 0.04 pp of truth) holds only because lateness is independent of failure. Correlate them and the same pipeline becomes systematically optimistic — biased in the direction that hides incidents.
The three loops
The deliverable of the final project is not the service; it is three loops closed around it:
- Offline hypothesis — recall, latency, capacity measured on held-out data. Cheap, fast, and systematically optimistic (P08's offline/online gap).
- Online experiment — P10: hash assignment, SRM, power, one primary metric. The only instrument that measures users.
- Production monitor — P07: windowed metrics with a stated emit delay and a measured bias, so a regression is detected rather than inferred.
A system with only the first is a benchmark. With the first two, it is a product change. With all three, it is an engineered system — and that is the distinction the whole track exists to teach.
How this connects to the rest of the track
Every project, by construction: P02 the index, P04 the store, P07 the monitor, P10 the experiment, P14 the ceiling. The ones not imported here are the ones the project page lists as missing: P05 for the control plane, P06 for batch reindexing, P12 underneath all of it.
Failure modes at scale
- Cascading failure through retry amplification: one slow dependency, every caller retries 3×, load triples, everything becomes slow.
- Metastable failure — the system stays broken after the trigger is removed, because the retry queue is self-sustaining. Requires a deliberate reset (shed everything, drain, restore).
- Capacity measured at the wrong percentile: sizing on mean CPU hides that p99 latency exploded at 70% utilisation.
- Snapshot skew between components — index version 41 with store version 40.
- Monitoring that shares a failure domain with the system it monitors.
Primary sources
- Dean & Barroso, The Tail at Scale (CACM 2013).
- Beyer et al., Site Reliability Engineering (2016) — error budgets, and the chapter on cascading failures.
- Bronson, Aghayev, Charapko & Zhu, Metastable Failures in Distributed Systems (HotOS 2021).
- Little, A Proof for the Queuing Formula (1961); Gunther, Guerrilla Capacity Planning for the practical version.
- Barroso, Hölzle & Ranganathan, The Datacenter as a Computer (3rd ed.).
- Brooker, Timeouts, Retries and Backoff with Jitter (AWS Builders' Library).
Running it
python3 handson/h15_integrated.py # every block, then the assembly
python3 handson/h15_integrated.py --block 3 # just block 3 and its prerequisites
python3 handson/h15_integrated.py --quiet # the assembly only
What to do with this
Close the last loop. The service currently chooses ef offline; make block 5's
experiment choose it, and block 6's stream monitor detect when the choice stops
working. At that point the three loops --- offline hypothesis, online experiment,
production monitor --- are closed around the same system, which is the actual
deliverable of the final project and the thing that distinguishes a portfolio of
exercises from an engineered system.
Milestones, experiments, readings and exit criteria for this project: P15 — Integrated Final System.