P15 — Integrated Final System

Run it first. There is a companion page that builds this project's machinery as numbered, independently runnable blocks and then assembles them into one measured system: P15 hands-on — block by block (handson/h*.py). Every number on it was produced by running the code. Read it alongside the milestones below.

Large · 143 hours · Weeks 118–130 · Stage 6 · mixed

The integrated system must not merely connect components. It must answer a specific research or engineering question. Everything on this page is organised around that requirement.

The architecture, the six candidate questions and their full evaluation designs are in Final System. This page is the project specification: schedule, milestones, criteria.

Table of Contents

• The Loop, Instantiated
• Why This Project Matters
• Prerequisites
• Duration and Size
• Choosing the Question
• Architecture
• Showcase — Sum the Latency Budget in Week 119
• Implementation Milestones
• Experiments
• Benchmarks and Metrics
• Correctness Tests
• Failure Tests
• Expected Difficulties
• Scope Boundaries
• Deliverables
• Exit Criteria
• Connections
• References

The Loop, Instantiated

At this scale the loop runs at the level of the question, not the code.

Why This Project Matters

Three reasons, in increasing order of importance.

Integration is a distinct skill. Fourteen projects have each been correct in isolation. Composing them exposes a different class of problem: interface mismatches, backpressure that propagates the wrong way, error semantics that do not compose, latency budgets that were each reasonable and collectively impossible. You cannot practise this on one component.

Cross-cutting questions are where the interesting results live. "How does index staleness affect recommendation quality?" is not answerable inside the index or inside the recommender. It lives at the seam, and seams are under-studied precisely because they require someone to have built both sides.

This is the artifact that represents the journey. Not fifteen repositories — one system, one question, one paper, one demonstration. It is the thing you point at.

Prerequisites

At least eight upstream projects past their exit criteria, and specifically every project the chosen question depends on. Starting P15 with half-built inputs produces a demo rather than a contribution, and it is one of the two illegal reorderings.

Duration and Size

Large, 143 hours, 13 weeks. The budget is deliberately weighted away from building:

23% of the budget is writing. That is not padding. A result nobody can read is not a result, and the paper is the deliverable that outlives the code.

Choosing the Question

Lock it in week 118 and write the falsifier down. A question whose answer you cannot imagine being "no" is not a research question.

The six candidates, with what each requires and what makes it good. Full evaluation designs in Final System.

Recommendation: Q2 or Q5, for a reason that is about you rather than the questions. Both sit at the intersection of retrieval and systems — the seam where your professional experience and this journey's new skills overlap — and neither requires external data. Q4 is the most scientifically interesting and the most likely to end in "I cannot validate this", which is itself a publishable finding but a harder one to sit with.

Whichever you choose, the question must be written as:

Claim: ⟨specific, quantified⟩ Falsifier: ⟨the observation that would make me abandon this⟩ Baseline: ⟨what I am comparing against, and why it is the fair comparison⟩ Ablations: ⟨which components I will remove to attribute the effect⟩

Architecture

The full system. Components you have already built are named with their project.

   article source ──► ingestion (P07 streaming: partitions, offsets, watermarks)
                            │
                            ├──► item store       (P04 LSM engine)
                            ├──► interaction log  (P07 log / P05 replicated log)
                            │
                            ▼
                   embedding service (P01 model on P13 framework,
                                      P14 batching policy)
                            │
                            ▼
                     index writer ──► ANN index (P02) inside vector DB (P03)
                            │              segments · filters · snapshots
                            ▼
   request ──► recommendation service (P08) ──► retrieve → filter → rank → diversify
                            │
                            ├──► simulated users (P09)
                            ├──► A/B assignment  (P10)
                            └──► observability: tracing, metrics, fault injection (P05)

You do not need all of it. The question decides the subset. Q2 needs P02/P03/P08 and a thin harness. Q5 needs the ingestion and storage path and can stub the ranker. Building components the question does not need is the single most common way this project overruns — and is exactly the "connecting things is not a contribution" failure the brief warns about.

Showcase — Sum the Latency Budget in Week 119

Twenty minutes, in milestone 2, before any integration code. The most common way this project overruns is a budget that was never added up.

# P15 -- the latency budget, summed in week 119 rather than discovered in week 128.
budget_ms = 150.0
stages = [("ingest -> visible",  "P07",  8.0),
          ("embed (batched)",    "P13/14", 22.0),
          ("index insert",       "P02/03", 5.0),
          ("retrieve (ANN)",     "P02",  12.0),
          ("filter",             "P03",   6.0),
          ("rank",               "P08",  18.0),
          ("diversify + dedupe", "P08",   9.0),
          ("serialise + IPC x4", "-",    24.0)]
tot = sum(s[2] for s in stages)
print(f"{'stage':<22}{'from':>8}{'ms':>8}{'% of budget':>13}")
for name, src, ms in stages:
    print(f"{name:<22}{src:>8}{ms:>8.1f}{ms/budget_ms*100:>12.1f}%")
print(f"{'TOTAL':<22}{'':>8}{tot:>8.1f}{tot/budget_ms*100:>12.1f}%")
print(f"\\nHeadroom: {budget_ms-tot:.1f} ms ({(budget_ms-tot)/budget_ms*100:.0f}%)")
print("\\nNote the largest single line is not a component -- it is the 4 process")
print("boundaries at 6 ms each. Cross-language integration cost is a line item, and")
print("if you do not budget it in week 119 you discover it in week 128 with no time")
print("to change the architecture.")

stage                     from      ms  % of budget
ingest -> visible          P07     8.0         5.3%
embed (batched)         P13/14    22.0        14.7%
index insert            P02/03     5.0         3.3%
retrieve (ANN)             P02    12.0         8.0%
filter                     P03     6.0         4.0%
rank                       P08    18.0        12.0%
diversify + dedupe         P08     9.0         6.0%
serialise + IPC x4           -    24.0        16.0%
TOTAL                            104.0        69.3%
\nHeadroom: 46.0 ms (31%)
\nNote the largest single line is not a component -- it is the 4 process
boundaries at 6 ms each. Cross-language integration cost is a line item, and
if you do not budget it in week 119 you discover it in week 128 with no time
to change the architecture.

The largest line is not a component — it is the four process boundaries. Integration cost is a line item, and a budget that omits it is a budget that will be wrong by 16%. If your stages do not sum to something under your target, the architecture is wrong now, while changing it is cheap.

Implementation Milestones

Milestone 3's deadline is real. A thin end-to-end path by the end of week 121 (4 weeks in) is the checkpoint that determines whether the scope is right. If it has not happened, cut components until it does. A system that works end to end at low quality can be improved; a system with three excellent components and no path between them cannot be finished in the remaining time.

Experiments

The specific experiment set depends on the question — see Final System. Every version must include:

The negative control is the one people skip and the one reviewers ask about first.

Benchmarks and Metrics

Latency decomposition is mandatory. A single end-to-end number tells you nothing actionable. The per-stage breakdown is what turns the system into evidence, and it is usually where the surprising result is.

Correctness Tests

1. End-to-end invariants no component can check alone: every ingested article eventually becomes recommendable or is explicitly rejected with a reason; no recommendation references a non-existent or deleted item; no article is recommended before its publish time.
2. Component contract tests at every interface, running in CI.
3. Idempotent ingestion: the same article twice produces one item.
4. Cross-component consistency: index contents match the item store after a quiescent period.
5. Trace completeness: every request produces a full trace with no missing spans.
6. Reproducibility: the same seed and configuration produce the same headline metric within its CI.
7. Every upstream component's own test suite still passes, unmodified.

Failure Tests

The interesting question is not whether the system survives — it is whether the research claim survives. Run the headline experiment under fault injection and report whether the effect persists. That is what makes it a systems result rather than a benchmark.

Expected Difficulties

1. Integration will take longer than any component did, and the time goes to interfaces, not features. That is why milestone 3 has a hard deadline.
2. You will want to rewrite components. You will look at P02 from two years ago and want to redo it. Do not. Fix only what the question needs, and note the rest as future work.
3. Four languages is real friction. Prefer process boundaries with a simple protocol over FFI. The cost of a subprocess call is a latency-budget line item; the cost of a broken FFI binding is a lost week.
4. The question may turn out to be uninteresting once you can measure it. If that happens by week 122, change it — with the change and its reason recorded. After week 124, finish the boring version and say plainly that the result was null. A documented null result is a completed project; an abandoned interesting one is not.
5. Scope creep is fatal here because there is no project after this to absorb the slip. The question decides the scope. Write the component list in milestone 2 and treat additions as requiring a written justification.
6. Writing 17 hours of paper is harder than it sounds if you start at week 129. Write the methods section during milestone 8, while you are doing the thing it describes.

Scope Boundaries

In scope: exactly the components the question requires, integrated, instrumented, fault-injected, and measured.

Out of scope: components the question does not need; production hardening beyond what the experiment requires; a UI beyond what the demonstration needs; multi-tenancy, auth, deployment automation; rewriting any upstream component; new algorithms unrelated to the question.

Deliverables

1. The integrated system — one command to bring it up, one to run the headline experiment
2. The paper, 6,000–10,000 words: abstract, introduction, related work, system design, methodology, results, ablations, threats to validity, limitations, future work, reproducibility appendix. Threats to validity and limitations are not optional sections
3. A benchmark suite others can run against their own systems
4. The demonstration video, 5–10 minutes
5. A reproducibility package: data or its generator, configs, seeds, environment specification, expected outputs with tolerances
6. An architecture diagram that is accurate, not aspirational
7. A postmortem: what you would do differently across all fifteen projects

Exit Criteria

• The system runs end to end, one command
• The research question is answered — including if the answer is "no" or "no measurable effect"
• ≥2 baselines, one of which is a degenerate configuration of your own system
• Ablations attribute the effect to specific components
• ≥5 seeds per configuration with bootstrap CIs on the headline metric
• A negative control was run and showed no effect
• The headline experiment was repeated under fault injection
• End-to-end latency decomposed by stage
• The paper is written, including threats to validity and limitations
• A person who is not you has cloned the repository and reproduced the headline number, and you have their report
• Demonstration video recorded
• Postmortem written

The reproduction-by-a-stranger criterion is the hardest one and the most important. Everything else you can grade yourself on. That one you cannot.

Connections

Backward: eight or more projects, depending on the question.

Forward: Portfolio — publication, and what comes after. If the result holds up, Research Directions lists which venues take work of this shape.

References

Beyond the question-specific literature (see Final System):

• Peyton Jones, S. How to Write a Great Research Paper. Microsoft Research, 2004.
• Zobel, J. Writing for Computer Science, 3rd ed. Springer, 2014.
• Shewchuk, J. R. Three Sins of Authors in Computer Science and Math. 1997.
• Wilson, G. et al. Best Practices for Scientific Computing. PLoS Biology 12(1), 2014.
• Collberg, C., Proebsting, T. A. Repeatability in Computer Systems Research. CACM 59(3), 2016. The study that found most systems papers are not reproducible — read it before writing your reproducibility appendix.
• Blackburn, S. M. et al. The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations. ACM TOPLAS 38(4), 2016. The best available checklist for a systems evaluation section.
• Hoefler, T., Belli, R. Scientific Benchmarking of Parallel Computing Systems. SC 2015. Twelve rules for reporting performance results; apply all twelve.