The Integrated Final System

Project 15's architecture, its six candidate research questions with full evaluation designs, and the rule that governs all of them.

Project specification, schedule and milestones: P15.

Table of Contents

• The Governing Rule
• The Full Architecture
• What Each Component Contributes
• The Six Candidate Questions
• Choosing Between Them
• The Evaluation Standard
• Integration Hazards
• References

The Governing Rule

The integrated system must not merely connect components. It must answer a specific research or engineering question.

The distinction is not rhetorical, and it has a mechanical consequence:

A system built to demonstrate includes every component you built, because leaving one out looks like a gap. It is assessed on whether it works.

A system built to answer a question includes only the components the question needs, because the others are latency, complexity, and risk with no evidentiary value. It is assessed on whether the answer is credible.

Those two systems look different, and you must decide which you are building in week 118 — before the integration code exists, because after that the sunk cost decides for you.

The practical test: for each component, ask "if I removed this, would the answer to my question change?" If no, it is not in the system. Write that list in milestone 2 and treat additions as requiring a written justification.

The Full Architecture

The complete pipeline. You will build a subset. Components are labelled with the project that produced them.

   ┌─────────────────────────────────────────────────────────────────────────┐
   │  INGESTION                                                              │
   │    article source ──► partitioned log ──► watermarks ──► checkpoints    │
   │                        [P07 streaming]                                  │
   └───────────────────────────────┬─────────────────────────────────────────┘
                                   │
        ┌──────────────────────────┼──────────────────────────┐
        ▼                          ▼                          ▼
   ┌──────────┐            ┌──────────────┐          ┌────────────────┐
   │ item     │            │ interaction  │          │ embedding      │
   │ store    │            │ log          │          │ service        │
   │ [P04 LSM]│            │ [P05/P07 log]│          │ [P01 model on  │
   │          │            │              │          │  P13 framework,│
   │          │            │              │          │  P14 batching] │
   └────┬─────┘            └──────┬───────┘          └───────┬────────┘
        │                         │                          │
        │                         │                          ▼
        │                         │              ┌────────────────────────┐
        │                         │              │ index writer ──► ANN   │
        │                         │              │ index [P02] inside     │
        │                         │              │ vector DB [P03]:       │
        │                         │              │ segments · filters ·   │
        │                         │              │ snapshots · compaction │
        │                         │              └───────────┬────────────┘
        │                         │                          │
        └─────────────────────────┴──────────────────────────┘
                                   │
                                   ▼
                    ┌──────────────────────────────┐
        request ──► │ recommendation service [P08] │
                    │  retrieve → filter → rank    │
                    │  → diversify → dedupe        │
                    └───────────────┬──────────────┘
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
   ┌─────────────┐          ┌───────────────┐        ┌────────────────────┐
   │ simulated   │          │ A/B assignment│        │ observability      │
   │ users [P09] │          │ + analysis    │        │  tracing · metrics │
   │             │          │ [P10]         │        │  fault injection   │
   │             │          │               │        │  [P05 injector]    │
   └─────────────┘          └───────────────┘        └────────────────────┘

What Each Component Contributes

Observability is on every row. A P15 without distributed tracing produces end-to-end numbers you cannot decompose, and an undecomposed number is a benchmark result rather than a finding. Build it at milestone 4, before hardening anything.

The Six Candidate Questions

Each is stated as a claim with a falsifier, because a question whose answer cannot be "no" is not a research question.

Q1 — How should a recency-sensitive recommender adapt when user interests drift?

Claim. There is a relationship between measured drift rate \(\eta\) and the optimal EMA decay \(\alpha\), and a recommender that estimates \(\eta\) per user and sets \(\alpha\) accordingly beats any fixed \(\alpha\) on a population with heterogeneous drift.

Falsifier. A single well-chosen fixed \(\alpha\) matches the adaptive policy within confidence intervals across the whole drift distribution.

Needs: P02, P03, P08, P09, P10.

Design. P09 generates a population with known per-user \(\eta\), drawn from a realistic heterogeneous distribution. Arms: fixed \(\alpha\) at five values; oracle \(\alpha\) (using true \(\eta\), an upper bound); adaptive \(\alpha\) estimated from observed profile displacement. The oracle arm is essential — it tells you the ceiling, and if the oracle barely beats the best fixed \(\alpha\), the adaptive question is dead and you have saved yourself six weeks.

Metrics. NDCG and simulated engagement per drift decile; adaptation lag after an induced interest change; stability.

Risk. The answer may be "α≈0.1 is fine for everyone", which is a real answer and a dull paper. The oracle arm surfaces this in week 1 of the experiment rather than week 10.

Q2 — Can an adaptive ANN search policy reduce latency while preserving recall?

Claim. A policy that predicts per-query difficulty from early search signals and sets efSearch per query achieves Pareto dominance over fixed efSearch: equal mean recall@10 at lower mean and p99 latency.

Falsifier. The adaptive policy fails to dominate a well-tuned fixed efSearch anywhere on the frontier — or improves the mean while degrading p99, which is not a win.

Needs: P02, P03, P08. The smallest component set of any question here.

Design. Signals available after ~20 hops: current best distance, improvement rate over the last k hops, beam distance variance, beam-replacement count. Train a small predictor (a decision tree is fine and is more interpretable than anything larger) on a held-out query set. Baseline: the full fixed-efSearch sweep, which is your P02 curve. Ablate each signal.

Metrics. recall@10, mean and p99 latency, distance computations per query, predictor overhead — which must be counted against the saving.

Why it is a strong choice. Self-contained, cleanly measurable, needs no external data, and it is research direction D1 so the work has a life after the journey.

Q3 — When does dynamic embedding generation outperform precomputed embeddings?

Claim. There is a crossover in catalogue-turnover rate above which generating embeddings on demand beats precomputing them, and the crossover is predictable from model cost, cache hit rate, and turnover rate.

Falsifier. Precomputation dominates at every realistic turnover rate.

Needs: P01, P13, P14, P02, P03, P08. The most components — which is a risk, not a virtue.

Design. Sweep turnover rate and query distribution skew. Arms: full precompute; on-demand with an LRU cache; hybrid (precompute the head, generate the tail). Cost accounting must be scrupulous — compute, storage, and latency for each arm, or the comparison is meaningless.

Metrics. Freshness, p99 latency, compute cost per recommendation, storage, cache hit rate.

Risk. The comparison is only as good as the cost model. Two arms with different resource profiles need an explicit exchange rate between compute and storage, and choosing it is a judgement call you must defend in the paper.

Q4 — Can simulated users predict the relative performance of ranking algorithms?

Claim. Simulated-user evaluation produces the same ordinal ranking of algorithms as real-data offline evaluation, and the agreement depends on a small identifiable set of user-model properties.

Falsifier. Ordinal agreement is unstable under plausible user-model variation, with no identifiable subset of properties controlling it.

Needs: P08, P09, P10, plus a real interaction dataset (MIND is the obvious candidate and is in your domain).

Design. Rank ≥15 algorithm variants three ways: offline replay on real logs, simulated-user evaluation, and — ideally — real online data if you can obtain any. Rank- correlate all pairs. Then run P09's sensitivity analysis to find which user-model parameters change the correlation.

Metrics. Spearman and Kendall correlation between rankings; per-variant disagreement; sensitivity of the correlation to each parameter.

Why it is the best science here. It is a methodological result about a widely used and rarely validated technique, and the negative outcome is as publishable as the positive one — "simulator rankings are unstable under plausible model variation" is something the field would benefit from knowing.

Risk. Highest data dependency of the six. Confirm dataset access in week 118, not week 125.

Q5 — How do storage and indexing choices affect recommendation freshness?

Claim. Publication-to-recommendable latency is dominated by a small number of identifiable stages, and there is a freshness/efficiency frontier along which recommendation quality varies measurably.

Falsifier. Quality is insensitive to freshness across the whole range achievable by parameter tuning.

Needs: P03, P04, P05, P07, P08. Heaviest on the systems stack, lightest on ML.

Design. Instrument every stage from publication to recommendable. Sweep the parameters that trade freshness against efficiency: segment size, flush interval, compaction trigger, index rebuild cadence, watermark delay, embedding batch size. At each point, measure both the freshness distribution and recommendation quality.

Metrics. Publication-to-recommendable latency decomposed by stage (this is the result); recommendation quality vs freshness; throughput and storage cost at each point.

Why it is strong. It is research direction D5, it sits in a genuine gap between the storage and recommender literatures, and it is directly your professional domain. The deliverable figure — a freshness/efficiency frontier with quality contours — does not currently exist anywhere.

Q6 — Can hardware-aware batching substantially reduce end-to-end embedding latency?

Claim. A batching policy informed by the roofline model (batch until the arithmetic-intensity ridge, subject to a latency budget) beats both fixed-size batching and no batching on the throughput/p99 frontier.

Falsifier. A simple fixed batch size, tuned once, matches the adaptive policy.

Needs: P01, P13, P14, P07.

Design. The theory is already in tools/roofline.py: decode arithmetic intensity equals batch size, and on an H100 at bf16 the ridge is at batch 295. Arms: no batching; fixed batch at several sizes; adaptive batching with a latency deadline; the roofline-informed policy. Sweep arrival rate and burstiness.

Metrics. Throughput, p50/p99 latency, achieved MFU, queue depth.

Why it is the weakest of the six as research: it is closest to known engineering — continuous batching is what every serving system already does. Why it is still a good choice: it is the most likely to produce a clean, correct, well-measured result, and if what you want from P15 is a defensible finished artifact rather than a novel one, this is the safe pick.

Choosing Between Them

The recommendation: Q2 or Q5.

Q2 if you want the highest probability of a clean, complete, defensible result. Three components, no external data, a well-defined baseline curve you already have from P02, and a claim that is easy to state and hard to fudge. It is the low-variance choice and there is nothing wrong with that after 117 weeks.

Q5 if you want the most interesting question and can accept more risk. It uses the systems stack you spent Stage 2 and 3 building, it sits in a real gap in the literature, and it is the one most likely to change how you think about your own production systems. The risk is integration surface — five components, and freshness instrumentation across all of them.

Q4 is the best science and the worst project-management risk. Take it only if you have confirmed dataset access before week 119, and accept that the likely finding is negative.

Do not take Q3. Six components is too many for thirteen weeks with no project after it to absorb a slip.

The Evaluation Standard

Whatever the question, the evaluation must have all eight. These are the difference between a result and a demo.

1. ≥2 baselines, one of which is a degenerate configuration of your own system — the intervention switched off. This controls for implementation quality, which an external baseline does not.
2. Ablations removing each component the claim depends on, separately. If removing one changes nothing, it is not part of the mechanism and the claim should not mention it.
3. ≥5 seeds per configuration with bootstrap confidence intervals.
4. A negative control — a configuration where you predict no effect. If it shows one, your harness is measuring itself, and you need to know that before you write the paper rather than after a reviewer asks.
5. Sensitivity analysis over the two parameters most likely to be doing the work.
6. The claim re-tested under fault injection. The interesting question is not whether the system survives; it is whether the effect survives.
7. Full cost accounting — latency, memory, storage, compute for every arm. An improvement that costs 10× the compute is a different claim.
8. Latency decomposed by stage. A single end-to-end number is not actionable and is usually hiding the interesting part.

Items 1 and 4 are the ones reviewers ask about first, and they are the two most often missing.

Integration Hazards

Named in advance because each has cost someone a project.

References

Question-specific literature is on the individual project pages. For the integration and evaluation itself:

• Blackburn, S. M. et al. The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations. ACM TOPLAS 38(4), 2016. The source of most of the evaluation standard.
• Hoefler, T., Belli, R. Scientific Benchmarking of Parallel Computing Systems. SC 2015. Twelve rules for reporting performance; apply all of them.
• Collberg, C., Proebsting, T. A. Repeatability in Computer Systems Research. CACM 59(3), 2016. Read before writing the reproducibility appendix.
• Peyton Jones, S. How to Write a Great Research Paper. Microsoft Research, 2004.
• Zobel, J. Writing for Computer Science, 3rd ed. Springer, 2014.
• Bailey, D. H. Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers. Supercomputing Review, 1991. A checklist of things not to do, still entirely current.
• Ousterhout, J. Always Measure One Level Deeper. CACM 61(7), 2018. The argument for evaluation standard item 8 — decomposition, not end-to-end numbers.