Projects That Can Become Original Research

Seven directions that emerge from this journey's projects and could produce a genuine contribution — a workshop paper, an industry-track submission, a well-cited blog post, or an open-source tool people use.

Calibration first. "Original research" here does not mean a NeurIPS oral. It means: a question nobody has answered, an experiment that answers it, and a write-up honest enough that someone else can build on it. Most of the value of these seven is that they are achievable by one person with a laptop and a well-scoped question — which is exactly what the fifteen projects prepare you to do.

Maintain this page as you go. At each stage review, add anything you saw that might be novel. One line each. Do not chase it then; chase it in Stage 6 or after.

Table of Contents

• How To Tell If Something Is Novel
• D1 — Adaptive ANN Search Policy
• D2 — Connectivity-Preserving Graph Indexes for Clustered Data
• D3 — Adaptive Compaction
• D4 — Simulator Validity for Recommender Evaluation
• D5 — Freshness as a First-Class Systems Metric
• D6 — Watermark-Delay Auto-Tuning
• D7 — Adaptive Task Sizing Instead of Speculation
• Ranking Them
• Where To Publish
• References

How To Tell If Something Is Novel

Before investing, run this four-step check. It takes an afternoon and it saves months.

1. Search properly. Google Scholar, DBLP, arXiv, and — critically — the related work sections of the three most recent papers in the area. If it is obvious, it has been done; your job is to find out how and what they missed.
2. Search for the negative result too. Many good ideas have been tried and did not work, and that is often unpublished. Check the issue trackers and mailing lists of the relevant open-source projects. A closed issue saying "we tried this, it did not help, here is why" is worth more than a paper.
3. Ask what would make it not-novel. If a well-known system already does this internally (and it often does), your contribution is the measurement, not the idea. That is still publishable — as an evaluation paper — but it is a different paper.
4. Write the abstract first. If you cannot write a 150-word abstract with a number in it, the question is not sharp enough yet.

A note on the honest case. The most likely outcome for each of these is: someone has explored it, your version adds a careful measurement in a specific regime, and the result is a good blog post rather than a paper. That is a fine outcome and it is worth doing. The failure is not "it turned out not to be novel" — it is spending six months before checking.

D1 — Adaptive ANN Search Policy

From: P02 extension, P15 candidate question Q2.

The question. efSearch is fixed per index. But queries differ enormously in difficulty — some land in a dense, well-connected region and converge in 200 distance computations; others land near a boundary and need 4,000. Can you predict per-query difficulty from the first few hops and set the beam width per query, achieving the same mean recall at lower mean latency?

Why it is plausible. Your own P02 data shows the raw material: recall is strongly concave in efSearch, and per-query recall variance (E10) is large. If 80% of queries reach target recall at ef=32 and 20% need ef=256, a fixed ef=128 overspends on most and underspends on the rest.

The signal to use. Candidate features available after ~20 hops: the distance to the best result so far, the rate of improvement over the last k hops, the variance of distances in the current beam, and the number of times the beam's worst element has been replaced. Cheap, all of them.

The experiment. Fixed ef sweep as the baseline curve. Then adaptive policy with a budget matched to each fixed point. The claim is Pareto dominance: at equal mean recall, lower mean and p99 latency. Report p99 specifically — an adaptive policy that improves the mean by making hard queries much worse is not an improvement.

The falsifier. Adaptive fails to dominate a well-tuned fixed ef on the Pareto frontier. Note that the honest baseline is well-tuned fixed ef, not a badly chosen one.

Prior art to check: learned index tuning, early-termination criteria in ANN search, adaptive query processing in databases (which has decades of literature and is the right place to look for how this goes wrong).

Difficulty: medium. Novelty: medium — early termination exists; per-query budget prediction with a p99 constraint is less explored.

D2 — Connectivity-Preserving Graph Indexes for Clustered Data

From: P02's E7 and the worked notebook entry.

The question. The measured result: on clustered data (RC 3.36), single-layer NSW hits a recall ceiling of 0.967 at ef=256 while uniform data reaches 0.993 — and the clustered search evaluates only 840 distances versus 4,059, because it runs out of reachable candidates. The mechanism is that a distance-based degree cap deterministically deletes every inter-cluster bridge (intra-cluster distance 0.521, inter-cluster 1.413).

HNSW's neighbour-selection heuristic addresses this. How completely? And is there a better rule for strongly-modal data specifically?

Why it matters practically. Real embedding corpora are strongly modal — multilingual spaces, category-structured catalogues, near-duplicate clusters. The failure mode is specific and nasty: recall looks fine on a uniform synthetic benchmark and degrades on the real corpus, in a way that increasing efSearch does not fix. That is a falsifiable claim about deployed systems.

The experiment. Sweep modality (number of clusters × separation, using RC as the axis rather than \(d\)). For each, compare: naive M-nearest pruning, HNSW's Algorithm 4, and a candidate rule that explicitly reserves a fraction of each node's degree budget for edges crossing a detected boundary. Measure the recall ceiling, the inter-cluster edge fraction, and the distance count at saturation.

The instrument that makes it credible: inter-cluster edge fraction before and after pruning. That is the mechanism metric, and it is what turns "recall is lower" into "the bridges were deleted".

The falsifier. Algorithm 4 already preserves enough connectivity that the reserved- budget rule adds nothing across the whole RC range.

Prior art: Filtered-DiskANN, and the HNSW paper's own §4 discussion. Check whether the ANN-Benchmarks datasets span enough modality to have surfaced this.

Difficulty: medium. Novelty: medium-high — the characterisation by RC is the part most likely to be new, since most ANN evaluation reports \(d\) and dataset name rather than a difficulty measure.

D3 — Adaptive Compaction

From: P04 extension.

The question. Leveled and size-tiered compaction sit at opposite corners of the read/write/space trade — at T=10 and 64 GB, leveled costs W/R/S of 31/4/1.10 and size-tiered 4/30/2.11. Real workloads shift: bulk ingest, then read-heavy serving, then a re-index. Can an engine observe its own read/write ratio and switch strategy, beating either fixed choice on a realistic shifting workload?

Why it is plausible. The two strategies differ by an order of magnitude in opposite directions, so the potential win is large. RocksDB exposes both and lets an operator choose; it does not choose for you.

The hard part, which is the actual research content. Switching is not free — it requires rewriting data into the new layout, so a policy that switches too eagerly pays migration cost repeatedly. The question becomes a hysteresis problem: how much evidence of a workload shift justifies a migration whose cost you can estimate? That framing is more interesting than the switching itself.

The experiment. A workload generator with phase transitions (ingest → serve → ingest) of varying frequency. Compare: fixed leveled, fixed size-tiered, an oracle that switches with perfect foresight, and your online policy. The oracle is the key baseline — it bounds the achievable gain and tells you whether the online problem is even worth solving.

The falsifier. The oracle's gain over the better fixed strategy is small, in which case no online policy can matter.

Prior art: Dostoevsky and the LSM-tuning literature (Dayan & Idreos); RocksDB's compaction-style options; adaptive indexing / database cracking.

Difficulty: medium-high. Novelty: medium — tuning is well studied; online switching with migration cost less so.

D4 — Simulator Validity for Recommender Evaluation

From: P09 + P10. Also P15 candidate question Q4.

The question. Recommender simulators are widely used (RecSim, RecoGym) and rarely validated. Under what conditions does a simulated user population correctly rank real algorithms? Not "is the simulator realistic" — that is unanswerable — but the useful version: which properties must the user model have for its ordinal conclusions to transfer?

Why this is the most scientifically valuable direction here. It is a methodological result. If you can show that ordinal fidelity depends on a small number of user-model properties (say, position bias and topic fatigue) and is insensitive to the rest, that is a genuinely useful finding for everyone who builds these simulators. And if you show it does not transfer, that is arguably more valuable and much less comfortable.

The experiment. You already have the machinery: P08's offline rankings, P09's simulated rankings, P10's rank correlation (E12), and P09's sensitivity analysis (E11). The research version adds a third leg — real interaction data, even a public dataset — and asks which of the three rankings agree.

The falsifier. Ordinal agreement is unstable across user-model parameters, with no identifiable subset of properties controlling it. This is the most likely outcome, and it is publishable as a negative result — "simulator-based recommender evaluation does not produce stable rankings under plausible model variation" is a useful thing for the field to know.

The honest risk: without real data you can compare P08-offline to P09-simulated, and neither is ground truth, so you get a disagreement without an arbiter. Securing a real validation set is the gating step. MIND (Microsoft News) is the obvious candidate and is in your domain.

Difficulty: medium. Novelty: high. Data risk: high.

D5 — Freshness as a First-Class Systems Metric

From: P03 + P04 + P07 + P08. P15 candidate question Q5.

The question. In news recommendation, an article's value decays in hours. Every storage and indexing decision — segment size, compaction schedule, index rebuild cadence, watermark delay — adds latency between publication and recommendability. Nobody measures that end-to-end, and nobody optimises for it. What is the publication-to-recommendable latency distribution of a realistic stack, where does it come from, and what is the quality cost of each contribution?

Why it is a real gap. Storage papers report throughput and amplification. Recommender papers report NDCG. The composition — how much recommendation quality is lost to indexing latency — falls between the two literatures, which is exactly where under-studied questions live.

The experiment. Instrument the full P15 pipeline for freshness. Decompose the publication-to-recommendable latency by stage: ingestion, embedding, index insert, segment flush, index visibility. Then sweep the parameters that trade freshness against efficiency (segment size, flush interval, batch size) and measure recommendation quality at each point. The output is a freshness/efficiency Pareto frontier with quality contours — a figure that does not currently exist.

The falsifier. Quality is insensitive to freshness in the range achievable by parameter tuning, i.e. everything is fast enough already and the question is moot.

What makes it credible: the decomposition. A single end-to-end number is not a result; "62% of publication-to-recommendable latency is index segment flush, and halving the segment size cuts it by X at Y cost in query latency" is.

Difficulty: high (needs the most of the stack). Novelty: high — genuinely cross-cutting.

D6 — Watermark-Delay Auto-Tuning

From: P07 extension.

The question. Watermark delay \(\delta\) is a fixed constant chosen by an operator, trading completeness against latency. The lateness distribution is observable at runtime. Can \(\delta\) be tuned online to hold a stated completeness SLO — "99% of events included in their correct window" — at minimum latency?

Why it is plausible. The lateness distribution is directly measurable and usually stable over hours but variable across days (mobile-sync patterns, batch uploads, regional traffic). A fixed \(\delta\) must be set for the worst case, so it over-delays most of the time.

The hard part. Watermarks must be monotonic, so you can lengthen the delay freely but shortening it is constrained — you cannot un-emit a watermark. That asymmetry makes the control problem interesting rather than trivial, and it is where the contribution is.

The experiment. Replay realistic lateness distributions (including regime changes). Compare fixed \(\delta\) at several values, an oracle with perfect foresight, and the adaptive policy. Metrics: completeness achieved vs SLO, mean and p99 emission latency, SLO violation rate during regime change.

The falsifier. The adaptive policy cannot beat a fixed \(\delta\) set at the observed 99th percentile of lateness — which is a strong and simple baseline.

Prior art: Dataflow's watermark discussion; Flink's watermark strategies; check whether the streaming-systems literature already has adaptive proposals (it has some).

Difficulty: medium. Novelty: medium.

D7 — Adaptive Task Sizing Instead of Speculation

From: P06 extension.

The question. MapReduce handles stragglers by duplicating slow tasks, which wastes the work already done and consumes capacity. An alternative: detect a slow task and split its remaining work among idle workers. Under what conditions does splitting beat duplication?

Why it is plausible. Duplication wastes up to 2× the task's work by construction. Your own simulation shows the stakes: 1% of tasks at 50× slowness inflates job time 4.48×, and backup tasks recover it to 1.09×. If splitting recovers the same and costs half the wasted CPU, that is a real result on cluster efficiency rather than latency.

The hard part. Splitting requires tasks to be resumable — the remaining input must be identifiable and the partial output combinable. That is a stronger contract than MapReduce requires, and characterising exactly what contract is needed is part of the contribution. It also connects directly to the project's own thesis about restriction buying automation.

The experiment. Straggler injection matrix (frequency × severity × cause: slow CPU vs slow disk vs contention). Compare: no mitigation, duplication, splitting, and both. Metrics: job completion time and wasted CPU-seconds, because the claim is about the second.

The falsifier. Split overhead exceeds the saving except in a narrow regime.

Prior art: SkewTune (which does something close for skew rather than stragglers), the LATE scheduler, Dryad's dynamic refinement. Check SkewTune carefully — this may already be it, in which case the contribution is the comparison against duplication under straggler (not skew) conditions.

Difficulty: medium-high. Novelty: low-medium — check prior art first, seriously.

Ranking Them

If you pursue one after the journey, this is the order.

D2 is first for a reason worth stating. It is not the most important question on the list — D5 probably is. It is first because you will finish P02 with the anomaly already measured, the mechanism already diagnosed, and the next experiment already written down in your notebook. The gap between "interesting idea" and "running experiment" is where research dies, and D2 has no gap.

Where To Publish

Ordered by effort, not by prestige. Each step is a legitimate destination, not merely a stepping stone.

Do not skip the blog post. It is the cheapest way to discover whether the result survives contact with readers, it produces the writing you would need anyway, and for systems work a well-measured post with reproducible code frequently reaches more of the relevant audience than a workshop paper does. See portfolio.md for the publication sequence.

References

• Hamming, R. W. You and Your Research. Bell Communications Research, 1986. On choosing important problems — the framing for ranking them.
• Peyton Jones, S. How to Write a Great Research Paper. Microsoft Research, 2004. Write the abstract first; the check in step 4.
• Malkov, Y. A., Yashunin, D. A. HNSW. IEEE TPAMI 42(4), 2020. — D1, D2
• Gollapudi, S. et al. Filtered-DiskANN. WWW 2023. — D2
• He, J., Kumar, S., Chang, S.-F. On the Difficulty of Nearest Neighbor Search. ICML 2012. — D2's RC axis
• Dayan, N., Idreos, S. Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores. SIGMOD 2018. — D3
• Idreos, S., Kersten, M. L., Manegold, S. Database Cracking. CIDR 2007. — D3's adaptive-indexing precedent
• Chaney, A. J. B., Stewart, B. M., Engelhardt, B. E. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. RecSys 2018. — D4
• Ie, E. et al. RecSim. arXiv:1909.04847, 2019. — D4
• Wu, F. et al. MIND: A Large-scale Dataset for News Recommendation. ACL 2020. — D4's validation data
• Akidau, T. et al. The Dataflow Model. VLDB 2015. — D6
• Kwon, Y. et al. SkewTune: Mitigating Skew in MapReduce Applications. SIGMOD 2012. — D7. Read this before starting D7
• Zaharia, M. et al. Improving MapReduce Performance in Heterogeneous Environments. OSDI 2008. — D7