Datasets, Hardware and Money

What the journey physically needs, what it costs, what to do when you cannot get something, and the two places where a missing resource actually blocks a project rather than merely inconveniencing it.

Headline: the whole journey runs on one laptop for well under $200 total, and only two projects have a genuine external dependency.

Table of Contents

• The Short Version
• Hardware
• The Two Real Dependencies
• Datasets, Project by Project
• Synthetic Data Done Properly
• Money
• Storage and Housekeeping
• Acquisition Timeline
• References

The Short Version

Hardware

What the reference machine was

Everything measured in numbers.md came from a 12-core arm64 laptop with macOS. Nothing in the journey requires that machine or anything like it.

Minimum

8 GB is workable with reduced scales — n=10⁵ in P02, fewer personas in P09 — and you should scale the experiments down rather than skip them. Say so in the reports.

Where less RAM is an advantage

Counterintuitively, P04 is easier to do honestly on a smaller machine. The single most common LSM benchmarking error is a working set that fits in the page cache, which measures memory and reports it as disk. On 16 GB you reach "exceeds RAM" at 20 GB of data; on 128 GB you need 200 GB and most people never bother.

Architecture notes

• arm64 (Apple Silicon): everything works. perf counters are limited, so derive DRAM traffic analytically in P14 where you cannot measure it, and say which you did. The 128 KB L1 is unusual — see numbers §1.
• x86-64 Linux: the best case. perf works fully, O_DIRECT exists (so you can measure device reads honestly, unlike TRAP 4), and QEMU/RISC-V tooling is smoothest.
• Windows: use WSL2. Note that WSL2's filesystem and timer behaviour will distort P04 and P12 measurements; prefer a native Linux partition or a VM for those two.

QEMU for P12

brew install qemu                       # macOS
apt install qemu-system-misc gdb-multiarch   # Debian/Ubuntu

Plus a RISC-V cross-toolchain (riscv64-elf-gcc via Homebrew, or the riscv64-unknown-elf prebuilt from SiFive). Install and verify these in week 1 of P12, not week 3 — the week-3 decision rule exists because toolchain time is pure waste, and the way to spend none is to hit the problem early with a fallback ready.

GPUs

No project requires one. P14 has three paths:

1. NVIDIA GPU available → CUDA, milestone 9 as written.
2. Apple Silicon → Metal compute shaders. Same learning objective, different API, and the roofline analysis is identical.
3. Neither → extend the systolic simulator to model a multi-core vector machine and compare three architectures in simulation. This is the documented substitution and it preserves the objective; you lose the experience of writing a real kernel.

If you want option 1 without owning hardware: a spot A10 or T4 is roughly $0.20–0.60 per hour. P14's milestone 9 is ~8 hours of work but only ~2–3 hours of GPU time if you develop the harness locally first. Budget $20–40, and develop against a CPU stub so you are not debugging syntax on a metered clock.

The Two Real Dependencies

Everything else has a synthetic fallback. These two do not, and both should be resolved early rather than discovered late.

1. Real interaction data — for P09's validation and research direction D4

P09's honest limitation is that a simulator's conclusions are conditional on a user model you wrote. The only way to escape that is to check the simulator's rankings against rankings derived from real behaviour.

MIND (Microsoft News Dataset) is the right corpus: news, which matches your domain; ~160k articles, 1M users, 24M clicks in the large variant; impression logs with both clicks and non-clicks, which is what makes off-policy evaluation possible at all.

• Free, registration required, research-use licence — read it and check it permits what you intend, particularly if you later publish.
• MIND-small (~50k users) is enough for everything in P09/P10 and much faster to iterate on.
• Get it in month 20, not month 23. If the licence or the download blocks you, you want to know while there is time to substitute.

Fallbacks, in order: MovieLens-25M (well-understood, has real ratings, but is not news and has no impression logs — you lose the non-click signal); Amazon Reviews; RecSys Challenge datasets; or your own consumption logs from a feed reader, which is small but genuinely yours.

If none works: P09 and P10 still run fully on simulated users. What you lose is D4 as a research direction and the strongest form of P09's validity claim. Write that limitation explicitly rather than quietly.

2. Article text with timestamps — for P08, P09, P15

You need a corpus with publication times, because freshness is a first-class metric in a news recommender and synthetic timestamps make every freshness result circular.

The RSS harvest is the highest-value action on this page, and it costs 30 minutes in month 1. A cron job appending a few dozen feeds to a JSONL file gives you, by the time you reach P08, a corpus that is real, timestamped, in your domain, and unencumbered. Nobody does this and everybody wishes they had.

month 1 : write a 40-line RSS poller, run it hourly, append JSONL
month 20: ~50k-200k timestamped articles ready for P08/P09/P15
cost    : 30 minutes and a few hundred MB

Datasets, Project by Project

Nine of fifteen projects have no external data dependency at all. The plan is deliberately laptop-and-generator shaped.

SIFT1M / GIST1M for P02

The standard ANN benchmark vectors (1M × 128 SIFT, 1M × 960 GIST), free from the TEXMEX corpus and used by ANN-Benchmarks. Worth having because it makes your recall/QPS curve directly comparable to published ones — which is the external reference point you otherwise lack. ~500 MB.

Synthetic Data Done Properly

Most of this journey runs on generated data, so generating it badly corrupts a lot of results. Three rules, each learned the hard way during construction.

1. Measure the property you think you are varying. The clustered-vs-uniform generator in P02 initially produced two datasets that were statistically identical (RC 1.393 vs uniform's 1.356) because a Gaussian perturbation with σ√d = 2.0 swamped the unit-norm cluster centres. Ten minutes of measurement saved a worthless run.

2. Report the difficulty parameter alongside the result. For vectors that is relative contrast; for key-value workloads it is the Zipf exponent; for streams it is the lateness distribution. A recall number without RC, or a throughput number without the key distribution, is not comparable to anything — including your own result from last month.

3. Include an adversarial case. Uniform-random data is not the easy case, it is usually the hard case for retrieval and the easy case for storage. Generate both the realistic distribution and the pathological one, and report both.

Money

Realistic total: $45 for one book, plus perhaps $40 of GPU time if you take P14's extension. Under $100.

The genuine cost of this journey is 1,430 hours. Everything else is rounding.

Storage and Housekeeping

Fifteen repositories over 34 months accumulate more than you expect.

Commit the raw samples and nothing else large. They are small, they are the evidence, and they are the one artifact that cannot be recreated once the machine changes.

Guard against toolchain rot. Pin versions from the first commit (the scaffold does this via requirements.lock, Cargo.lock, go.mod). In month 30 you will want to re-run a month-3 benchmark, and an unpinned dependency makes that a research project of its own.

Acquisition Timeline

Nothing here is urgent, and two items are worth doing early because their value compounds or their failure modes are slow.

Two items in bold. The RSS harvest because it is the only thing on this page that gets better by starting early, and MIND because it is the only external dependency whose absence changes what research you can do.

References

• Wu, F. et al. MIND: A Large-scale Dataset for News Recommendation. ACL 2020. msnews.github.io — registration and licence terms.
• Jégou, H., Douze, M., Schmid, C. Product Quantization for Nearest Neighbor Search. IEEE TPAMI 33(1), 2011. Source of the SIFT1M/GIST1M benchmark corpora (corpus-texmex.irisa.fr).
• Aumüller, M., Bernhardsson, E., Faithfull, A. ANN-Benchmarks. Information Systems 87, 2020. The evaluation protocol and dataset conventions worth imitating.
• Harper, F. M., Konstan, J. A. The MovieLens Datasets: History and Context. ACM TiiS 5(4), 2015. The main fallback dataset, and an honest account of its biases.
• Clauset, A., Shalizi, C. R., Newman, M. E. J. Power-Law Distributions in Empirical Data. SIAM Review 51(4), 2009. How routinely synthetic power-law data is generated and fitted wrongly — relevant to every Zipfian workload you will generate.
• He, J., Kumar, S., Chang, S.-F. On the Difficulty of Nearest Neighbor Search. ICML 2012. Relative contrast, the difficulty parameter to report with synthetic vectors.
• Blackburn, S. M. et al. The Truth, The Whole Truth, and Nothing But the Truth. ACM TOPLAS 38(4), 2016. On workload selection as a source of misleading evaluation.