The Research Notebook

Fourteen fields, each explained: what it is for, what a good answer looks like, and the specific way each one is usually filled in badly.

Table of Contents

How To Use It

One entry per experiment, not per project. A Medium project produces 4–8 entries; notebook/ accumulates roughly 80 of them over the journey.

Entries are written in order, and sections 1–8 are written before the experiment runs. That is the entire mechanism. A notebook filled in afterwards is a report with extra headings; a notebook filled in beforehand is an instrument that catches you being wrong.

Commit each entry as soon as sections 1–8 exist, before running anything. The commit timestamp is what makes the prediction credible — to a reader, and more importantly to you in six months when you are tempted to remember having predicted correctly.

1. Problem

What is it for: forcing you to state the problem independently of any solution.

A good answer describes what breaks in the world if this system does not exist, in terms of an actual quantity. "Exact kNN over 10M items at 5,000 QPS is 5×10¹⁰ distance computations per second, which no single machine can do."

The bad version names a technology. "I need to implement HNSW" is not a problem statement, it is a solution statement, and writing it means sections 4 and 5 will be worthless because you have already chosen the answer.

Test: could someone who has never heard of the canonical solution understand what you need? If not, rewrite.

2. Constraints

What is it for: surfacing the assumptions that will later turn out to be doing all the work.

A good answer separates three kinds: hard (physics, the machine you have), chosen (design decisions you could revisit), and assumed (things you believe but have not verified). Label them. The third category is where the interesting failures come from.

The bad version lists only the hard constraints, so the assumed ones stay invisible until they break.

Test: the thinking exercise — which assumption is doing the most work? If it is not in this list, add it.

3. Existing Approach

What is it for: honestly recording what you know before you design, so section 4 is not contaminated and you can tell later how much was yours.

A good answer says exactly how much of the literature you have read, and stops reading at a stated point. From the worked example: "I read the abstract and the algorithm pseudocode, and then stopped before the neighbour-selection heuristic, on purpose, so that section 4 would be mine."

The bad version is a thorough summary of the canonical solution — after which your "naive design" is a half-remembered version of it, and the whole exercise is dead.

Test: did you stop reading somewhere deliberate, and did you write down where?

4. My Naive Design

What is it for: this is the single most important field in the template.

Reconstructing a system from its constraints is the skill this entire journey exists to build, and it is only trainable if you actually attempt the design before seeing the answer. Once you have read the paper you cannot un-read it, and the opportunity for that week is gone permanently.

A good answer is a real design: components, data structures, the algorithm, and — critically — the reasoning that led to it. The reasoning matters more than the design, because section 11 is going to diagnose the reasoning, not the code.

The bad version is one line, written grudgingly, so that the form is filled in.

Test: could you implement from it? And does it record why you chose each part?

The worked example's naive design was wrong in a specific, diagnosable way, and its recorded reasoning — "early insertions happen into a nearly-empty graph, so their edges are necessarily long" — is exactly what section 11 was able to take apart. A one-line design would have produced a one-line failure analysis.

5. Predictions

What is it for: calibration. Over 80 entries you learn how good your intuition actually is, which is information you cannot get any other way.

A good answer is a table of specific, quantified, falsifiable statements, each with a confidence level. "10–100× faster at n=10,000, high confidence."

The bad version is hedged. "It should be faster" cannot be wrong, so it teaches nothing.

Test: could each row be marked confirmed or falsified without argument? Track your hit rate across entries — a rate near 100% means your predictions are too safe.

In the worked example, two of five predictions survived. That ratio is healthy. If most of yours survive, you are predicting things you already know.

6. Hypothesis

What is it for: the difference between an experiment and a measurement.

A good answer has two parts, both mandatory:

H: ⟨a specific claim about a causal relationship⟩ Falsifier: ⟨the observation that would make me abandon it⟩

The bad version omits the falsifier, at which point you will unconsciously interpret whatever happens as support.

Test: can you describe, concretely, a result that would make you say "I was wrong"? If not, this is a measurement, which is fine — but label it as one and skip to section 7.

7. Experimental Setup

What is it for: making the result mean something to a reader, including future you.

A good answer records: hardware including current load, software versions, data including how it was generated, parameters including the ones you did not vary, the exact command, and the seed.

The bad version omits the load average and the library versions — the two things that most often explain an irreproducible number.

And validate the setup before running. The worked example caught a data generator whose independent variable did not vary (σ√d = 2.0 made "clustered" data statistically identical to uniform). That check cost ten minutes and saved a worthless run. Measure your independent variable to confirm it varies.

8. Baseline

What is it for: a number without a comparison is not a result.

A good answer names a baseline that is fair and explains why. The strongest baseline is usually a degenerate configuration of your own system — the intervention turned off — because it controls for implementation quality.

The bad version is a straw man: an unoptimised comparison chosen because you can beat it.

Test: would a skeptical reviewer accept this as the right comparison? Would you, if someone else's paper used it?

9. Results

What is it for: the data. Only the data.

A good answer is tables and plots with units, sample sizes, and uncertainty. No interpretation — interpretation is sections 10 and 11, and mixing them lets a conclusion smuggle itself in as an observation.

The bad version reports only the configurations that worked, or reports means without spread.

Always include: the verdict on each prediction from section 5, as a table. That is what closes the loop.

10. Surprises

What is it for: this is where the learning is, and it is the section most often left empty.

A good answer names each result that contradicted section 5, and quantifies the gap. A surprise you cannot quantify is a vague feeling.

The bad version is "everything went as expected", which is nearly always false and means you did not look at the secondary metrics.

If this section is genuinely empty, ask the thinking exercise: what variable am I failing to measure? The worked example's biggest finding came from the distance counter, not from recall or latency.

11. Failure Analysis

What is it for: turning a wrong prediction into a mechanism.

A good answer names the specific design decision that caused the failure, and traces the causal chain. Not "clustering is hard" but "the degree cap prunes by raw distance; intra-cluster distance is 0.521 and inter-cluster is 1.413, so pruning deletes every bridge deterministically."

The bad version is a category. "Cache effects", "Python is slow", "high dimensions are hard" — these are labels, not analyses, and none of them tells you what to change.

Test: does the analysis point at a specific line you could change? And is it specific enough to be wrong? A failure analysis that cannot be refuted is a story.

12. Next Experiment

What is it for: choosing the cheapest experiment that resolves the most uncertainty.

A good answer gives one experiment with its cost, its prediction, and what each outcome would mean. "E1 — instrument inter-cluster edge fraction. Cost: 40 minutes. Prediction: below 1% after pruning. If confirmed, the analysis above is established; if not, my explanation is wrong."

The bad version is a list of six things you might do, in no order, with no costs.

The rule: do the cheap decisive experiment before the expensive thorough one. A 40-minute test that settles the main question beats a three-hour test that improves a number.

13. Generalization

What is it for: the step from "I ran an experiment" to "I know something".

A good answer states where the result should hold, where it should not, and what it predicts about systems you have not built. That last part is what makes it knowledge rather than a data point.

The bad version over-generalises from one dataset on one machine.

Test: does it make a falsifiable claim about something outside this experiment? The worked example's "recall that looks fine on a uniform synthetic benchmark and degrades on a real clustered corpus, in a way that adding efSearch does not fix" is a prediction about other people's production systems. That is the shape to aim for.

14. Reproducibility

What is it for: the criterion that separates a result from an anecdote.

A good answer is a block someone else can execute:

commit    : <sha>
command   : <exact invocation>
seed      : <value>
runtime   : <expected>
output    : <where the raw data landed>
expect    : <headline number ± tolerance>

The bad version says "run the script".

Test: delete your build directory, follow your own instructions, and check the number. Do this before writing the report, not after someone asks.

The Ordering Rule

The template's power is entirely in the ordering, so it is worth stating once more as a rule:

Sections 1–8 are written and committed before the experiment runs. Sections 9–14 are written after.

Every mechanism in this journey that produces the skills you asked for depends on that line:

  • Writing 4 before reading trains reconstruction.
  • Writing 5 before running trains calibration.
  • Writing 6 before running makes it an experiment rather than a demonstration.
  • Writing 8 before running stops you from picking the baseline you beat.

Fill it in afterwards and you have a well-organised report that taught you nothing new about your own reasoning. The commit timestamp on sections 1–8 is not bureaucracy — it is the only thing standing between you and the near-universal human tendency to remember having predicted the thing that happened.

References

  • Feynman, R. P. Cargo Cult Science. Caltech, 1974. "The first principle is that you must not fool yourself — and you are the easiest person to fool." Sections 5, 6 and 8 are that principle turned into a form.
  • Platt, J. R. Strong Inference. Science 146(3642), 1964. Devise alternative hypotheses, devise an experiment to exclude one, repeat. Section 12 is strong inference.
  • Popper, K. The Logic of Scientific Discovery. Hutchinson, 1959. Falsifiability; section 6's falsifier field.
  • Kahneman, D. Thinking, Fast and Slow. FSG, 2011. Hindsight bias and the illusion of validity — the empirical reason section 5 must be timestamped.
  • Tetlock, P., Gardner, D. Superforecasting. Crown, 2015. Calibration improves only with recorded predictions and scored outcomes; the reason to track your hit rate.
  • Wilson, G. et al. Best Practices for Scientific Computing. PLoS Biology 12(1), 2014.
  • Sandve, G. K. et al. Ten Simple Rules for Reproducible Computational Research. PLoS Computational Biology 9(10), 2013. Section 14 is these ten rules compressed.
  • Collberg, C., Proebsting, T. A. Repeatability in Computer Systems Research. CACM 59(3), 2016.