Systems From First Principles

← All Roles

A 15-project, 130-week program that trains you to reconstruct foundational computer systems from their constraints — Transformers, ANN indexes, vector databases, LSM engines, distributed key-value stores, MapReduce, stream processors, recommenders, experimentation platforms, a programming language, an OS kernel, a tensor framework, and a matrix-multiplication accelerator simulator — then measure them, break them, modify them, and write them up like papers.

Not a curriculum. There are no lectures to consume. Every unit ends in running code, a benchmark with tail latencies, a falsified or surviving hypothesis, and a technical report.

Table of Contents

• The Honest Duration
• What This Trains
• Start Here
• The Twenty Deliverables
• Repository Layout
• The Runnable Tools
• The Universal Project Loop
• Operating Rules
• What This Journey Refuses To Do
• Honest Status
• References

The Honest Duration

You asked for 18–30 months at 8–12 focused hours per week, and you asked me not to compress the plan artificially. Those two constraints are in tension, so here is the arithmetic before anything else.

The fifteen projects, scoped to minimum-viable plus standard implementation and excluding every optional research extension, total 1,430 focused hours across 130 active weeks. The per-project derivation is in Project Portfolio; the weeks are the same numbers divided by an 11 h/week pace.

Calendar time is not active weeks. Assume 46 productive weeks per year — six weeks lost annually to travel, illness, work crunch, and the two weeks after any production incident at your day job. Then:

Headline: 34 months. Roughly two years and ten months. I am not going to tell you 18, because 18 months × 12 h/week is 936 hours and you cannot build a kernel and a language and a distributed KV store and a tensor framework in 936 hours at a depth that produces defensible original work. The compressed version of this plan exists and it is called a tutorial series.

If the 30-month ceiling is genuinely hard

There are exactly six scope cuts that preserve the core mechanism of every project. Take all six — 15 weeks — and you land at 115 active weeks / 1,265 hours / exactly 30 months at 11 h/week:

Taking only the first five saves 13 weeks, which lands at 117 weeks / 1,287 hours / 30.5 months — near enough that the sixth cut is optional.

Do not achieve 30 months by working faster on the full scope. That produces fifteen half-built systems and no reports, which is worth less than five finished ones.

Full derivation, dependency graph, and stage boundaries: Roadmap.

What This Trains

The stated goal is the technical depth of people who built MapReduce, Spanner, TensorFlow, TPUs, language runtimes, kernels, databases, and Transformers. That depth is not a body of knowledge. It is a set of habits, and each one is trained by a specific mechanic that appears in every project:

The failure mode this program is designed against is yours, specifically: you told me you spread attention across too many subjects and stay in consumption mode. So the guardrails are not advice, they are rules with a stated enforcement mechanism. See Guardrails. The single most important one:

One primary implementation project at a time. Reading never completes anything.

Start Here

You do not need to read the fifteen project pages before you start. Read Project 1 and the Roadmap; read each subsequent project page in the week before you begin it. Reading all fifteen now is itself the consumption-mode failure this program exists to break.

The Twenty Deliverables

Every item you asked for, and where it lives.

Plus seven you did not ask for but will need:

Repository Layout

README.md              this file
roadmap.md             duration derivation, dependency graph, 6 stages, 34 monthly milestones
projects.md            master table, effort model, three scope tiers
projects/              p01..p15 — the full specification of each build
first-12-weeks.md      week-by-week for weeks 1-12
week-one.md            hour-by-hour for week 1, starting now
week-generator.md      how to decompose weeks 13-130 yourself, with 3 worked examples
calibration.md         the 3-hour battery, taken before week 1
STATE.md               the ledger spanning all 130 weeks
portfolio-audit.md     what the fifteen cover, cover twice, and miss
reading-method.md      three-pass reading, and reading code
retention.md           review queue, rebuild drills, forced reuse, teaching tests
feedback-loops.md      adversarial self-review, publishing, reproduction exchange
datasets-and-hardware.md  what it needs, what it costs, the two real dependencies
numbers.md             every constant, derived and measured, with what it decides
proofs.md              18 derivations, each verified by tools/proofs.py
glossary.md            ~90 concepts, explained rather than defined, 18 with runnable code
walkthroughs.md        six 45-minute executable miniatures
walkthroughs/          w1..w6 -- the scripts behind them
handson/               h01..h15, the 15 generated pages, and build_pages.py
scaffold/              new-project.sh + templates for python / rust / go
scores/                scorecard.json, trended by tools/scorecard.py
operating-model.md     weekly rhythm, time allocation, thinking exercises, guardrails
notebook-template.md   the research notebook, explained field by field
notebook/              example-filled.md — a real, completed notebook entry
templates/             notebook.md, report.md, experiment.md — copy these
scorecard.md           12 categories, 1/3/5 anchors, completion definition
ai-policy.md           the ten rules, with enforcement
languages.md           Python / Go / Rust / C / CUDA — which mechanism each exposes
math.md                just-in-time mathematics, tied to the project that needs it
readings.md            every primary source, by project, with why-read-this notes
portfolio.md           artifacts, publication venues, the paper
not-yet.md             what to actively refuse to study, and when it unlocks
research-directions.md the seven that can become original work
final-system.md        Project 15's architecture and its research question
sustainability.md      completing this while employed full-time
tools/                 bench.py, roofline.py, metrics.py, annlab.py, bloom.py

The Runnable Tools

Five scripts you use across the whole journey. All are executed and their real output is quoted in the pages that reference them — no unverified numbers appear anywhere in this track.

cd tools

cc -O2 -o machine-baseline machine-baseline.c && ./machine-baseline
                               # regenerates every hardware constant in numbers.md
./machine-baseline --json > mine.json && python3 baseline.py mine.json
                               # diff vs the reference + 8 invariant checks
python3 calibrate.py 3 4 2 1 3 # calibration scores -> adjusted schedule
python3 scorecard.py trend     # 12 categories x 15 projects, what is rising
python3 proofs.py              # 116 checks over every derivation in proofs.md
python3 -m pytest tests/ -q    # 63 tests over the tools themselves

python3 bench.py demo          # the measurement harness: p50/p95/p99 + bootstrap CI
                               # + an honest "no measurable difference" verdict
python3 roofline.py table      # ridge points; the batch size that ends memory-boundedness
python3 roofline.py decode --params 7e9 --hw h100
python3 roofline.py gemm --m 4096 --n 4096 --k 4096
python3 metrics.py demo        # recall/precision/NDCG/MRR/coverage/novelty/Gini,
                               # Welch t, power sizing, sample-ratio-mismatch chi-square
python3 bloom.py               # Bloom theory vs measurement, and where measurement fails
python3 annlab.py              # brute force vs a small-world graph: the recall/latency curve

bench.py, metrics.py, and roofline.py are dependency-free. annlab.py needs numpy.

Read tools/bench.py before you write a single benchmark. It encodes four rules — distribution not mean, warmup separated from measurement, bootstrap interval on the median, environment recorded — and if you internalise only one artifact from this whole track, that is a good one to pick.

The Universal Project Loop

Every one of the fifteen projects runs the same fourteen steps, in order. The loop is the actual curriculum; the projects are just the substrate it runs on.

  1  Problem definition        What breaks if this system does not exist?
  2  Requirements/constraints  What is fixed? What is negotiable? What is assumed?
  3  Naive design              Your design, written BEFORE you read the paper.
  4  Predicted failure points  Where and at what scale you expect it to break.
  5  Minimal implementation    The smallest thing that exhibits the mechanism.
  6  Correctness testing       Property tests and invariants. Before any timing.
  7  Instrumentation           Counters and timers in the code, not around it.
  8  Baseline measurement      A number you can be beaten by.
  9  Bottleneck analysis       Where the time actually goes. Measured, not guessed.
 10  Hypothesis                One falsifiable claim, with its falsifier stated.
 11  Modification              The smallest change that tests the hypothesis.
 12  Controlled experiment     One variable. Fixed seeds. Repeated trials.
 13  Failure analysis          What broke, why, and what it generalises to.
 14  Technical report          Including the results that made you wrong.

Steps 3 and 4 are the ones people skip, and they are the ones that produce the ability you actually want. A person who has read the MapReduce paper knows what MapReduce is. A person who designed a batch framework, predicted that stragglers would dominate, and then measured a 6× tail inflation from one slow worker understands why the paper has a section on backup tasks. Those are different skills and only one of them survives a hard design review.

Full field-by-field guidance: Notebook Template. A completed example with real measured numbers: Worked Notebook Entry.

Operating Rules

Applied without exception. Each is here because it prevents a specific documented failure mode of self-directed systems study.

• One primary implementation project at a time. Plus at most one small maintenance or writing task from an earlier project, plus at most one bounded reading thread that directly supports the current project. Nothing else.
• Reading never completes anything. A unit is done when code runs, a benchmark is recorded, and a report exists.
• No optimization without a baseline. If you cannot state the number you are trying to beat, you are not optimizing, you are fiddling.
• No performance claim without tail latency. Mean latency is the number that hides the bug.
• No performance work while a correctness test is red. Fast and wrong is a regression, not a result.
• Predictions are written before runs. A prediction recorded after the result is not a prediction, and the habit it builds is the opposite of the one you want.
• Negative results are shipped, not buried. The report section titled "What I Expected And Did Not Get" is mandatory and is scored.
• Extensions are locked until exit criteria are met. Every project has an attractive optional extension. It is not available until the standard version passes.
• Every project produces an artifact that is useful on its own. If it only makes sense as part of the journey, it is not a portfolio piece.
• A stalled project gets a written postmortem, not a silent abandonment. See the two-week rule.

What This Journey Refuses To Do

Stated up front so you can disagree now rather than in month nine.

• It will not make anything production-grade. No auth, no multi-tenancy, no packaging, no ops. Every hour spent on production polish is an hour not spent on a mechanism. The one exception is Project 15, and only for the specific subsystem its research question depends on.
• It will not chase completeness. The kernel will not run a shell. The language will not have a standard library. The database will not be ACID. Completeness is a substitute activity for depth.
• It will not let you rewrite things in a new language because the new language is interesting. Language choice is fixed per project and justified by which mechanism it exposes.
• It will not treat complexity as originality. A system with nine components and no measured claim scores lower than a system with two components and a falsified hypothesis. See Scorecard.
• It will not pretend the schedule survives contact with your job. See Sustainability, which budgets for the bad quarters instead of assuming they will not happen.

Honest Status

This track is a plan and a toolkit, not a completed body of work. What exists today:

• Complete: the roadmap and its arithmetic, all fifteen project specifications, the first twelve weeks in day-level detail, the week-one hour-by-hour plan, the notebook template plus one fully worked entry with real measured numbers, the scorecard, the AI policy, the language and mathematics strategies, the portfolio plan, and five executed, verified tools.
• By design absent: the fifteen implementations. Those are yours. This track will not contain a reference solution for any project, because a reference solution available on day one destroys step 3 of the loop, which is the step that matters most.
• Numbers: every measured figure quoted in these pages was produced by running the script named next to it on the machine described in tools/bench.py's environment block — a 12-core arm64 macOS laptop, CPython 3.14.0, under a nonzero load average. They are illustrative of shape, not of your hardware. Re-run them on your machine in week 1; the numbers will differ and the shapes will not.

References

The complete, per-project reading list with annotations is in Readings. The handful below define the intellectual posture of the whole track and are worth reading in the first month.

On the method

• Hamming, R. W. You and Your Research. Bell Communications Research Colloquium Seminar, 1986. The canonical talk on choosing important problems and working with open doors. Read this in week 1.
• Bell, C. G., Newell, A. Computer Structures: Readings and Examples. McGraw-Hill, 1971. The original argument that systems should be studied as designed artifacts with explicit tradeoffs.
• Lampson, B. W. Hints for Computer System Design. SOSP '83, ACM Operating Systems Review 17(5), 1983. Thirty-odd design maxims from someone who built the systems. "Handle normal and worst case separately" is the one you will re-derive most often.
• Feynman, R. P. Cargo Cult Science. Caltech commencement address, 1974. On the discipline of not fooling yourself, which is the entire content of the experimental sections here.
• Wilson, G. et al. Best Practices for Scientific Computing. PLoS Biology 12(1), 2014. Reproducibility as an engineering practice.

On measurement

• Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM 52(4), 2009. Implemented in tools/roofline.py.
• Gregg, B. Systems Performance: Enterprise and the Cloud, 2nd ed. Pearson, 2020. The USE method, and the best available treatment of "which of the four resources is actually saturated".
• Dean, J., Barroso, L. A. The Tail at Scale. CACM 56(2), 2013. Why p99 is the number that matters and why it gets worse as you add machines.
• Mytkowicz, T. et al. Producing Wrong Data Without Doing Anything Obviously Wrong! ASPLOS '09. Measurement bias from link order and environment size — read before you trust your first speedup.
• Fleming, P. J., Wallace, J. J. How not to lie with statistics: the correct way to summarize benchmark results. CACM 29(3), 1986. Why the arithmetic mean of ratios is wrong.

On writing systems up

• Shewchuk, J. R. Three Sins of Authors in Computer Science and Math. 1997.
• Zobel, J. Writing for Computer Science, 3rd ed. Springer, 2014. The standard reference for the report format used in templates/report.md.
• Peyton Jones, S. How to Write a Great Research Paper. Microsoft Research, 2004. Specifically: write the paper first, then do the research — which is the same instruction as "write your hypothesis before you run the experiment".