Calibration — Before Week One

A 3-hour battery, taken cold, that sizes the first three projects to you rather than to my guess about you.

Without it, P01's eight weeks assume a starting point I invented. If attention is already second nature, weeks 1–4 are waste. If Rust ownership defeats you in P11-I, the whole Stage-2 schedule is wrong and you discover it in month five.

This is not a test you can fail. Every outcome maps to a schedule, and the "you already know this" outcomes save you more time than the "you need longer" outcomes cost.

Table of Contents

• Rules
• The Five Tasks
• C1 — Attention, From the Description
• C2 — Nearest Neighbour, Naive to Better
• C3 — Systems Arithmetic
• C4 — Ownership and Lifetime
• C5 — Measurement Judgement
• Scoring
• What Your Scores Change
• Re-Calibration
• References

Rules

Take it in one sitting. Split across days it measures your best day rather than your working level.

The Five Tasks

C1 — Attention, From the Description

45 minutes. No reference to any Transformer material.

Here is a specification with no code and no formula. Implement it.

You have \(T\) vectors \(x_1 \dots x_T\), each of dimension \(d\), as a matrix X of shape (T, d).

Produce an output Y of the same shape, where each output row \(y_i\) is a weighted average of transformed input rows \(1 \dots i\) — never rows after \(i\).

The weights must depend on the content of the rows, not their positions: row \(i\) decides how much attention to pay to row \(j\) by comparing a vector derived from \(x_i\) against a vector derived from \(x_j\). The weights over \(j\) must be non-negative and sum to 1.

The number of learnable parameters must not depend on \(T\).

Deliverables, in this order:

1. The mechanism in words, before any code. What are the learnable parameters and what are their shapes?
2. A numpy implementation, attention(X, Wq, Wk, Wv) -> Y.
3. A test that the weights sum to 1 for every row.
4. A test that row \(i\) is unaffected by any change to rows \(> i\).
5. Its computational cost in FLOPs, as a function of \(T\) and \(d\).

Self-score after time is up:

What separates 3 from 5 is not implementation skill; it is whether the arbitrary constants prompt a question. That is exactly the habit P01 exists to build, so a 3 here is a good reason to do P01 in full.

C2 — Nearest Neighbour, Naive to Better

40 minutes.

You have 100,000 vectors of dimension 128 in memory, and a stream of query vectors. For each query, return the 10 most similar by cosine similarity.

1. Write the exact solution. (10 min)
2. Estimate its per-query cost — arithmetic operations and bytes moved — before running it. Write the estimate down, then measure. (10 min)
3. Without looking anything up, propose two ways to make it sub-linear, and for each state precisely what you give up. (15 min)
4. For one of them, state the experiment that would tell you whether it works, and what result would make you abandon it. (5 min)

Self-score:

Step 2 is the one that matters. The habit of estimating before measuring is what the whole notebook is built around, and it is independent of whether you have seen ANN search before.

C3 — Systems Arithmetic

25 minutes. Pen and paper. No calculator beyond arithmetic.

Eight questions. Order-of-magnitude answers are fine; state your assumptions, which matter more than the number.

1. A service handles 10,000 requests/second at 50 ms mean latency. How many requests are in flight on average?
2. Your process reads 4 KB from a file 100,000 times. The data is in the OS page cache. Roughly how long does the syscall overhead alone take?
3. A 7-billion-parameter model in bf16 generates one token. How many bytes must move from memory, at minimum? At 2 TB/s, how long is that?
4. A key-value store does one fsync per write. What write rate can it sustain, to within an order of magnitude?
5. An LSM tree with fanout 10 holds 64 GB with a 64 MB first level. Roughly how many levels, and how many times is a byte rewritten under leveled compaction?
6. A request fans out to 50 services, each exceeding its p99 latency 1% of the time. What fraction of requests are slow?
7. A 512×512 fp32 matrix multiply. How many FLOPs? At 20 GFLOP/s, how long?
8. You want to detect a 1% relative change in a metric with standard deviation equal to its mean. Order of magnitude of users per arm?

Answers with the reasoning are in the numbers reference — but score yourself before looking:

Self-score: 1 = 0–2 correct to an order of magnitude · 3 = 4–5 correct · 5 = 7–8 correct with assumptions stated.

This is the highest-signal task in the battery. Every project in the track opens with a prediction, and back-of-envelope fluency is what makes predictions worth writing down.

C4 — Ownership and Lifetime

40 minutes. Rust if you have it installed; otherwise answer in prose and C.

Implement a scope chain: an Environment with a map from name to value and an optional parent. Support get (searching up the chain) and define.

Then: a Closure holds a function body and a reference to the environment where it was created. Two closures created in the same scope must see each other's writes to that scope.

1. Write the type definitions. (15 min)
2. Implement get and define. (15 min)
3. Answer in writing: what keeps the environment alive after the function that created it returns? What happens if two closures both mutate the same variable? (10 min)

Self-score:

A 1 or 2 is the single most schedule-relevant result in the battery, because Rust is the journey's one real language cost and P11-I is where it is paid.

C5 — Measurement Judgement

30 minutes. Five benchmarks. For each: what is wrong, and what would you measure instead? Roughly 6 minutes each.

B1. To measure syscall cost:

t0 = now(); for (i=0;i<1e7;i++) getpid(); t1 = now();
printf("syscall: %.2f ns\n", (t1-t0)/1e7*1e9);   // prints 1.23 ns

B2. To measure DRAM latency, a pointer chase where each pointer points 64 bytes ahead, wrapping at the end of a 512 MB buffer. Reports 1.3 ns per load.

B3. To compare two sorting functions:

t0=time.time(); sort_a(data); t1=time.time()
t2=time.time(); sort_b(data); t3=time.time()
print(f"b is {(t1-t0)/(t3-t2):.2f}x faster")     # prints 1.04x faster

B4. A load generator sends a request, waits for the response, then sends the next. Under a 1-second server stall it reports p99 = 12 ms.

B5. A Bloom filter at 24 bits/key is tested with 200,000 absent keys. Zero false positives observed. Reported as "false-positive rate: 0%, better than the predicted 1e-5".

Self-score: 1 = spotted 0–1 · 3 = spotted 3 · 5 = spotted all five and named the correct replacement measurement for each.

Every one of these is a mistake made and documented during the construction of this track (§14). Spotting them cold is genuinely hard, and a low score here is the least worrying of the five — it is the most teachable.

Scoring

Record in notebook/000-calibration.md:

C1 attention          _/5
C2 nearest neighbour  _/5
C3 systems arithmetic _/5     <- highest signal
C4 ownership          _/5     <- most schedule-relevant
C5 measurement        _/5
                     ___
total                 _/25

Then run:

cd tools && python3 calibrate.py C1 C2 C3 C4 C5     # e.g. python3 calibrate.py 3 4 2 1 3

It prints the adjusted schedule and the total delta in weeks.

What Your Scores Change

Totals

A high score does not mean skip the experiments. Every reduction above cuts implementation weeks. The experiments, the notebook entries and the reports are the part that builds the capability you asked for, and they are never the thing to cut.

Re-Calibration

Re-take C3 and C5 at each stage review — M7, M15, M22, M26, M31. Both are 25–30 minutes and both measure skills the track claims to build, so a flat score across two stages is evidence the program is not working for you and something needs to change.

C1, C2 and C4 do not repeat well: once you have built the thing they ask about, they measure memory rather than aptitude.

Record each re-take alongside the first in notebook/000-calibration.md so the trajectory is visible. Along with the scorecard trend, that is the only objective evidence you will have that 34 months of evenings changed something.

References

• Ericsson, K. A. et al. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review 100(3), 1993. Practice must target the edge of current ability; a plan not sized to the learner is not deliberate practice.
• Vygotsky, L. S. Mind in Society. Harvard University Press, 1978. The zone of proximal development — the argument for calibrating difficulty rather than fixing it.
• Bjork, R. A., Bjork, E. L. Desirable Difficulties in Theory and Practice. JARMAC 9(4), 2020. Why the correct adjustment for a high scorer is less scaffolding, not less work.
• Kruger, J., Dunning, D. Unskilled and Unaware of It. JPSP 77(6), 1999. Self-assessment is least reliable at the low end, which is why every task here has concrete 1/3/5 anchors rather than asking how confident you feel.
• Wiggins, G., McTighe, J. Understanding by Design, 2nd ed. ASCD, 2005. Diagnostic assessment that feeds directly into instructional design, which is what the adjustment table is.