The First Twelve Weeks

Weeks 1–8 complete Project 1. Weeks 9–12 begin Project 2. Every week has one objective, one experiment with a prediction, one reflection, and one deliverable — the six weekly outputs.

Week 1 is broken down hour by hour in Week One. For weeks 13 onward, run The Week Generator — the procedure that produced this page.

Table of Contents

• Overview
• Week 1 — Baseline and Naive Design
• Week 2 — One Attention Head
• Week 3 — Masking and the Leak Test
• Week 4 — Multi-Head and the Block
• Week 5 — Training Loop and Deliberate Overfit
• Week 6 — Positional Encodings and RoPE
• Week 7 — Depth, Norm Placement, and Inference
• Week 8 — BPE, Sampling, and the P01 Report
• Week 9 — Brute Force and Ground Truth
• Week 10 — The Naive Graph
• Week 11 — NSW and the Distance Counter
• Week 12 — HNSW and the Hierarchy
• Checkpoint at Week 12
• If You Fall Behind

Overview

Milestone references are to P01's and P02's tables.

Week 1 — Baseline and Naive Design

Objective: the harness runs on your machine, the repository exists, and your attention design is written before you read anything.

Hour by hour: Week One.

Week 2 — One Attention Head

Objective: a single attention head whose forward pass you have verified by hand.

The Monday diff is the week's real content. Three questions, answered in writing:

1. What did the paper do that I did not think of?
2. What did I do that the paper does not — and is my version wrong, or just different?
3. Which of my design decisions did the paper make differently for a reason I can now state?

Most people's naive designs miss one of: the \(1/\sqrt{d_k}\) scale, the separate value projection (many people reuse the keys), or the fact that multiple heads cost the same as one big head. Whichever you missed, that is your finding.

The Wednesday hand-computation is not optional and cannot be delegated. Three tokens, four dimensions, one head. Compute the scores, the softmax, and the weighted sum with a pen. It takes twenty minutes and it is the difference between knowing the formula and knowing the mechanism. Every shape bug for the next six weeks will be diagnosed against this.

Week 3 — Masking and the Leak Test

Objective: causal masking, proven correct rather than assumed.

The leak test, which is the single most valuable test in P01:

def test_causal_mask_no_leak():
    x = make_sequence(T=16)
    out_a = model(x)
    x2 = x.clone(); x2[:, 10:] = shuffled_tokens()   # perturb positions >= 10
    out_b = model(x2)
    assert torch.equal(out_a[:, :10], out_b[:, :10]) # bit-identical

This catches masking after the softmax, off-by-one in the mask, and any accidental bidirectionality — three bugs that all produce a model that trains beautifully and reports an impossibly good validation loss.

Thursday's experiment is the one that calibrates your loss intuition permanently: run with the mask removed and watch validation loss drop below the entropy floor for the task. Seeing label leakage from the outside once means you will recognise it forever.

Week 4 — Multi-Head and the Block

Objective: a complete pre-norm Transformer block.

Monday's parameter-count check is where multi-head attention stops being magic. With \(H\) heads of dimension \(d/H\), the projections are the same total size as one head of dimension \(d\). You get \(H\) independent attention distributions for free. Verify the count numerically; do not take it on faith.

Thursday's gradient check must pass before any training. A model that trains with wrong gradients trains slowly and you will blame the learning rate for a week.

Week 5 — Training Loop and Deliberate Overfit

Objective: it learns. Proven by overfitting.

Wednesday is a gate, not a milestone. If the model cannot drive a 200-token corpus to near-zero loss with no regularisation, there is a bug — almost always a mask applied after the softmax, a detached gradient, or a loader that reshuffles the target. Do not proceed to Thursday until it passes. Debugging a subtle bug on a real corpus costs ten times what it costs here.

The weekend test — train on randomly shuffled targets — should converge to exactly \(\ln(V)\) nats. Below means leakage. Far above means an optimizer problem. Two lines of code, and it calibrates every loss number you will read for the rest of the project.

Week 6 — Positional Encodings and RoPE

Objective: three encodings behind one interface, plus the control.

Write the RoPE tests before the implementation:

def test_rope_relative_position():
    q, k = randn(64), randn(64)
    for (m, n) in [(5,3), (105,103), (1000,998)]:
        d = dot(rope(q, m), rope(k, n))
        assert abs(d - reference_for_offset_2) < 1e-6

def test_rope_preserves_norm():
    q = randn(64)
    for m in [0, 17, 1000]:
        assert abs(norm(rope(q, m)) - norm(q)) < 1e-6

Reference values confirm this holds exactly — the dot product for offset 2 is −8.115791933 at \(m,n\) of (5,3), (105,103), (7,5) and (1000,998) alike, across a 200× range of absolute position, and the norm is preserved to nine decimals. These two tests catch the two standard RoPE bugs: wrong dimension pairing, and rotating the values as well as the queries and keys.

The "none" arm in E4 is the control and it is mandatory. Without positional information, attention is permutation-equivariant and the model can only learn unigram statistics. The gap between "none" and the others is the actual value of positional encoding, and reporting the three encodings without it means reporting differences without a scale.

Week 7 — Depth, Norm Placement, and Inference

Objective: the interaction experiment, and inference that scales.

E6 is the best hypothesis in P01. At depth 2, pre-norm and post-norm perform similarly. Somewhere deeper they diverge sharply. Predict the depth on Monday, before launching, and explain your prediction via gradient magnitude at initialisation.

The mechanism: in post-norm, the residual stream passes through a normalisation on every layer, so gradients are repeatedly rescaled on the way back and can shrink multiplicatively with depth. In pre-norm, the residual path is an unnormalised identity from output to input. That is why pre-norm needs no warmup to train deep models and post-norm does.

E12 is a derivation checked by measurement. KV cache bytes per sequence \(= 2 \times L \times T \times d \times \text{bytes}\) — two for K and V, \(L\) layers. Derive it, then measure it. They must agree within 5%; a gap means you have misunderstood what is cached, which is worth finding out now rather than in P15.

Week 8 — BPE, Sampling, and the P01 Report

Objective: finish Project 1.

Tuesday's compression ratio is the number that connects tokenization to everything else: bytes per token trades directly against effective context length. A BPE tokenizer at 4 bytes/token gives you 4× the context of a character tokenizer at the same \(T\). Record it against week 1's character baseline of 1.0.

Thursday and the weekend are the report, and it takes longer than you expect the first time. Use templates/report.md. The section that matters most is "What I Expected And Did Not Get" — go back through weeks 1–7's predictions and tabulate which were wrong and by how much.

Check the exit criteria explicitly, one box at a time. If a box is unticked, either tick it this weekend or write the limitation into the report. Do not carry an unticked box into P02.

Week 9 — Brute Force and Ground Truth

Objective: the baseline you are allowed to trust, and a difficulty measurement.

Wednesday is where the journey's method shows up as a habit. Before running any index, measure whether your datasets actually differ. Reference values at n=10,000: uniform d=64 gives RC 1.356; 100 clusters at σ=0.05 gives 3.371; and at σ=0.25 it gives 1.393 — statistically indistinguishable from uniform, because \(\sigma\sqrt{d} = 2.0\) exceeds the unit-norm cluster centres.

Verify your independent variable varies. Ten minutes; saves a worthless week.

Weekend's metric-equivalence test proves that on normalised vectors, ranking by cosine, dot product, and L2 give identical results — because \(\|a-b\|^2 = 2 - 2\langle a,b \rangle\). Prove it in a test, because forgetting it in week 13 is a silent recall bug.

Week 10 — The Naive Graph

Objective: a graph index that works badly, measured.

This week deliberately builds the wrong thing, and that is step 3 of the loop at the level of a week. A random graph has no locality, so greedy descent has nothing to descend. Measuring exactly how bad it is gives you the number that NSW must beat, and understanding why it is bad is what makes NSW's insertion rule obvious in week 11 rather than arbitrary.

Weekend's diagnosis is the real work. For queries where recall is 0, instrument where the walk terminated. Was it a local minimum? Did the beam exhaust? Was the true neighbour unreachable from the entry point? Those three failure modes have three different fixes, and distinguishing them now is what makes week 11 fast.

Week 11 — NSW and the Distance Counter

Objective: a real graph, and the instrument that explains it.

Wednesday's counter is the most important instrument in P02. It separates the algorithmic question (how many distances?) from the implementation question (how much does each cost?). Without it, week 11's result is "my index is slower than brute force and I don't know why."

With it, you get the two-factor model, which on the reference machine closed to two significant figures:

algorithmic win : 10,000 / 1,459 distances =    6.9x fewer
constant factor : 899 ns vs 13.3 ns/dist   =   67.5x slower each
predicted       : 6.9 / 67.5               =   0.10x
measured        :                              0.10x

Your graph index will be slower than brute force at n=10,000, and that is correct. Expect it, write the prediction down on Monday, and let the counter tell you which of the two factors is responsible. This is the week that teaches the most transferable debugging technique in the whole journey.

Week 12 — HNSW and the Hierarchy

Objective: the layered index, and the first cross-project synthesis.

Wednesday's comparison must be at equal distance count, not equal wall clock. The hierarchy's contribution is algorithmic — fewer distances for the same recall — and comparing wall clock conflates it with implementation noise. If the curves are identical at equal distance count, your hierarchy is not doing anything and the layer assignment is probably wrong.

Thursday's reachability test is the one that will matter in week 14, when you hit the clustered-data recall ceiling. Write it now.

Checkpoint at Week 12

Not a full stage review — that is week 26 — but an honest look.

Artifacts that should exist:

• P01: complete, report shipped, exit criteria ticked or limitations written
• P02: brute force, random graph, NSW, HNSW, recall harness, distance counter
• ~6 notebook entries, sections 1–8 committed before results in every one
• 12 weekly reviews
• RESUME.md current

Habits that should be forming:

• You write predictions before running things without being reminded
• You reach for p50/p95/p99 automatically and feel uneasy reporting a mean
• You measure your independent variable before trusting an experiment
• You have caught yourself in at least one wrong belief with a measurement

Numbers to check against the plan:

If you are more than 15% over on hours, do not resolve to work faster. Cut scope from P02 using its tier table — the MVI is 35 hours and still teaches the core mechanism — and record the cut.

The most important question: are you still doing the loop, or have you started just building? Check notebook entries 002 and 003 against your most recent one. If sections 4, 5 and 6 have got thinner, the discipline is eroding, and week 13 is the cheapest possible moment to fix it.

If You Fall Behind

Falling behind in the first twelve weeks is normal and is not a signal about the next 118. Three graduated responses:

One week behind — absorb it. Week 8's report can take a weekend more; P02 starts in week 10 instead of 9. Do not compress a milestone to catch up.

Two to three weeks behind — cut scope, not quality. Specifically: skip BPE (week 8) and keep the character tokenizer, deferring BPE to P01's extension. That saves 8 hours and costs nothing downstream — no later project needs BPE.

Four or more weeks behind — the pace is wrong, not your effort. Recompute the whole schedule at your actual hours using the duration table, move the end date, and continue at the real pace. A 40-month journey completed beats a 34-month journey abandoned in month nine.

What not to do: attempt to work 20 hours a week to catch up. It does not work, and the crash costs more than the deficit. See when work takes a quarter.