Week One — The Action Plan

Eleven hours, five sessions, starting today. By Sunday you will have: a repository, a working benchmark harness calibrated on your own machine, the journey's reference numbers reproduced, a character tokenizer with a passing test suite, a data loader, and your first committed prediction.

No reading of Vaswani et al. this week. That is scheduled for week 2, after your naive design exists.

Table of Contents

• Before Session 1
• Session 1 — Monday, 2 h
• Session 2 — Tuesday, 2 h
• Session 3 — Wednesday, 2 h
• Session 4 — Thursday, 2 h
• Session 5 — Weekend, 3 h
• End-of-Week Checklist
• If You Only Have Four Hours
• What Week Two Looks Like

Before Session 1

First: have you taken the calibration battery? Three hours, cold, before week 1. It sizes P01, P02 and P11-I to you rather than to my guess, and the adjustments it produces change this week's plan. If you have not, do that first.

Then, fifteen minutes, right now.

1. Decide your five slots. Actual days and times, in your calendar, as recurring events, for the next twelve weeks. Not "evenings" — Monday 20:00–22:00. See designing your week.
2. Write your pace on a note you will see. 11 h/week → 34 months. If you have chosen differently, use the table and write that number instead.
3. Create the directories:

mkdir -p ~/systems/{notebook/weekly,projects}
cd ~/systems && git init

That is it. Do not spend session 1 on tooling. See Not Yet.

Session 1 — Monday, 2 h

Objective: the harness runs on your machine and you have your own numbers.

1.1 — Commit the AI policy (15 min)

Read the ten rules. Then:

cd ~/systems
cp <this-track>/ai-policy.md notebook/ai-policy-reference.md
$EDITOR notebook/000-ai-commitment.md   # paste and date the commitment
git add -A && git commit -m "AI assistant policy, committed before week 1"

Copy the commitment verbatim and date it. Committing it before any code is the point — it is a precommitment, and it is worthless afterwards.

1.2 — Run the tools (30 min)

cp -r <this-track>/tools ~/systems/tools
cd ~/systems/tools

python3 bench.py demo
python3 roofline.py table
python3 roofline.py decode --params 7e9 --hw h100
python3 metrics.py demo
python3 bloom.py

Record your bench.py demo environment block and numbers in notebook/001-machine-baseline.md. They will differ from the reference values in these pages. That is expected and it is why you are running them.

The reference machine produced, among others:

Look at the sum(list) comparison specifically. Two identical functions, and the harness says so rather than manufacturing a 1% win. That is the behaviour you want from every benchmark you write for the next 34 months.

1.3 — Reproduce the machine numbers (45 min)

The P14 measurements and P12 costs are quoted throughout this track. Get yours now — they take twenty minutes and you will refer back to them for two years.

Predict each before you measure. Write the predictions in notebook/001-machine-baseline.md first.

# CPU matmul progression — the P14 table
cat > mm.c <<'EOF'
#include <stdio.h>
#include <string.h>
#include <time.h>
static double now(void){struct timespec t;clock_gettime(CLOCK_MONOTONIC,&t);
  return t.tv_sec+t.tv_nsec/1e9;}
#define N 512
static float A[N*N],B[N*N],C[N*N];
int main(void){
  for(int i=0;i<N*N;i++){A[i]=(float)((i*31)%7)-3;B[i]=(float)((i*17)%5)-2;}
  double f=2.0*N*N*N,t;
  t=now();
  for(int i=0;i<N;i++)for(int j=0;j<N;j++){float s=0;
    for(int k=0;k<N;k++)s+=A[i*N+k]*B[k*N+j];C[i*N+j]=s;}
  t=now()-t; printf("naive   i,j,k  %8.2f GFLOP/s\n",f/t/1e9);
  memset(C,0,sizeof(C)); t=now();
  for(int i=0;i<N;i++)for(int k=0;k<N;k++){float a=A[i*N+k];
    for(int j=0;j<N;j++)C[i*N+j]+=a*B[k*N+j];}
  t=now()-t; printf("reorder i,k,j  %8.2f GFLOP/s\n",f/t/1e9);
  return 0;
}
EOF
cc -O2 -o mm mm.c && ./mm
cc -O3 -ffast-math -o mm3 mm.c && ./mm3

Reference values at -O2: naive 1.91, reordered 27.33 GFLOP/s — a 14.3× difference from changing the loop order and nothing else. If your ratio is far from that, your compiler auto-vectorised the naive version; check with -Rpass=loop-vectorize.

Then your platform's BLAS, for the ceiling:

python3 -c "
import numpy as np, time
N=512; A=np.random.rand(N,N).astype(np.float32); B=np.random.rand(N,N).astype(np.float32)
for _ in range(5): A@B
ts=[]
for _ in range(30):
    t=time.perf_counter(); A@B; ts.append(time.perf_counter()-t)
ts.sort(); t=ts[len(ts)//2]
print(f'BLAS {N}x{N}: {t*1e3:.3f} ms  {2*N**3/t/1e9:.1f} GFLOP/s')"

Reference: 1,679 GFLOP/s (Apple Accelerate, which dispatches to the AMX matrix coprocessor). Your hand-written reordered loop will be 20–60× behind. That gap is Project 14's entire thesis and you now have it on day one.

1.4 — Close the session (30 min)

Write notebook/001-machine-baseline.md properly: your predictions, your numbers, the ratios, and one paragraph on which prediction was furthest off and why. Then:

cd ~/systems
cat > notebook/RESUME.md <<'EOF'
# RESUME
PROJECT : P01, milestone 1
STATE   : tools running, machine baseline recorded
NEXT    : create projects/p01-transformer/, write the character tokenizer
          with encode/decode round-trip test
COMMAND : cd ~/systems/projects/p01-transformer && pytest
EOF
git add -A && git commit -m "week 1: machine baseline"

Session 2 — Tuesday, 2 h

Objective: the repository, the tokenizer, and a test that fails then passes.

2.1 — Project skeleton (20 min)

cd <this-track>/scaffold
./new-project.sh p01-transformer python ~/systems/projects/p01-transformer
cd ~/systems/projects/p01-transformer
make setup                      # venv, deps, and requirements.lock

The scaffold gives you the layout every project page assumes: RESUME.md, EXIT-CRITERIA.md with the eight-item completion gate, AI-LOG.md, a benchmark driver wired to the shared harness, and results/ committed rather than ignored. See the scaffold's README for why each of those is there.

make setup writes requirements.lock from pip freeze. Pin versions now — in eighteen months a version drift will explain a discrepancy you would otherwise spend a day on.

make test is red on a fresh scaffold, on purpose. Your first act in the repo is to open the placeholder test and replace it, which is where the reminder lives that property tests come before the code they cover.

2.2 — Get a corpus (10 min)

Anything 1–5 MB of plain text you find interesting. Public-domain books, your own writing, a documentation dump. Prefer something you know well — you will be reading generated samples for eight weeks and it helps to recognise when they are wrong.

Save as data/corpus.txt. Record its size and character-set size in the README.

2.3 — Character tokenizer, test first (60 min)

Write the test before the implementation.

# tests/test_tokenizer.py
import pytest
from src.tokenizer import CharTokenizer

def test_round_trip_ascii():
    t = CharTokenizer.from_text("hello world")
    assert t.decode(t.encode("hello world")) == "hello world"

def test_round_trip_unicode():
    s = "héllo 世界 🌍"
    t = CharTokenizer.from_text(s)
    assert t.decode(t.encode(s)) == s

def test_vocab_is_sorted_and_unique():
    t = CharTokenizer.from_text("banana")
    assert t.vocab == sorted(set("banana"))

def test_ids_in_range():
    t = CharTokenizer.from_text("banana")
    assert all(0 <= i < t.vocab_size for i in t.encode("banana"))

def test_unknown_character_policy():
    t = CharTokenizer.from_text("abc")
    with pytest.raises(KeyError):      # decide the policy, then TEST it
        t.encode("z")

Run it, watch it fail, then write src/tokenizer.py until it passes.

The last test matters more than it looks. What happens on an unseen character is a design decision, and most tokenizers make it silently. Decide — raise, or map to an <unk> id — and encode the decision in a test. This is the smallest possible instance of the habit the whole journey trains.

2.4 — Close (30 min)

Record the compression ratio (bytes per token — 1.0 for a character tokenizer, which is the baseline BPE must beat in milestone 8). Update RESUME.md. Commit.

Session 3 — Wednesday, 2 h

Objective: the data loader, and the bigram baseline you must beat.

3.1 — Data loader (45 min)

Train/validation split with a held-out contiguous tail, not a random split — random splitting on text leaks context across the boundary and inflates your validation score.

def get_batch(data, batch_size, block_size, rng):
    ix = rng.integers(0, len(data) - block_size - 1, size=batch_size)
    x = np.stack([data[i:i+block_size]     for i in ix])
    y = np.stack([data[i+1:i+block_size+1] for i in ix])
    return x, y

Test: shapes are (B, T); y is x shifted by exactly one; the same seed gives the same batch; no index ever reads past the end.

3.2 — Bigram baseline (60 min)

An embedding table straight to logits. Twenty lines. Train it.

Before you run it, write two numbers in the notebook:

1. The entropy floor: \(\ln(V)\) nats for your vocabulary size — the loss of a model that has learned nothing. For \(V=65\), that is 4.17.
2. Your predicted bigram validation loss.

Then train and compare. Most people predict too low, because character-level bigram statistics are weaker than intuition suggests.

This number is the thing your Transformer must beat, and knowing it now stops week 7 from being either falsely triumphant or falsely disappointing.

3.3 — Close (15 min)

Record both numbers, your prediction error, and one sentence on why you were off. RESUME.md. Commit.

Session 4 — Thursday, 2 h

Objective: your first committed prediction, and the attention cost model.

4.1 — Start notebook entry 002 (45 min)

Copy templates/notebook.md to notebook/002-attention-cost.md and fill sections 1–8 only.

The question: at what sequence length does attention's quadratic term start to dominate the linear feed-forward term, for a model with \(d_{model} = 384\), 6 heads?

Section 4 — your naive design — is: work it out on paper. Count the multiply- accumulates in each component. Do not look at P01's table yet.

Section 5 — predict the crossover \(T\), before computing it.

git add notebook/002-attention-cost.md
git commit -m "P01 E-cost: predictions committed before computation"

The commit is the mechanism. It is what makes the prediction real.

4.2 — Write the cost model (60 min)

# src/cost_model.py
def transformer_flops(B, T, H, d_head, n_layers, vocab):
    d = H * d_head
    qkv    = 3 * 2 * B * T * d * d
    scores = 2 * B * H * T * T * d_head      # quadratic
    av     = 2 * B * H * T * T * d_head      # quadratic
    proj   = 2 * B * T * d * d
    ffn    = 2 * 2 * B * T * d * (4 * d)
    per_layer = qkv + scores + av + proj + ffn
    return {
        "per_layer": per_layer,
        "total": per_layer * n_layers + 2 * B * T * d * vocab,
        "quadratic_share": (scores + av) / per_layer,
        "score_matrix_bytes": B * H * T * T * 4,
    }

Sweep \(T \in \{128, 512, 1024, 2048, 4096, 8192\}\) for your configuration. Compare against your prediction, then against the reference table (which uses \(H=12, d_h=64\), so your numbers will differ — the shape should not).

Two things to notice, and to write down:

• At \(T=1024\), attention is only 18.2% of the FLOPs in the reference configuration. The quadratic term does not dominate until ~4096.
• The materialised score matrix is 3.2 GB at \(T=8192\) for one batch element in fp32. Memory hits the wall long before FLOPs do, and that is why FlashAttention exists.

4.3 — Close (15 min)

Fill notebook sections 9–14. Be honest in section 10 about how far off your prediction was. Commit.

Session 5 — Weekend, 3 h

Objective: the naive design, the week's reflection, and week 2's objective.

5.1 — Write your naive attention design (75 min)

The most important 75 minutes of week 1. Still no Vaswani.

In notebook/003-attention-naive-design.md, sections 1–5:

You have a sequence of \(T\) vectors. Each position must gather information from earlier positions, and which earlier positions must depend on content, not on fixed offset. Design it.

Constraints to hold yourself to:

• It must parallelise across positions during training (no sequential recurrence)
• It must handle variable \(T\) without changing the parameter count
• It must be differentiable end to end

Write the mechanism, the shapes, the parameter count, and the cost. Then section 4's second half: why you chose each part. Then section 5: where you predict it breaks, and at what \(T\).

Most people invent one of: averaging embeddings (loses order), a fixed-window MLP (parameter count grows with window, and it cannot generalise across positions), or a recurrence (does not parallelise). Some invent something close to attention. All four outcomes are useful — what matters is that the reasoning is written down before you read the paper, because in week 2 you will diff your design against the real one and that diff is the learning.

git commit -am "P01: naive attention design, pre-literature"

5.2 — Weekly review (45 min)

Fill in notebook/weekly/2026-W01.md using the template. All six weekly outputs should be present. Be specific in "what I avoided because it was hard".

5.3 — Set up week 2 (30 min)

Week 2's objective: a single attention head, forward pass, hand-verified on three tokens.

Prepare it now so Monday starts with action:

• Create tests/test_attention.py with the hand-computed test stubbed out — the arithmetic you will do on paper on Monday
• Read P01's milestone 3
• Queue the reading: Vaswani et al. §3 only, 1 hour, Monday. Not before

5.4 — Buffer (30 min)

Overrun, or cleanup, or stop early. If everything is done, stop early — finishing a week with time left is a signal the pace is sustainable, and you should notice it.

End-of-Week Checklist

• Five calendar slots booked for the next twelve weeks
• notebook/000-ai-commitment.md committed and dated
• All five tools run; your environment block recorded
• Your machine's matmul progression measured (naive / reordered / BLAS)
• Repository created, dependencies pinned
• Tokenizer passes five tests including the unknown-character policy
• Data loader tested; contiguous held-out split
• Bigram baseline trained; entropy floor and validation loss recorded
• Cost model written; crossover \(T\) computed and compared to your prediction
• Notebook entry 002: sections 1–8 committed before the computation
• Notebook entry 003: naive attention design, written before reading Vaswani
• Weekly review written
• RESUME.md current
• Week 2's objective written and its first test stubbed

Eleven of these are artifacts. Two are habits. The two habits — committing predictions before results, and designing before reading — are the ones the other 129 weeks depend on.

If You Only Have Four Hours

A bad first week happens. Do these three things, in this order, and nothing else:

1. Run the tools and record your machine's numbers (1 h). Everything downstream compares against these.
2. Repository, tokenizer, data loader, bigram baseline (2 h). This is the spine of P01; without it week 2 has nothing to build on.
3. The naive attention design (1 h). Non-negotiable and non-deferrable — once you read the paper it is gone forever.

Skip the cost model and the C benchmark; fold them into week 2. Declare it as maintenance mode in the log, with the reason.

What Week Two Looks Like

The full twelve weeks: First 12 Weeks.

Weeks 13 onward are not pre-written. You generate them, 45 minutes per project, with The Week Generator.