Project Scorecard

Twelve categories, scored 1–5 at the end of every project, with anchors defined for 1, 3 and 5. Scores 2 and 4 are interpolations.

Score down when unsure. A generous rubric is the one thing that guarantees this journey produces nothing. The purpose of the scorecard is not to feel good about a project; it is to find the category that is weakest so the next project can train it.

Table of Contents

How To Score

  1. Do the adversarial self-review first — eleven questions, one hour, one night after finishing. Scoring before that scores your own view of your work.
  2. Score after the report is written, not before. Record with python3 tools/scorecard.py record <project> --interactive.
  3. For each category, name the evidence. A score without a cited artifact is a mood.
  4. If you are between two scores, take the lower one.
  5. Record all twelve in the report's self-assessment table.
  6. At each stage review, plot the trend. The trend matters more than any single score.

A realistic first project scores 2–3 across most categories. Scoring 4s in Stage 1 means the rubric is being read generously, not that the work is exceptional.

1. First-Principles Understanding

Can you derive the design rather than recall it?
1You implemented what a tutorial or paper described. Asked "why this constant?", you say "that is what the paper uses."
3You can explain every major design decision and what it trades against. You derived at least one constant (e.g. the \(\sqrt{d_k}\) scale, the 10-bits/key Bloom default) rather than adopting it.
5You designed a substantial part before reading the canonical solution, can explain why the canonical design differs from yours, and can construct the conditions under which your version would have been correct.

Evidence: the notebook's section 4, and the report's section 3.

2. Correctness

Does it work, and how do you know?
1It runs on the examples you tried. Testing is manual.
3A real test suite: property tests, invariants asserted in code, edge cases covered. You can state what is not tested.
5Model-based or differential testing against a reference; fuzzing; invariants asserted continuously in production paths; deliberate fault injection. At least one real bug was found by a test rather than by observation.

Evidence: the test suite, and a named bug that testing caught.

A 5 requires that a test found something you did not know. Tests that only confirm what you already believed are documentation.

3. Implementation Depth

Did you build the mechanism, or configure one?
1The core mechanism comes from a library. You wrote the glue.
3The central mechanism is hand-written and complete. Peripheral concerns use libraries, and the boundary is explicit and defensible.
5The mechanism is hand-written, and you also built at least one layer below the one required — a compiled inner loop, a custom allocator, a hand-vectorised kernel — with the improvement measured.

Evidence: the permitted-library line from the project page, and what you did not import.

4. Code Quality

Could a competent engineer work in this codebase?
1One file, unclear names, no structure. Only you can navigate it.
3Sensible module boundaries, meaningful names, comments that explain why, a README that gets a stranger running.
5The interfaces are the kind another project can build on — and one did. Errors are informative. The code reads as an explanation of the design.

Evidence: a later project importing this one without modification is the strongest possible evidence for a 5.

Note this is weighted lower than it would be at work. These are research artifacts. Beautiful code with no measurements scores far worse overall than workmanlike code with a falsified hypothesis.

5. Systems Reasoning

Do you know where the time and bytes go?
1Performance is a mystery. You changed things until it got faster.
3You profiled, identified the bottleneck, and can classify it as computational, algorithmic, architectural, or operational.
5You predicted the bottleneck before measuring and were approximately right; you decomposed a ratio into its independent factors (as in P02's algorithmic-vs-constant-factor split); and you can state what the next bottleneck will be after this one is fixed.

Evidence: a prediction recorded before profiling, and a decomposition rather than a single ratio.

6. Experimental Rigor

Would the result survive a skeptical reviewer?
1One run, one configuration, no seed, no baseline.
3Fixed seeds, repeated trials, one variable at a time, a stated baseline, uncertainty reported.
5Predictions timestamped before runs; a negative control; sensitivity analysis over the parameters most likely to be doing the work; the setup validated before the experiment (independent variable confirmed to vary); and the measurement's resolution stated before any ratio is reported.

Evidence: the commit timestamp on notebook sections 1–8.

The resolution requirement is specific. tools/bloom.py demonstrates it: at 24 bits/key the predicted false-positive rate is ~10⁻⁵, so 200k probes expect 1.2 events and observing 0 is noise, not a result. Reporting a ratio your experiment cannot resolve is a 2, regardless of everything else.

7. Benchmark Quality

Do the numbers mean anything?
1Mean latency. No warmup. Unrealistic workload. Environment unrecorded.
3p50/p95/p99, warmup separated, environment recorded, realistic and skewed workloads, raw samples saved.
5Confidence intervals on the statistics; an honest "no measurable difference" verdict where the intervals overlap; adversarial and degenerate workloads included; and a comparison against a credible external implementation, reported honestly including where you lose.

Evidence: tools/bench.py output, and an external comparison you did not win.

Reporting a mean latency caps this category at 2.

8. Failure Analysis

Do you know how it breaks?
1You tested the happy path. Failures were fixed as encountered, not studied.
3Deliberate fault injection with predicted and observed behaviour; degradation characterised; the overload behaviour known.
5A fault-injection harness with replayable schedules; at least one failure diagnosed to a specific design decision with the causal chain written out; and the system's behaviour at its breaking point characterised rather than merely avoided.

Evidence: the harness, and a failure analysis that names a line rather than a category.

9. Originality of Hypotheses

Are you asking questions, or following instructions?
1No hypothesis. You implemented and measured.
3A falsifiable hypothesis with a stated falsifier, tested with a controlled experiment.
5The hypothesis came from something you observed rather than from a paper; it was falsified or survived on evidence; and the generalization section makes a checkable claim about systems you have not built.

Evidence: notebook sections 6 and 13.

A falsified hypothesis scores the same as a confirmed one. What is scored is whether the question was worth asking and whether the test could have answered "no". If every hypothesis you have ever tested was confirmed, this category caps at 3 — you are testing things you already know.

10. Communication

Can someone else learn from this?
1A README. The results live in your head.
3A complete report: problem, design, method, results, analysis. Figures with units and captions. Someone in your field could follow it.
5The "What I Expected And Did Not Get" section is substantial and specific; limitations and threats to validity are stated without prompting; and the report would survive being read by someone who wanted to find a hole in it.

Evidence: the report's required sections 13 and 15.

An empty section 13 caps this category at 2.

11. Reproducibility

Can someone else get your number?
1It runs on your machine. Setup is undocumented.
3One command to build, one to test, one to reproduce the headline benchmark. Versions pinned. Seeds fixed.
5You followed your own instructions from a clean clone and got the number within tolerance; raw sample data is committed, not just summaries; the environment is specified precisely enough to explain a discrepancy.

Evidence: the report's section 17, verified.

For P15 the bar is higher and absolute: a person who is not you must have reproduced the headline number.

12. Completion Discipline

Did you finish, on the terms you set?
1Abandoned, or drifted past its size class with no decision made.
3Exit criteria met within the class ceiling. Scope cuts, if any, were deliberate and documented.
5Finished within budget; the optional extension was correctly deferred rather than started early; and any scope cut was recorded with its reasoning at the time rather than rationalised afterwards.

Evidence: the exit-criteria checklist, and the weekly logs.

This is the category most predictive of whether you finish the journey. A pattern of 3s here across five projects is a stronger signal than a pattern of 5s anywhere else. Watch its trend specifically.

The Completion Gate

Separate from the scores. A project is complete only when all eight are true:

  • The essential mechanism works
  • Correctness tests pass
  • At least one baseline exists
  • Meaningful metrics are collected
  • At least one hypothesis was tested
  • Negative or unexpected results are documented
  • A technical report is complete
  • The repository can be reproduced by another engineer

These are binary and they are not negotiable by scoring well elsewhere. A project with five 5s and no baseline is not complete; it is an impressive incomplete project, and recording it as complete is how a 34-month plan quietly becomes a 50-month one.

Note that the gate does not require the project to be good. It requires it to be finished and honest. A project whose hypothesis was falsified, whose implementation lost to the baseline, and whose report explains both, passes the gate cleanly.

Using the Scores

Per project: find the lowest category. Name the specific mechanic in the next project that will train it, and write that into the next project's weekly objectives. One category at a time — trying to raise all twelve is trying to raise none.

Per stage (stage reviews): run python3 tools/scorecard.py stage <n> and trend, which plot all twelve across the stage's projects. Three patterns to look for:

  • A category that never moves. You are avoiding it. Failure analysis and originality are the usual suspects, because both require sitting with being wrong.
  • A category that drops when projects get harder. Usually experimental rigor or communication — the things that get cut under time pressure. That is a scheduling problem, not a skill problem, and the fix is in the allocation.
  • Everything rising smoothly. Suspicious. Re-read the anchors; you have probably started scoring against your own past work rather than against the definitions.

Across the journey: the categories you asked to develop map to specific ones. First-principles reasoning → 1. Low-level implementation → 3. Systems intuition → 5. Experimental discipline → 6. Performance analysis → 5 and 7. Original technical thinking → 9. Research-quality communication → 10. Those seven are the journey's actual objectives; the other five are the conditions that make them credible.

References

  • Wiggins, G., McTighe, J. Understanding by Design, 2nd ed. ASCD, 2005. Rubric design: anchors at 1/3/5 with observable evidence, rather than adjectives.
  • Ericsson, K. A. et al. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review 100(3), 1993. Why identifying and targeting the single weakest component beats general practice.
  • Blackburn, S. M. et al. The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations. ACM TOPLAS 38(4), 2016. The source of several anchors in categories 6 and 7.
  • Hoefler, T., Belli, R. Scientific Benchmarking of Parallel Computing Systems. SC 2015. Twelve rules; category 7's 5-anchor is rules 1–4.
  • Kruger, J., Dunning, D. Unskilled and Unaware of It. Journal of Personality and Social Psychology 77(6), 1999. The empirical reason for "score down when unsure".