External Feedback

The weakest structural property of this track, stated plainly: every score in it is self-assigned, against a rubric I wrote, by the person who did the work. The scorecard even cites Kruger–Dunning as the reason to score down when unsure — which is an admission that self-assessment is unreliable, followed by no mechanism to fix it.

Thirty-four months of that produces a confident practitioner of your own blind spots.

This page is the correction: five mechanisms, ordered by cost, of which the first two are free and mandatory.

Table of Contents

• Why Self-Assessment Is Not Enough
• The Five Mechanisms
• 1. The Adversarial Self-Review
• 2. Publishing Early
• 3. The Reproduction Exchange
• 4. A Reviewer
• 5. The Public Artifact
• The Minimum Viable Loop
• How To Receive a Review
• What Feedback Cannot Fix
• References

Why Self-Assessment Is Not Enough

Three distinct failures, each needing a different remedy — which is why one mechanism does not suffice.

1. You cannot see what you do not know. The scorecard asks whether your failure analysis is thorough. If your model of "thorough" is missing a category of failure, you score yourself 4 and are right by your own standard and wrong by the field's. Remedy: a reviewer, or published work that attracts correction.

2. Standards drift toward your own past work. By project eight you are scoring against project seven, not against the anchors. The scorecard warns about this ("everything rising smoothly is suspicious") and cannot detect it. Remedy: a fixed external reference — someone else's code, a published benchmark, a reproduction.

3. Reproducibility is unfalsifiable from the inside. "Another engineer could run this" is a claim about a person who is not you, and you cannot evaluate it. Your machine has your environment, your assumptions, your undocumented step. Remedy: an actual other person, which is why P15 requires one — and month 34 is far too late for the first attempt.

The Five Mechanisms

1 and 2 are mandatory and free. 3 costs a little coordination and fixes the failure nothing else can. 4 is the highest-value and requires another person. 5 is optional.

1. The Adversarial Self-Review

One hour, at the end of every project, before you score it.

Not a re-read. A deliberate role switch: you are now a reviewer who has been asked to find the reason this work should not be believed, and you are being judged on how good your objections are.

The mechanics matter, because the role switch fails if you do it from the same chair:

• Wait at least one night after finishing the report. Same-day is re-reading.
• Read the report only, not the code. A reviewer does not have your code loaded.
• Write objections as a numbered list, in notebook/<project>-adversarial.md, in the second person. "You claim X; the data shows Y."
• Then answer each one, in writing, in the first person.
• Objections you cannot answer become report limitations. That is the output.

The eleven questions

Work through all of them. The ones that feel inapplicable are often the productive ones.

1. Which claim in this report has the weakest evidence?
2. What baseline did you choose, and what would a hostile reviewer say you should have chosen instead?
3. Which number would change most if the workload were realistic rather than synthetic?
4. What did you measure because it was easy, rather than because it mattered?
5. Where does the report say "we observe X" when it means "X happened once"?
6. Which experiment has no negative control, and what would one have shown?
7. What is the sample size, and can it resolve the effect being claimed?
8. Which conclusion depends on an assumption stated nowhere in the report?
9. If this result is wrong, what is the most likely reason?
10. What would someone who built this professionally immediately notice is missing?
11. Would you believe this report if someone else wrote it?

Question 11 is the whole exercise compressed. Ask it out loud.

Using an assistant here is explicitly encouraged

This is the one place where an AI is unambiguously on the right side of the policy: asking for a solution substitutes for your thinking, asking for an attack forces more of it.

Prompts that work:

• "Here is my report. Argue that the conclusion does not follow from the data."
• "What is the strongest objection a reviewer would raise about the baseline?"
• "Which sentences here assert something the data does not show?"
• "What did I fail to measure that would change the conclusion?"

An assistant will produce a more honest attack on your work than you will, because it has nothing invested in the answer. It cannot fix failure (1) — it does not know your field's standards better than the literature does — but it is genuinely good at (2).

2. Publishing Early

Already scheduled in portfolio.md: eight posts across the journey, starting at week 20. That timeline exists for portfolio reasons. It is also, and more importantly, the cheapest source of external correction available, and it should be treated as load-bearing rather than incidental.

What makes a post produce feedback rather than silence:

The first post is week 20 and it is deliberately small — the bytecode-vs-tree-walk result. Counterintuitive, fully reproducible in one file, and the kind of thing where someone who knows more will tell you so. That is the point.

Read the corrections as data. When someone says "your benchmark is measuring dispatch, not the VM", that is a scorecard adjustment you could not have made yourself. Record it in the project's notebook/<project>-adversarial.md.

3. The Reproduction Exchange

The only mechanism that fixes failure (3), and the one most likely to be skipped.

At each stage boundary, get one other person to run one of your repositories from a clean clone and report what broke. In exchange, do the same for them.

The protocol

1. Pick the project with the clearest headline number.
2. Give them only the repository URL and the headline claim: "make bench should print recall@10 ≥ 0.95 at efSearch=128, in under 10 minutes."
3. Answer no questions during the attempt. Every question they need to ask is a defect in your README, and you want the list.
4. They report: did it run, how long did setup take, what did they have to work out, what number did they get.
5. You fix the README. Then someone else tries.

Why this is worth the coordination

The failures it finds are invariably the same shape and invariably invisible to you: an undocumented dependency, a hard-coded path, a Python version assumption, a step you do without thinking. Every one is a genuine reproducibility defect and none is findable alone.

It also converts P15's hardest exit criterion from a surprise into a rehearsal. By month 34 you will have done this four times and the final reproduction will work.

Where to find the other person

In descending order of likelihood: a colleague on your team (offer to reciprocate on their side project); a friend who codes; someone who commented on one of your posts; a local meetup. Failing all of those: a fresh VM or container with nothing installed, following your own README literally and refusing yourself any recalled knowledge. Weaker, because you cannot un-know your own setup — but far better than nothing, and it still catches the missing dependency.

4. A Reviewer

The highest-value mechanism, and the only one that fixes failure (1).

One person, one hour per quarter, who reads one report and tells you what is wrong with it. Eleven or twelve conversations across the journey.

What to ask for, concretely

Vague requests get vague answers. Ask for exactly this:

"I've written up a benchmark of X. Could you spend 45 minutes on the report and tell me: (a) which claim you would not believe, (b) what baseline you would have expected, and (c) what a person who does this professionally would notice is missing? I'm not looking for encouragement — I'm looking for the objection I can't see."

That framing does three things: it bounds the time, it asks three specific questions, and it explicitly licenses criticism, which most people withhold by default.

Who

The realistic answer for most people is the first row, and one hour a quarter is a small ask that most senior engineers enjoy being asked for.

If you genuinely cannot find one

Say so in the honest-status section of your portfolio, and lean harder on mechanisms 2 and 3. Publishing with an explicit question is a way of asking the internet to review you, and it works often enough to matter. But do not pretend this gap is closed — an unreviewed 34-month body of work has a knowable weakness and naming it is more credible than not.

5. The Public Artifact

Optional. Two or three times, at most.

A conference talk, a workshop paper, an open-source tool people actually use. High cost, high signal — the feedback is unsolicited, from strangers, with no social reason to be kind.

The tools most likely to attract real users are named in portfolio.md: the fault injector and the linearizability checker. A GitHub issue saying "this deadlocks under condition X" is the purest feedback in this document.

Do not schedule this. It should emerge from work that turned out well, not from a calendar entry — a talk given because a deadline arrived is a talk with nothing in it.

The Minimum Viable Loop

If you do nothing else on this page, do these. They cost ~1 hour per project plus four half-days across 34 months, and they close two of the three failures.

• Adversarial self-review before every scorecard, eleven questions, written down
• Publish at week 20, then follow the schedule in portfolio.md
• One reproduction exchange per stage boundary — five total, any willing person
• Record every external correction in the project's adversarial note, and let it move a scorecard score

That last item is what makes the loop real. A correction that does not change a score did not close a loop.

How To Receive a Review

The mechanism only pays if the response to criticism is useful, and the default human response is not.

The single most useful follow-up question: "What would you have expected to see instead?" It converts a vague discomfort into a specific missing experiment, and it is the question that most often produces the next week's work.

And on defensiveness: notice it, name it to yourself, and continue. The reflex to defend is strongest exactly where the criticism is most accurate, which makes it a useful signal rather than a failing. The cost of acting on it is a wrong belief you keep for thirty months.

What Feedback Cannot Fix

Honest boundaries, so you do not over-invest.

• Whether the journey is worth doing. That is a question about your goals, and outsiders will answer it with their goals.
• Motivation. Praise is pleasant and does not survive week 61. The mechanisms in sustainability.md do.
• Whether your implementation is correct. That is what tests and fault injection are for. A reviewer reading a report cannot find your race condition.
• Taste, quickly. Design judgement comes from building fifteen systems and reading their literature. A reviewer accelerates it; nothing substitutes for it.

Feedback fixes blind spots, drifting standards, and reproducibility. Those three, and they are exactly the three that self-assessment cannot reach.

References

• Kruger, J., Dunning, D. Unskilled and Unaware of It. Journal of Personality and Social Psychology 77(6), 1999. Self-assessment is least reliable precisely where ability is lowest — the reason this page exists.
• Ericsson, K. A. et al. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review 100(3), 1993. Immediate, informative feedback is one of the defining conditions of deliberate practice; without it, repetition does not produce improvement.
• Collberg, C., Proebsting, T. A. Repeatability in Computer Systems Research. CACM 59(3), 2016. The study that found a large fraction of published systems results could not be rebuilt — evidence that authors systematically overestimate their own reproducibility.
• Kahneman, D. Thinking, Fast and Slow. FSG, 2011. On the difficulty of seeing one's own reasoning errors from the inside.
• Blackburn, S. M. et al. The Truth, The Whole Truth, and Nothing But the Truth. ACM TOPLAS 38(4), 2016. What a rigorous evaluation looks like — a useful external standard when no human reviewer is available.
• Peyton Jones, S. How to Write a Great Research Paper. Microsoft Research, 2004. On seeking criticism early and treating it as the most valuable input a project receives.
• Edmondson, A. Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly 44(2), 1999. Why the explicit "I'm not looking for encouragement" framing materially changes the quality of the review you get.