Diagnostic Rubric and Level Mapping
Score down when unsure. A generous rubric is the one thing that guarantees you fail the real loop — it costs nothing to be told you are L1 in week one and discover in week six that you are L2; the reverse costs the offer.
Every band is calibrated to Staff altitude, because AI-lab levelling is compressed and a "Senior" title reportedly carries Staff-equivalent scope (
../research/findings.md).
Table of Contents
- The Four Levels
- Part 1: Coding
- Part 2: System Design
- Part 3: Python Internals
- Part 4: Behavioral
- Composite Level Map
- How Levels Become the Plan
- The Hire-Bar Translation
The Four Levels
| Level | Meaning | What the plan does with it |
|---|---|---|
| L0 | Foundations missing — cannot reliably produce a correct solution in the format | Rebuild-from-primitives phase before any timed work. Concepts first |
| L1 | Correct but slow — you get there, the clock beats you | Volume and time-pressure drills. No new concepts |
| L2 | Interview-passable at senior — would probably clear a senior bar today | Maintenance plus depth on the two weakest sub-areas |
| L3 | Staff-altitude — clears the bar and adds something | Spaced-repetition maintenance only. Hours reallocate to weaker tracks |
Part 1: Coding
Primary metric: gates passed
| Gates | Level | Reading |
|---|---|---|
| 0–1 | L0 | The format itself is the obstacle, not the problem |
| 2 | L1 | Reported pass bar per some sources; assume the stricter one and treat this as not yet clearing |
| 3 | L2 | Clears the reported bar with margin |
| 4 | L3 | Rare per reported sources |
Secondary metric: time-to-first-passing-gate
This is the metric that predicts the real round, because an unopened gate scores zero regardless of how good the unwritten design was.
| Time to G1 | Reading |
|---|---|
| ≤ 8 min | Strong. The extensibility question is where your remaining risk lives |
| 9–15 min | Normal. Trainable to ≤8 with volume |
| 16–25 min | You are designing too long before writing. This is the highest-leverage single fix in the program |
| > 25 min | L0 on format regardless of gate count |
Modifier: the rewrite penalty
If you rewrote from scratch at any gate rather than extending, subtract one level. The
gated format punishes rewrites brutally, and the underlying cause is a representation chosen
for the stated requirement rather than for the invariant. See
ANSWER-KEY.md.
Narration sub-score (0–5)
| Score | Level contribution |
|---|---|
| 0–1 | Cap the whole Part-1 level at L1 regardless of gates |
| 2–3 | No modifier |
| 4–5 | +0.5 toward the next level |
Going silent while stuck is the costliest single narration failure: the interviewer cannot give a hint they do not know you need, and silence reads as lost even when you are thinking productively.
Part 2: System Design
25 points, five sections of 5.
2A. Requirements, scope, and scale numbers (5)
| Pts | Standard |
|---|---|
| 0 | No clarification, no numbers, started drawing immediately |
| 2 | Asked about scope; no arithmetic |
| 3 | Stated scale assumptions but did not use them |
| 4 | Numbers stated and used to size at least one component |
| 5 | Numbers used, and you asked the delivery-semantics question (at-least-once vs at-most-once) unprompted |
The delivery-semantics question is worth its own point because it is the fulcrum of this entire problem and most candidates never ask it.
2B. Architecture and API (5)
| Pts | Standard |
|---|---|
| 0–1 | Boxes with no data flow, or an API with no request/response shapes |
| 2–3 | Coherent architecture; the data model's key choices are unexplained |
| 4 | Coherent, with keys and indexes justified |
| 5 | Above, plus you named the component that is the throughput ceiling before being asked |
2C. Deep dive on the two hardest components (5)
The highest-weight section. Selection matters more than depth.
| Pts | Standard |
|---|---|
| 0 | No deep dive — stayed at box level for 45 minutes |
| 1–2 | Deep dive on the easy components (API tier, storage schema) |
| 3 | Identified one of the two hard components (dispatch semantics under scheduler failure; worker liveness/leases/split brain) |
| 4 | Identified both |
| 5 | Both, and named fencing tokens or an equivalent mechanism unprompted |
2D. Failure and recovery (5)
Every failure needs three legs. "It retries" is not a failure analysis.
| Pts | Standard |
|---|---|
| 0–1 | No failure section |
| 2 | Failures listed, no detection mechanism |
| 3 | Detection and recovery; no containment (blast radius, backpressure, shedding) |
| 4 | All three legs for the main failures |
| 5 | Above, plus you stated a failure mode you are choosing to accept and why |
Deliberately accepting a failure mode with a stated reason is a staff behavior. Claiming to have handled everything is a junior one.
2E. Rejected tradeoffs (5)
| Pts | Standard |
|---|---|
| 0 | Section absent |
| 2 | Alternatives named, no reasons |
| 3 | Reasons given but qualitative ("wouldn't scale") |
| 4 | At least two alternatives rejected for specific, quantified reasons |
| 5 | Above, plus one rejection where the alternative was genuinely close and you said what would flip your choice |
Part 2 level map
| Total | Level |
|---|---|
| 0–8 | L0 |
| 9–14 | L1 |
| 15–20 | L2 |
| 21–25 | L3 |
Hard cap: if 2C scored ≤2, the part caps at L1 regardless of total. Identifying the load-bearing components is the signal; a polished design of the wrong parts is a fail.
Part 3: Python Internals
| Correct | Level |
|---|---|
| 0–7 | L0 |
| 8–12 | L1 |
| 13–16 | L2 |
| 17–20 | L3 |
The confident-wrong modifier
| Confident-wrong | Modifier |
|---|---|
| 0–1 | none |
| 2–3 | −0.5 level |
| 4+ | −1 full level, and every one goes into ../review/ at the 1-day interval |
A gap you know about is a study item. A gap you are confident about is a landmine — it is what you will assert in an interview and be corrected on, and the correction costs far more than the admission would have.
Section weighting
Questions 1–8 (iterators/generators) and 9–15 (async/concurrency) are double-weighted for
planning purposes — not for the score, but for where hours go. Those are the two areas
reported to actually surface in the loop
(../research/source-report.md rows 25–27). Six correct out
of eight on iterators with weak memory answers is a very different plan from the reverse.
Part 4: Behavioral
Each prompt scored 0–5.
Prompts 1 and 2
| Pts | Standard |
|---|---|
| 0 | No decision — a project tour |
| 1 | A decision, no alternatives |
| 2 | Alternatives named, qualitative reasons only |
| 3 | Alternatives with specific reasons; no disagreement in the story |
| 4 | Above, plus a named opponent whose position is stated fairly, plus a measured outcome |
| 5 | Above, plus a specific self-critique with a generalizable lesson, plus the alignment mechanism was evidence (a prototype, a measurement) rather than authority or persistence |
Scope cap: if the decision affected only your own team, cap at 3. That is a Senior story. Row 34 requires cross-team.
Prompt 3 (where is AI headed)
| Pts | Standard |
|---|---|
| 0–1 | Platitudes |
| 2 | Accurate trend list, no position |
| 3 | A specific claim, no evidence |
| 4 | Specific claim with evidence you can cite or measured |
| 5 | Above, plus an explicit falsifier, plus a connection to what you would build |
Part 4 level map
| Total (of 15) | Level |
|---|---|
| 0–4 | L0 |
| 5–8 | L1 |
| 9–12 | L2 |
| 13–15 | L3 |
Composite Level Map
Do not average. Report per-track levels — the plan is built from the profile, not from a single number.
| Track | Driven by |
|---|---|
| A — Coding under time pressure | Part 1 gates + time-to-first-gate + rewrite penalty |
| B — Python internals | Part 3, iterator/async sections weighted |
| C — Distributed systems design | Part 2 |
| D — ML/inference infra | Not measured by this battery. Assume L0/L1 and confirm with the Track D entry quiz in week 2 |
| E — Take-home / deep dive | Not measurable in 3 hours. First real measurement is the week-8 48-hour run |
| F — Behavioral | Part 4 |
| G — Agentic coding | Not measured. First measurement is the week-6 timed agent run |
Three of seven tracks are unmeasured by design — a 3-hour battery cannot measure a 48-hour take-home. That is honest, and it means the plan's first month is partly provisional and gets corrected at the week-4 re-test.
How Levels Become the Plan
With 22 hours/week over 26 weeks (~570 hours), the baseline allocation is:
| Track | Baseline share | If L0 | If L1 | If L2 | If L3 |
|---|---|---|---|---|---|
| A — Coding | 25% | 35% | 30% | 20% | 10% |
| B — Python internals | 12% | 20% | 15% | 10% | 5% |
| C — Systems design | 15% | 22% | 18% | 12% | 6% |
| D — ML/inference infra | 20% | 28% | 22% | 15% | 8% |
| E — Take-home | 12% | fixed | fixed | fixed | fixed |
| F — Behavioral | 10% | 16% | 12% | 8% | 5% |
| G — Agentic | 6% | 10% | 8% | 5% | 3% |
Track E is fixed because it is two 48-hour blocks plus two deep-dive drills — a schedule, not a dial. Percentages are renormalized to 100% after applying the per-track adjustments.
Rebalancing rule: at each monthly re-test, any track that reaches L3 drops to its L3 share and the freed hours go to the lowest-level track. This is the mechanism that stops a six-month plan from becoming a six-month ritual.
The Hire-Bar Translation
Every weekly mock gets scored on this scale, not on the L0–L3 scale. Learn what it means now.
| Verdict | Coding | Design | Behavioral |
|---|---|---|---|
| No hire | Did not reach a working solution, or needed substantial hints | Wrong components deep-dived; no failure analysis | A tour, no decisions, no disagreement |
| Hire (senior) | Working solution, some prompting, reasonable complexity | Coherent design, right hard parts identified, thin failure analysis | Real decisions with tradeoffs; single-team scope |
| Strong hire (senior) | Working, unprompted, clean, tested the tricky invariant | Right hard parts, three-legged failure analysis, explicit rejections | Cross-team decision, named opponent, measured outcome |
| Hire (staff) | Above, plus anticipated the follow-up stage in the initial design | Above, plus named a failure mode being deliberately accepted | Above, plus changed an organization's mind with evidence |
| Strong hire (staff) | Above, plus taught the interviewer something | Above, plus reframed the problem in a way the interviewer adopted | Above, plus a decision that was expensive and right, and one that was expensive and wrong |
I will tell you plainly which one you hit. Not the one you nearly hit.