Track C — Distributed Systems Design

The reported technical screen's second round was a job scheduler with fault tolerance (../../research/source-report.md row 8), reportedly in Excalidraw. The reported anti-pattern is name-dropping technologies without being able to defend the tradeoff.

This track produces written design artifacts, not reading notes. Twelve of them, each attacked in writing by a hostile staff-level interviewer, then revised.

→ Study guide: WARMUP.md — every primitive from zero: quorums, Raft, leases and fencing, consistency models, partitioning, delivery semantics, load control. → d01-job-scheduler.md — a complete worked design with a hostile critique and the revision.

Table of Contents

• The Design Template
• Concept Inventory
  • C1. Replication and Consensus
  • C2. Partitioning and Placement
  • C3. Storage and Consistency
  • C4. Messaging and Delivery
  • C5. Control Under Load
  • C6. Time
• The Failure-Mode Catalog
• Back-of-Envelope Calculators
• The Twelve Design Exercises
• The Critique Loop
• Drill Set
• Failure Modes
• Self-Assessment Rubric
• References

The Design Template

Use it every time, in this order. It is what keeps you from rambling when the clock is running, and it makes the 45-minute budget survivable.

# Design: <name>

## 1. Requirements and scope
Functional. Non-functional. Explicitly out of scope.

## 2. Scale numbers
The numbers I assumed and the arithmetic I did with them.

## 3. API surface
The three to five calls that matter, with request/response shapes.

## 4. Data model
Tables/collections, keys, indexes — and WHY those keys.

## 5. High-level architecture
Components and flow. This is the diagram.

## 6. Deep dive: the two hardest components
Not the easy ones. The two where the design could actually fail.

## 7. Failure and recovery
For each: DETECTION, CONTAINMENT, RECOVERY. All three legs.

## 8. Bottlenecks and evolution
What breaks first at 10x. What I would change.

## 9. Tradeoffs I explicitly rejected, and why

Section 9 is the one that separates candidates. It is also the one that gets skipped when time runs short, which is why the time budget puts it at minute 35, not minute 44.

Time budget inside 45 minutes: 5 clarify · 5 API and data · 10 architecture and diagram · 15 deep dive · 10 failure and tradeoffs. If you are still drawing boxes at minute 25, the round is lost regardless of how good the boxes are.

Concept Inventory

Each primitive gets a one-page design note in designs/ and appears as a required element in at least one exercise.

C1. Replication and Consensus

Fencing tokens are the single highest-value item in this list. A lease expires; the holder does not necessarily know. Without a monotonically increasing token that the storage layer checks, a partitioned worker's late write silently corrupts state after its replacement has already run. Naming it unprompted is a reliable staff-level signal, and almost nobody does.

C2. Partitioning and Placement

C3. Storage and Consistency

C4. Messaging and Delivery

C5. Control Under Load

C6. Time

The Failure-Mode Catalog

Every design you produce must include a failure section, and every failure needs all three legs. "It retries" is not a failure analysis.

Fail-slow deserves its own emphasis. A crashed node is easy: it stops answering. A node that answers health checks in 2ms while serving real requests in 40s is invisible to liveness probes and poisons every load balancer pointing at it. If your design's only health signal is "does it respond," you have not handled the common case.

Back-of-Envelope Calculators

cd tracks/systems-design/calculators
python3 envelope.py --help
python3 envelope.py qps --rps 50000 --ms 8          # concurrency and cores
python3 envelope.py queue --rho 0.8                  # the utilization knee
python3 envelope.py storage --rows 1e10 --bytes 512  # bytes, replicated
python3 envelope.py retry --rps 10000 --fail 0.3     # retry amplification

The point is not the tool. The point is that you do the arithmetic out loud in the round. "50k rps at 8ms means 400 concurrent in flight, so at 200 per box that's 2 boxes plus headroom — call it 4 for failure tolerance" is worth more than a paragraph of adjectives. Any arithmetic beats none.

Numbers to have memorized cold — verify them yourself with envelope.py latencies:

The Twelve Design Exercises

Written into designs/ as artifacts, in this order.

d01 and d02 are mandatory and come first. The rest are ordered by leverage, not by difficulty.

The Critique Loop

The mechanism that makes this track work. Every design goes through it.

1. You write the design against the template, to a 45-minute clock.
2. I attack it in writing, playing a hostile staff-level interviewer. Not "have you considered" — specific, adversarial, with a concrete failure scenario:

   "Your scheduler claims at-least-once. Worker A claims job J with a 30-second lease, then GC-pauses for 45 seconds. You re-dispatch to worker B. B completes and writes the result. A wakes up, finishes, and writes its result too. Walk me through what the user sees, and tell me which line of your design prevents it. If none does, say so."

3. You revise, in place, with a changelog at the bottom.
4. I re-attack the revision, harder.
5. It is done when I cannot find a failure you have not named — including the ones you name and choose to accept.

Deliberately accepting a failure mode with a stated reason is a staff behavior. Claiming to have handled everything is a junior one, and it is trivially falsified by one good question.

Drill Set

Failure Modes

Self-Assessment Rubric

Scoring detail in ../../diagnostics/RUBRIC.md. The hard cap applies: if you deep-dived the wrong components, the round caps at L1 no matter how good the rest was.

References

• Kleppmann, M. Designing Data-Intensive Applications, 2nd ed. — Ch. 5 (replication), 6 (partitioning), 7 (transactions, write skew), 8 (unreliable clocks, fencing), 9 (consistency and consensus)
• Ongaro, D. and Ousterhout, J. In Search of an Understandable Consensus Algorithm (Raft). USENIX ATC 2014. https://raft.github.io/raft.pdf
• Lamport, L. Paxos Made Simple. 2001.
• Burrows, M. The Chubby Lock Service for Loosely-Coupled Distributed Systems. OSDI 2006.
• Corbett et al. Spanner: Google's Globally-Distributed Database. OSDI 2012.
• Kingsbury, K. Jepsen analyses. https://jepsen.io/analyses — the best available catalog of how real systems break
• Beyer et al. Site Reliability Engineering. O'Reilly, 2016 — Ch. 21 (handling overload), Ch. 22 (cascading failures)
• Brooker, M. Exponential Backoff and Jitter. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Brooker, M. Timeouts, retries, and backoff with jitter. Amazon Builders' Library.
• Kulkarni et al. Logical Physical Clocks (HLC). OPODIS 2014.
• Shapiro et al. Conflict-free Replicated Data Types. SSS 2011.