D2 — System Design: Fault-Tolerant Distributed Job Scheduler

45 minutes. Hard stop. Produce a written design document and a diagram. Both are graded.

This is the reported technical-screen design question, verbatim in shape (../research/source-report.md row 8).

Table of Contents

The Prompt, As An Interviewer Would Say It

"Let's design a distributed job scheduler. Users submit jobs — some run once at a specific time, some run on a recurring schedule, like a cron. The system runs them on a fleet of workers.

The important part is that it has to be fault-tolerant. Workers die. The scheduler itself can die. The network partitions. Jobs still need to run, and we care a lot about not silently dropping one.

Take it wherever you think is interesting. I'll interrupt with questions."

The deliberate under-specification is the test. "Take it wherever you think is interesting" means you choose which components are load-bearing. Choosing badly — spending twenty minutes on the REST API surface and four on execution semantics — is the most common way this round is lost, and it is lost silently: nothing goes wrong, you just never get to the part that mattered.

Clarify out loud, before designing. At minimum:

  • Scale. How many jobs, at what submission rate, at what concurrency? (Pick numbers and say them. "Let's say 10M scheduled jobs, 50k executions/minute peak, jobs run from 100ms to 6 hours." An interviewer will correct you if it matters. Silence about scale is a red flag.)
  • Delivery semantics. At-least-once or at-most-once? (This is the single most important clarifying question in the problem. Exactly-once execution of a side-effecting job is not achievable without cooperation from the job itself. If you do not say this, you have missed the core of the question.)
  • Latency tolerance. Is a job firing 30 seconds late a bug or a shrug?
  • Ordering. Do jobs for the same user/tenant need serialization?
  • Multi-tenancy. Isolation and fairness between tenants, or a single trusted user?

What You Must Produce

Two artifacts, both inside the 45 minutes:

  1. A design document at attempt-d2.md (create it next to this file), following the template below.
  2. A diagram, drawn in Excalidraw — reported to be the actual design-round tool. Export a PNG next to your doc. Drawing under time pressure is a motor skill and it is being measured.

The Design Template

This is the template you will use for every design in this program. Learn it now; it is the thing that keeps you from rambling when the clock is running.

# Design: <name>

## 1. Requirements and scope
Functional. Non-functional. Explicitly out of scope.

## 2. Scale numbers
The numbers I assumed, and the arithmetic I did with them.
QPS, storage, network, concurrency, growth.

## 3. API surface
The three to five calls that matter. Request/response shapes.

## 4. Data model
Tables/collections, keys, indexes, and WHY those keys.

## 5. High-level architecture
Components and the flow between them. This is the diagram.

## 6. Deep dive: the two hardest components
Not the easy ones. The two where the design could actually fail.

## 7. Failure and recovery
For each failure: how it is DETECTED, how it is CONTAINED, how it RECOVERS.

## 8. Bottlenecks and evolution
What breaks first at 10x. What I would change.

## 9. Tradeoffs I explicitly rejected, and why
The alternatives I considered and turned down.

Section 9 is not optional. Reported sources name "name-dropping technologies without being able to defend the tradeoff" as the leading design-round anti-pattern. Section 9 is where you prove you did not do that — and writing it forces you to have actually considered alternatives rather than pattern-matching to the first architecture you have seen before.

Time Budget Inside the 45 Minutes

MinutesActivity
0–5Clarify. Requirements, scope, scale numbers. Write them down
5–10API surface and data model
10–20High-level architecture + the diagram
20–35Deep dive on the two hardest components
35–45Failure modes, bottlenecks, rejected tradeoffs

If you are still drawing boxes at minute 25, you have failed the round regardless of how good the boxes are. The deep dive is where senior and staff signal lives, and it needs fifteen uninterrupted minutes.

Hidden Follow-Ups

Do not read these until your 45 minutes are up. They are what a real interviewer interrupts with, and the rubric awards points for having pre-empted them in your written doc without being asked.

Open only after the timer ends

  1. "A worker picks up a job, starts running it, and then its network partitions from the scheduler for ten minutes. The job is still running. What does the scheduler do?" The lease/fencing question. If your answer re-dispatches the job, you now have two copies of a side-effecting job running concurrently. Do you have a fencing token? Does the job's write path check it?

  2. "How do you guarantee a job scheduled for 09:00:00 doesn't run twice when you have three scheduler replicas for availability?" Leader election, or partitioned ownership, or a compare-and-swap on a claim row. Each has a different failure profile. Which did you pick and why?

  3. "Your scheduler was down for two hours. It comes back. There are 40,000 jobs whose fire time has passed. What happens?" The thundering-herd / catch-up-storm question. Do you run them all? Skip them? Run only the most recent occurrence of each recurring job? This is a product decision that the design must expose as a per-job policy, not silently decide.

  4. "One tenant submits a job that takes 6 hours and pins a worker. Now their other jobs are starving everyone else's. What do you do?" Isolation, fair queuing, per-tenant concurrency caps, separate pools by expected duration.

  5. "How do you know a job actually ran?" Execution records, idempotency keys, the difference between dispatched and completed, and what your at-least-once guarantee actually promises the user.

  6. "What's your storage? Why not just Postgres?" And if you said Postgres: at what scale does SELECT ... WHERE next_run_at <= now() FOR UPDATE SKIP LOCKED stop working, and what does the next thing look like?

  7. "Clocks. Your scheduler node thinks it's 09:00 and the worker thinks it's 08:59:30." Clock skew, monotonic vs wall clock, why leases must be measured in elapsed time on a single node, not in absolute timestamps compared across nodes.

What Is Actually Being Measured

Five things, in descending order of weight:

  1. Did you identify the right hard parts? For this problem they are (a) exactly-once dispatch semantics under scheduler failure, and (b) worker liveness and lease expiry with the resulting split-brain risk. Everything else is plumbing.
  2. Do your failure sections have all three legs — detection, containment, recovery? "It retries" is not a failure analysis.
  3. Did you do arithmetic? Any number at all beats no numbers. A claim like "50k executions/minute is ~830/s, so at 200 jobs/s/worker that's 5 workers plus headroom" is worth more than a paragraph of adjectives.
  4. Did you reject something explicitly? Section 9.
  5. Did you stay at the right altitude? Sketching a class hierarchy for the job model is too low. "We'll use a queue" with no discussion of visibility timeouts is too high.

Notably absent: whether your design is the same as the interviewer's. It does not need to be.

Submission

  • Design doc: attempt-d2.md (next to this file)
  • Diagram: attempt-d2.png
  • Score against RUBRIC.md → Part 2
  • Compare against ANSWER-KEY.md → D2 after you have scored yourself unaided, so you measure your own judgement rather than your ability to recognize a good answer when shown one