Track E — Warmup: The 48 Hours and the Interrogation

Self-contained. The hour-by-hour playbook, the webhook system designed and specified, the decision log worked, and — the part nobody prepares — the deep-dive interrogation with 40 questions and full model answers generated from a real diff.

Reported: a 48-hour window to "build something real" — the example being a distributed webhook delivery system with retries and dead-letter queues — followed by an interviewer walking your code line by line, from a question list he wrote after reading it.

Table of Contents

• Chapter 0: The Reframe That Changes Everything
• Chapter 1: The 48-Hour Playbook
  • 1.1 The hour-by-hour allocation
  • 1.2 The non-negotiables
  • 1.3 What "beyond the ask" actually means
  • 1.4 What to cut, in order
• Chapter 2: The Decision Log
• Chapter 3: The Webhook System, Specified
  • 3.1 The brief and its deliberate ambiguities
  • 3.2 The design
  • 3.3 The seven decisions, logged
  • 3.4 The benchmark
• Chapter 4: The Interrogation, Worked
  • 4.1 How the question list is built
  • 4.2 Choice questions
  • 4.3 Magic-number questions
  • 4.4 Scale questions
  • 4.5 Data-loss questions
  • 4.6 Omission questions
  • 4.7 Hostile questions
  • 4.8 The closing move
• Chapter 5: The Second Project
• The Rubric
• References

Chapter 0: The Reframe That Changes Everything

The take-home and the deep dive are one round, not two.

The take-home's real function is to generate a personalized interrogation surface. The interviewer reads your code and writes questions from it — reportedly a list covering every choice and every decision. So:

Every decision you make in the 48 hours is a question you will be asked in week three.

Which inverts the optimization target. It is not "best code." It is:

Code every line of which I can defend, plus a written record of the alternatives I rejected.

Three consequences that should change what you build:

1. A simpler system you can defend completely beats a sophisticated one with three choices you made on autopilot at hour 31. The sophisticated one loses the moment the interviewer asks "why 30 seconds?" and you say "it seemed reasonable."

2. Reasoning is a deliverable. A decisions.md is not documentation overhead; it is the artifact that makes the deep dive winnable. Ninety seconds per entry at hour 12 buys a complete answer at week 3, versus a reconstruction the interviewer will correctly hear as one.

3. Deliberately unbuilt things are answerable. "I didn't implement per-destination ordering, here's why, and here's what it would cost" is a strong answer. "I didn't get to it" is not. The difference is whether you decided or ran out of time.

This is inference I1 in ../../research/findings.md — labelled as inference, not sourced. But it follows directly from the reported fact that the question list is written after reading your code.

Chapter 1: The 48-Hour Playbook

1.1 The hour-by-hour allocation

The 48 hours include sleep. Treat it as ~26 working hours, not 48.

The walking-skeleton milestone at hour 8 is the one that matters. It is the same "time-to-first-correct" principle as the gated coding round: get something end-to-end working early, then improve it. A beautiful half-system at hour 40 scores worse than a complete simple one at hour 20 that you then spent 20 hours hardening.

1.2 The non-negotiables

Reported grading criteria converge tightly: code quality, test coverage, a written design doc explaining tradeoffs, and how you handled the under-specified parts. One source states the principle directly — a working solution with a thoughtful README beats a clever solution with no docs.

So these ship regardless of what gets cut:

On commit history: no wip, no fix, no asdf. Each commit is one coherent change with a message saying why. It is graded, it costs nothing, and a history that reads feat: walking skeleton → feat: retry with full jitter → test: failure injection for DLQ → docs: design doc + decision log tells a reviewer more about how you work than the code does.

1.3 What "beyond the ask" actually means

A narrow definition, deliberately. Not more features — extra features read as poor judgement, not enthusiasm, because they signal you optimized for surface area over depth.

Pick one of:

(a) A measured benchmark with an honest methodology. Including the number that disappointed you. "Throughput plateaus at 4,200/s because the DB connection pool saturates; here's the flame graph" is worth more than any feature.

(b) A failure-injection test that proves a recovery path actually works. Not "I handle crashes" — a test that kills the process mid-write and asserts recovery. This is rare and it is memorable.

(c) An operational concern nobody asked for. Structured logs with a correlation ID, a /health endpoint that checks the real dependency, a runbook for draining the DLQ. It signals you have operated software, not only written it.

1.4 What to cut, in order

When hour 38 arrives and it is not all going to fit, cut in this order — and write down what you cut and why, because a documented cut is a decision and an undocumented one is a gap:

1. Extra features. Always first.
2. Breadth of configuration. Hard-code a sensible value, document it as configurable-later.
3. Performance optimization that is not the benchmark.
4. Admin/UI surface. A CLI is fine.
5. Persistence sophistication — SQLite over Postgres is a defensible choice, stated.

Never cut: tests, the README, the design doc, error handling on external boundaries, or the decision log. Those are the graded artifacts.

Chapter 2: The Decision Log

The highest-leverage file in the whole track, and it costs ~90 seconds per entry.

## D-007 — Retry backoff: full jitter
**Decision:** exponential backoff with FULL jitter — uniform over
[0, min(cap, base * 2^attempt)] — base 200 ms, cap 30 s, 6 attempts.

**Alternatives considered:**
- *No jitter.* Rejected: every consumer that failed at the same instant retries
  at the same instant. A destination that dropped 1,000 deliveries gets all
  1,000 back simultaneously and stays synchronized. This is the actual failure
  mode AWS documented.
- *Equal jitter* (half fixed + half random). Rejected: keeps a latency floor we
  don't need, and AWS's published simulation found full jitter minimized both
  total work and completion time under contention.
- *Decorrelated jitter.* Rejected: smoothest, but its worst case is harder to
  bound and I wanted a number I could state.

**Assumes:** failures across destinations are correlated (a shared outage), which
is what makes desynchronization valuable.

**Would revisit if:** we needed a minimum retry latency for rate-limit
compliance, which would push me to equal jitter.

**Not tested:** behaviour when the system clock jumps backwards mid-backoff.

Five fields, and the last two are what make it interview-grade:

• "Would revisit if" proves you know the decision is contingent, not dogma.
• "Not tested" pre-empts the omission question — and volunteering a gap before it is found is the single most credibility-generating move in the round.

Write one for every constant. Every timeout, every batch size, every retry count. Those are exactly the "why 30 seconds?" questions, and they are the ones candidates fumble.

Chapter 3: The Webhook System, Specified

Build this in ../../projects/webhook-delivery/ under a real 48-hour clock.

3.1 The brief and its deliberate ambiguities

Build a service that delivers webhooks to customer endpoints. Customers register a URL and subscribe to event types. When an event occurs, we deliver it. Customer endpoints are unreliable — they time out, return 500s, and occasionally go away entirely. We must not lose events, and we must not hammer a struggling endpoint into the ground.

Build something real. We care about how you handle the parts we did not specify.

The under-specification is the test. Nine ambiguities; each needs a decision, a reason, and a README line:

3.2 The design

  POST /events ──▶ ┌─────────────┐
                   │  API tier   │ validate, authn, size cap
                   └──────┬──────┘
                          │  ONE transaction:
                          │    INSERT event
                          │    INSERT delivery per matching subscription
                          ▼
              ┌───────────────────────────┐
              │  Store (system of record) │  events · subscriptions
              │  deliveries               │  deliveries: the outbox
              └────────┬──────────────────┘
                       │ claim due deliveries
                       │ FOR UPDATE SKIP LOCKED
                       ▼
        ┌──────────────────────────────────┐
        │  Delivery workers                │  per-destination concurrency cap
        │  • lease + attempt counter       │  circuit breaker per destination
        │  • HMAC sign                     │  backoff with full jitter
        │  • POST with timeout             │
        └────────┬──────────────┬──────────┘
                 │ success      │ exhausted
                 ▼              ▼
          delivered        dead_letter ──▶ replay API

The single most important structural decision: the delivery rows are created in the same transaction as the event. That is the outbox pattern, and it solves the dual-write problem — an event exists if and only if its deliveries do. Writing the event and then publishing to a queue is two writes to two systems, and a crash between them loses or invents work.

The two hard parts (the deep dive will go here):

1. At-least-once delivery with a bounded blast radius. Leases, attempt counters, and the fact that a customer endpoint may have received and processed a delivery whose response you never saw.
2. Per-destination isolation. One dead endpoint must not consume your worker pool, and must not slow deliveries to healthy endpoints.

3.3 The seven decisions, logged

Compressed; each is a full decisions.md entry in the real build.

And the deliberate non-goals, written down:

• No fan-out beyond ~1,000 subscriptions per event (would need batched delivery rows).
• No multi-region.
• No customer-facing UI; the status API is JSON.
• Payload cap 256 KB; larger goes to blob storage with a reference. (Not implemented — stated as a limit and enforced with a 413.)

3.4 The benchmark

One benchmark, honestly reported. The shape:

## Benchmark

**Setup:** single process, 8 workers, SQLite WAL mode, 200 destinations served by a
local mock returning 200 OK after a 20 ms delay. MacBook Pro M-series, Python 3.13.
Load generator submits events as fast as they are accepted. Measured over 60 s
after a 10 s warm-up. Methodology and script: `bench/run.py`.

**Result:** 4,180 deliveries/sec sustained, p50 41 ms, p99 210 ms end-to-end.

**Where it plateaus and why:** throughput is flat from 8 workers to 16. Profiling
shows 62% of wall time in the claim query — `SELECT ... FOR UPDATE SKIP LOCKED`
against a single deliveries table. The index on (next_attempt_at, state) is being
scanned and the write amplification from updating next_attempt_at on every attempt
churns it.

**What I'd do about it with more time:** partition deliveries by destination hash so
each worker claims from its own partition, removing the contention. I'd expect that
to scale roughly linearly to the connection-pool limit. I did not do it because it
complicates the fairness story between destinations and I judged the honest
measurement more valuable than an untested optimization.

**Number I'm least confident in:** the p99. The mock destination has no variance;
real endpoints have long tails and I'd expect p99 to be dominated by them, not by us.

Why this is worth more than a feature: it has a methodology, a real number, a bottleneck identified by measurement rather than guess, a stated next step not taken with a reason, and an explicit statement of which number is least trustworthy. That last line is the one interviewers remember.

Chapter 4: The Interrogation, Worked

Reportedly the interviewer walks the take-home line by line, from a question list written after reading it. Here is that list, for the system above, with model answers.

4.1 How the question list is built

Seven classes, and a good interviewer draws from all of them:

4.2 Choice questions

Q: Why did you put the delivery rows in the same transaction as the event? Because writing the event and then publishing to a queue is two writes to two systems that fail independently. Crash between them and you either have an event nobody will deliver, or a delivery for an event that doesn't exist. There's no ordering of two independent writes that's safe — that's the dual-write problem. Putting the delivery rows in the same transaction makes it one write: the event exists if and only if its deliveries do. That's the outbox pattern, and the cost is that my worker has to poll a table rather than consume from a queue, which is the bottleneck my benchmark found.

Q: Why SQLite and not Postgres? Because the brief is a 48-hour take-home and I wanted the reviewer to be able to run it with one command and no infrastructure. The design doesn't depend on it — the claim query uses standard SQL, and the only SQLite-specific thing is WAL mode, which I set explicitly and documented. If this were real I'd use Postgres, mainly for FOR UPDATE SKIP LOCKED with real concurrency; SQLite serializes writers, which is why my throughput plateaus where it does. I'd rather be honest that the storage choice was for reviewability than pretend it was for correctness.

Q: Why a per-destination concurrency cap instead of one shared worker pool? Head-of-line blocking. A shared pool means a dead endpoint's timeouts occupy workers — if I have 8 workers and a 10-second timeout, 8 deliveries to one dead endpoint stall every other destination for 10 seconds. Capping at 4 per destination bounds the damage: one dead endpoint can consume at most 4 workers. The circuit breaker then stops even those from being wasted. The cost is that a legitimately high-volume destination is capped too, which is why the cap is per-subscription configurable rather than global.

Q: Why HMAC and not mutual TLS? mTLS is stronger and it's operationally much harder for the customer — they need to manage a client certificate. Webhooks go to arbitrary customer endpoints, many of them behind platforms-as-a-service where installing a cert isn't possible. HMAC over a shared secret is what Stripe and GitHub do, and the reason is adoptability rather than security purism. I sign timestamp.body, not just the body, so a captured request can't be replayed indefinitely, and I use hmac.compare_digest for the comparison so it's constant-time.

Q: Why is ordering opt-in rather than the default? Because per-destination ordering forces concurrency 1 for that destination, which caps throughput at 1/latency — with a 50 ms endpoint that's 20/sec, no matter how much capacity I have. Most webhook consumers are idempotent and don't care about order; the ones that do care a lot. Making it opt-in means the common case is fast and the rare case is correct. If I'd made it the default I'd have made everyone pay for a guarantee most don't use.

4.3 Magic-number questions

Q: Why 200 milliseconds as the base backoff? It's roughly one round trip plus a margin, so the first retry is fast enough that a transient blip is invisible to the customer. Shorter and I'm retrying before a genuinely transient problem has cleared; longer and a one-off 500 turns into a visible delay. I'll be honest that it's a judgement call rather than a measurement — if I had production data I'd set it from the observed distribution of transient-failure durations.

Q: Why 6 attempts? Because with base 200 ms, doubling, capped at 30 s, six attempts spans about an hour of retrying. That's the number I actually chose — I picked the duration I wanted, which is "long enough to ride out a deploy or a short outage, short enough that the customer isn't getting hour-old events," and derived the attempt count from it. Attempt count alone is a meaningless knob; the total retry window is the thing with a product meaning.

Q: Why cap the backoff at 30 seconds? Beyond about 30 s the retry interval stops being useful for transient failures and starts being a queue-drain problem — if the endpoint is down for 10 minutes, whether I retry at 30 s or 5 min intervals barely changes when it succeeds, but the longer interval makes recovery lumpier. The cap also bounds how long a delivery holds a lease.

Q: Why 50% failures over 20 attempts for the circuit breaker? The rate matters more than the threshold, and the minimum volume is the important half. An absolute count breaks on low-traffic destinations — 3 failures out of 5 requests is noise, not a signal. Requiring at least 20 attempts in the window means I don't trip on a destination that gets one delivery an hour. 50% is high enough to not trip on a flaky-but-working endpoint and low enough to catch a genuinely broken one.

Q: Why a 256 KB payload cap? It's an abuse and memory bound rather than a product decision — at 8 workers, an unbounded payload means unbounded memory. 256 KB × 8 in flight is 2 MB, which is fine. The real answer is that payloads above that should be a reference to blob storage, which I documented as a non-goal rather than building.

4.4 Scale questions

Q: What happens at 100× the event rate? The claim query breaks first, and I know that because the benchmark found it — 62% of wall time is already there at 4,000/s. At 400,000/s a single deliveries table with an index on (next_attempt_at, state) is a write hotspot, and updating next_attempt_at on every attempt churns the index badly. The fix is partitioning by destination hash so each worker claims from its own partition. Second thing to break is the DLQ table, which grows without bound and needs partitioning by month with old partitions dropped rather than deleted.

Q: One customer has 100,000 subscriptions to the same event type. What happens? My design inserts one delivery row per subscription in the event's transaction, so that single transaction writes 100,001 rows and holds locks for a long time. That's a real flaw. I'd fan out lazily instead: write the event plus a fan-out job, and have a separate worker create delivery rows in batches. I noted the ~1,000-subscription limit as a stated non-goal rather than pretending it scales.

Q: A destination is down for six hours. What does that do to the rest of the system? The circuit breaker opens after the first 20 attempts, so it stops consuming workers within about a minute. Its deliveries keep accumulating with next_attempt_at in the future, so they sit in the table without being claimed. After six hours the retry window has expired and they're dead-lettered. The damage is bounded to that destination's rows plus 4 workers for the first minute. What I didn't handle is the DLQ flood when it recovers — 6 hours of dead-lettered deliveries all become replayable at once, and my replay endpoint has no rate limit. That's the first thing I'd add.

4.5 Data-loss questions

Q: Where exactly can this lose a delivery? Three places, and I'll take them in order of likelihood.

One: it can't lose it between the API and the store, because the event and its deliveries are one transaction — either both or neither.

Two: it can deliver twice. A worker POSTs, the endpoint processes it, and the response is lost. The worker times out, retries, the endpoint sees it twice. That's not a loss but it's the failure customers notice, and it's why the idempotency key is in the payload. It's also theoretically unavoidable — that's the Two Generals problem, not a gap in my implementation.

Three: a zombie worker. A worker claims a delivery with a lease, GC-pauses past the lease, and a second worker takes it. Both POST. My design does not prevent this — I don't have fencing on the delivery, because the "resource" is the customer's endpoint and I can't make their server check my token. What I do instead is make the duplicate harmless via the idempotency key. If the customer ignores that key, they get a duplicate. That's a limitation I'd document to customers, not hide.

Q: What if the process crashes right after the POST succeeds but before you mark it delivered? It retries, and the customer gets a duplicate — same case as above. I deliberately chose that over marking it delivered before the POST, which would lose the delivery on a crash. Given the brief says "must not lose events," a duplicate is the correct direction to fail in, and the idempotency key is what makes it tolerable.

4.6 Omission questions

Q: What did you not test? Three things, and I know which. The clock-skew path — I never test what happens if the system clock jumps backwards mid-backoff; next_attempt_at would be in the future and the delivery would stall. The circuit breaker's half-open state under concurrency — I test that it opens and that it closes, but not that exactly one probe gets through when 4 workers hit it simultaneously. And I have no test for the replay endpoint under load, which is where the recovery-flood problem I mentioned would show up.

Q: What's the riskiest line in your diff? The lease renewal in worker.py. It's correct now, but it's the one place where a future change could silently break the at-least-once guarantee — if someone made it renew unconditionally rather than checking that the delivery is still in in_flight, two workers could both believe they hold it and neither would notice. It's also the least-covered path, because testing it needs a controllable clock and I only test the happy path there.

Q: What's the least-defensible thing in here? The _should_retry function. It decides retryability from the status code, and I hard-coded the set. 429 should probably honour Retry-After rather than using my backoff, and I treat all 4xx as permanent, which is wrong for 408 and 425. It works, and it's the code I'd be least comfortable defending as correct rather than reasonable.

4.7 Hostile questions

Q: This function does four things. Why? You're right, and I'd split it. deliver() signs, POSTs, interprets the response, and updates state. The reason it's one function is that all four share the delivery row and I wanted the state transition to be obviously atomic in the reading. That's a weak reason — I could pass the row through. If I were extending this, the first thing I'd do is extract the response interpretation, because that's the part with the most edge cases and it's currently untestable without a real HTTP round trip.

Q: You catch a bare Exception here. Yes, and it's deliberate but under-documented. It's in the worker loop, and it's there so that one delivery raising an unexpected exception doesn't kill the worker and stall every other delivery it would have handled. But you're right that it's too broad — it'll swallow programming errors and make them look like delivery failures. It should catch the transport exceptions specifically and let anything else propagate to a top-level handler that logs and restarts the worker. Note it does not catch CancelledError, since that's a BaseException — but that's luck rather than intent, and I should make it explicit.

Q: This is O(n²) and you know it. Where? (Wait for the answer. If they're right:) Yes — the subscription matching does a linear scan per event over all subscriptions. At my stated 1,000-subscription limit that's fine, and past it it's exactly the fan-out problem I flagged as a non-goal. The fix is an index on event type, which is a one-line schema change I chose not to make because it wasn't the interesting part of the problem. (If they're wrong, say so calmly and show why.)

Q: Why should I believe your benchmark? You shouldn't, entirely, and I said so in the write-up — the mock destination has no latency variance, so my p99 is optimistic in a way real endpoints wouldn't be. What I'd stand behind is the relative finding: throughput is flat from 8 to 16 workers and the profiler puts 62% of time in the claim query. That's a bottleneck identification, and it holds regardless of the absolute numbers.

4.8 The closing move

At the end, unprompted:

"Two things I'd want to flag that you haven't asked about. First, the riskiest thing in here is the lease renewal — it's correct but it's the place where a future change silently breaks at-least-once, and it's the least-covered path. Second, the thing I'd build next isn't a feature, it's a rate limit on the DLQ replay endpoint, because a destination recovering from a long outage makes six hours of dead letters replayable at once and I have nothing bounding that."

Volunteering your own design's weakest point before the interviewer finds it is the single most credibility-generating move available in this round. It demonstrates you have a model of your own system's risk — which is precisely what they are testing — and almost nobody does it, because it feels like arguing against yourself.

Chapter 5: The Second Project

Week 16, different domain, same discipline. Row 15's real requirement is generalization: if you only ever defend the webhook system, you have memorized answers rather than built the skill.

Pick one you have not built. The point is the 48 hours, not the familiarity.

Score the second interrogation against the first. The metric is not "did I answer well" — it is "did I need fewer 'I'd have to look' answers than last time." That delta is the skill.

The Rubric

Hire-bar translation

References

• ../README.md — the track index and the interrogation harness
• ../../projects/README.md — where the builds live
• ../systems-design/WARMUP.md — the outbox pattern, delivery semantics, circuit breakers, retry budgets
• ../systems-design/designs/d01-job-scheduler.md — the same reasoning as a design artifact, with a hostile critique
• ../coding/WARMUP.md#chapter-9-deduplication-and-probabilistic-structures — the idempotency and exactly-once argument
• ../../research/source-report.md — rows 9–15
• Brooker, M. Exponential Backoff and Jitter. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Amazon Builders' Library. Timeouts, retries, and backoff with jitter. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
• Stripe. Idempotent Requests. https://docs.stripe.com/api/idempotent_requests
• Stripe. Webhook signatures. https://docs.stripe.com/webhooks#verify-official-libraries
• GitHub. Validating webhook deliveries. https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
• Kleppmann, M. Designing Data-Intensive Applications, 2nd ed. — Ch. 11 (stream processing, exactly-once)
• Richardson, C. Pattern: Transactional outbox. https://microservices.io/patterns/data/transactional-outbox.html