d04 — Webhook Delivery System

A fully worked design. The reported take-home example (../../../research/source-report.md rows 10–12), approached as a 45-minute design round rather than a 48-hour build.

Design it before you build it. Track E's guide covers the 48-hour execution and the line-by-line interrogation; this covers the architecture round, where the scale is bigger and the deep dives are different.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Per-Destination Isolation
• 7. Deep Dive B: At-Least-Once Without Losing or Flooding
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"We need to deliver webhooks to customer endpoints. Customers register a URL and subscribe to event types. Endpoints are unreliable — they time out, return 500s, and sometimes disappear for days. We must not lose events, and we must not make a struggling endpoint worse.

Design it. Assume we're at meaningful scale."

Two clauses do all the work. "Must not lose events" selects at-least-once and everything that follows. "Must not make a struggling endpoint worse" is the one people skip, and it is where the interesting design is: it means your retry policy is a client of someone else's capacity, and you have to behave.

1. Requirements and Scope

Clarifying questions asked

"At-least-once or at-most-once?" The prompt says do not lose → at-least-once, which obligates me to say: consumers must be idempotent, and I will give them a stable key so they can be.

"Does ordering matter?" Assumed no by default, opt-in per subscription. Ordering forces per-destination concurrency 1, which caps throughput at 1/latency — with a 50 ms endpoint that is 20/s regardless of how much capacity I have. Most consumers do not need it; the ones that do need it a lot.

"How long do we keep trying?" Assumed ~24 hours with exponential backoff, then dead-letter. Long enough to ride out a customer's deploy or an outage; short enough that they are not getting day-old events as news.

"Can one customer's failures affect another's deliveries?" Assumed no — and this is the requirement that produces deep dive A.

"What are the payloads?" Assumed ≤256 KB, larger by reference.

Functional

1. Register subscriptions: URL, event types, secret, options (ordering, custom retry).
2. Accept events; fan out to matching subscriptions.
3. Deliver with signing, retry, and a dead-letter path.
4. Customer-visible delivery status and manual replay.

Non-functional

Explicitly out of scope

• Guaranteed global ordering across destinations.
• Customer-side delivery infrastructure.
• Exactly-once execution on the customer's side — impossible; we provide the key.
• Multi-region active-active.

2. Scale Numbers

Fan-out. 100k events/s with an average of 10 matching subscriptions = 1M deliveries/s. That 10× multiplier is the number that shapes everything, and it is why the delivery table, not the event table, is the scaling problem.

Storage. Deliveries at ~500 B of metadata: 1M/s × 500 B = 500 MB/s = 43 TB/day. That is not storable at that rate for long, so retention is a first-class design decision: keep delivery records for 7 days (300 TB, partitioned by day, dropped not deleted), keep event payloads for 30 days in object storage, and keep an aggregate counter forever.

Worker fleet. 1M deliveries/s at ~200 ms per HTTP attempt: by Little's law, L = 1e6 × 0.2 = 200,000 concurrent HTTP requests in flight. At 500 concurrent per worker (async I/O), that is 400 workers, ×1.5 for AZ tolerance ≈ 600. If it were 2-second endpoints instead of 200 ms, it would be 6,000 workers — so the p99 of your customers' endpoints sizes your fleet, which is a slightly alarming thing to say out loud and exactly right.

Retry amplification. If 5% of destinations are failing and we retry 6 times, those deliveries cost 6× — so 5% of traffic becomes 30% of attempts. The failing minority dominates the fleet, which is the quantitative argument for deep dive A.

Ingest. 100k events/s × (1 event row + 10 delivery rows) = 1.1M row-inserts/s. That does not fit in one database. Sharded by event ID, ~50k inserts/s/shard, ~22 shards. Say it: this is a write-throughput problem, not a storage problem.

3. API Surface

POST /events              {type, payload, idempotency_key}      -> 202 {event_id}
POST /subscriptions       {url, event_types[], secret,
                           ordered?, retry_policy?}             -> 201 {sub_id}
GET  /subscriptions/{id}/deliveries  [?status,&since]           -> [{delivery, attempts, last_error}]
POST /deliveries/{id}/replay                                    -> 202
POST /subscriptions/{id}/replay      {since, until, types[]}    -> 202 {job_id}
GET  /subscriptions/{id}/health                                 -> {circuit_state, success_rate, lag}

Three choices worth defending:

• 202, not 201, on POST /events. We have accepted responsibility for delivery, not completed it. The status code is the contract.
• Bulk replay is a job, not a synchronous call. Replaying a day of deliveries for a recovered destination is thousands of items and must be rate-limited — see the critique.
• /health is customer-facing. Customers cannot fix an endpoint they do not know is failing, and every support ticket you avoid is worth more than the endpoint costs.

4. Data Model

events                                    -- sharded by event_id
  event_id uuid PK, type, payload_ref, created_at, idempotency_key
  UNIQUE (idempotency_key)                -- producer retries don't duplicate

subscriptions                             -- small, replicated everywhere, cached
  sub_id uuid PK, customer_id, url, host, event_types[], secret_ref,
  ordered bool, retry_policy jsonb, state

deliveries                                -- sharded by DESTINATION, partitioned by day
  delivery_id uuid PK, event_id, sub_id,
  destination_host text NOT NULL,         -- denormalised: the shard key
  attempt int, state text,                -- pending|inflight|delivered|failed|dead
  next_attempt_at timestamptz,
  lease_expires timestamptz, worker_id,
  last_status int, last_error text,
  UNIQUE (event_id, sub_id)               -- the fan-out dedupe guarantee

INDEX deliveries_due ON deliveries (destination_host, next_attempt_at)
  WHERE state = 'pending'                 -- partial: sized by PENDING work only

The one decision that matters: deliveries is sharded by DESTINATION, not by event.

Sharding by event is the obvious choice and it is wrong here. Every scan for "what is due" would have to touch every shard, and — much worse — a destination's failures would be spread across every shard, so its retry load and its circuit-breaker state would be global. Sharding by destination means a bad destination's problems are confined to one shard, which is the containment property the whole design needs. That is why destination_host is denormalised onto the row.

The partial index on state = 'pending' keeps the index sized by outstanding work rather than by all work — at 43 TB/day of delivery rows, an index over all of them is not viable.

5. High-Level Architecture

  POST /events
       │
       ▼
  ┌──────────────┐   ONE transaction per shard:
  │  Ingest tier │     INSERT event
  │              │     INSERT delivery rows for matching subs
  └──────┬───────┘   (the OUTBOX pattern — no dual write)
         │
         ▼
  ┌──────────────────────────────────────────────┐
  │  Delivery store   sharded by DESTINATION      │
  │                   partitioned by day          │
  └────────┬─────────────────────────────────────┘
           │ claim due rows for owned shards
           │ FOR UPDATE SKIP LOCKED
           ▼
  ┌───────────────────────────────────────────────────────────┐
  │  Delivery workers                                          │
  │   ┌─────────────────────────────────────────────────┐     │
  │   │ Per-destination admission                        │     │
  │   │  • concurrency cap    • circuit breaker          │     │
  │   │  • token bucket       • deadline                 │     │
  │   └──────────────────┬──────────────────────────────┘     │
  │   HMAC sign → POST (timeout) → interpret → record         │
  └────────┬────────────────────────────┬─────────────────────┘
           │ delivered                  │ attempts exhausted
           ▼                            ▼
      status API                  dead-letter ──▶ rate-limited replay

The two hard parts — say these at minute 10:

1. Per-destination isolation. A dead destination must consume a bounded share of the fleet and must not slow anyone else.
2. At-least-once without either losing events or flooding a recovering destination. The second half of that sentence is the part people miss.

6. Deep Dive A: Per-Destination Isolation

The problem, quantified. 600 workers, 1M deliveries/s. One customer's endpoint starts timing out at 30 seconds. They have 50k deliveries/s. Without isolation:

50,000 deliveries/s × 30 s timeout = 1,500,000 concurrent stuck requests

against a fleet sized for 200,000. The fleet is 7.5× oversubscribed by one customer, every other destination's deliveries queue behind them, and the whole system is down for everyone. That is head-of-line blocking at scale, and it is the failure this design exists to prevent.

Layer 1 — Per-destination concurrency cap

At most N in-flight deliveries per destination (default 20, per-plan configurable). Enforced by a semaphore keyed on destination_host, held in the shard's worker set.

Worst case now: 20 × timeout / mean_latency workers occupied by one destination. At 20 concurrent and a 30 s timeout, that is 20 in-flight slots — 0.01% of the fleet, not 750%.

Layer 2 — Circuit breaker per destination

Concurrency caps bound the damage; they do not stop the waste. 20 slots × 30 s of timeouts, over and over, is pure burn — and worse, it is load we are adding to a struggling endpoint, which the prompt explicitly forbids.

closed  → open       50% failures over ≥20 attempts in 60 s
open    → half-open  after backoff (30 s, doubling to 30 min, jittered)
half-open → closed   ONE probe succeeds
half-open → open     the probe fails; back off further

Three details that matter:

• A rate over a minimum volume, never an absolute count. Three failures out of five is noise on a low-traffic destination.
• One probe in half-open, not a flood. Reopening the gates on a recovering endpoint re-kills it immediately.
• Circuit state is per destination and shared across workers — otherwise 600 workers each need 20 failures to learn, which is 12,000 wasted attempts. It lives in the shard's coordination store with a short TTL.

Layer 3 — Per-destination rate limit

Even a healthy destination has a capacity. Blasting a customer with 50k/s because they subscribed to a high-volume event is us being a bad citizen, and it is how you get blocked.

A token bucket per destination, default derived from their observed successful throughput, overridable per plan. Rate-limited deliveries are re-queued, not dropped — this is backpressure, not shedding, because we own the durability guarantee.

Layer 4 — Shard-level containment

Sharding by destination means all of a bad destination's rows are on one shard, so even the database load from their retries is confined. Their deliveries_due index churn does not touch anyone else's shard.

The four layers answer different failure modes and that is the point:

7. Deep Dive B: At-Least-Once Without Losing or Flooding

The durability boundary

Where exactly is an event "accepted"? At the commit of the transaction that writes the event and its delivery rows, together. Both, or neither.

Writing the event and then publishing to a queue is a dual write — two systems that fail independently, with no safe ordering. Crash between them and you have an event nobody will deliver, or a delivery for an event that does not exist. The outbox pattern makes it one write.

The cost is that workers poll rather than consume from a queue, which is the bottleneck §9 identifies. I would take that trade every time, because the alternative is silently losing events at a rate proportional to your crash rate.

The duplicate is unavoidable — make it harmless

Three places a duplicate arises, and only the last is a bug:

1. Response lost after the endpoint processed it. We time out, retry, they see it twice. Unavoidable — the Two Generals problem. Not a gap in the design.
2. Worker crashes after the POST, before recording. Same shape.
3. A zombie worker: claims a delivery, GC-pauses past its lease, another worker takes it, both POST.

For (3), the standard answer is a fencing token — but here it does not apply, and saying so precisely is the strong move: fencing requires the resource to check the token, and the resource is the customer's HTTP endpoint. We cannot make their server reject a stale write.

So the mitigations are:

• A stable idempotency key — sha256(event_id | sub_id), stable across every attempt of that delivery — in both the payload and an Idempotency-Key header. This is the only real fix, and it requires the customer to use it. Document that.
• Short leases with a deadline shorter than the lease, so the window in which two workers can both be in flight is small: lease 90 s, HTTP timeout 30 s, renewal at 30 s.
• Signed timestamps, so a very delayed duplicate is at least detectable by the customer.

Retry policy, and why jitter is not enough

Exponential backoff with full jitter — uniform(0, min(cap, base·2^n)), base 1 s, cap 1 h, ~15 attempts spanning 24 hours.

Jitter alone is insufficient and this is the part to emphasize. Perfectly jittered retries still multiply offered load. With 6 attempts against a destination failing 95% of the time, we send 2.85× its normal traffic — to an endpoint that is already struggling. The circuit breaker is what actually bounds it; jitter only desynchronizes what remains.

The order is: circuit breaker → retry budget → jitter. Most people say jitter first.

The recovery flood — the failure people forget

A destination is down for 6 hours. We have 6 hours × their rate of pending deliveries. It comes back. The circuit closes. And we deliver 6 hours of backlog as fast as the fleet allows — which kills it again, immediately.

Three mitigations:

1. Ramped admission. On circuit close, start at 10% of their rate limit and ramp over several minutes, watching the success rate. This is a load balancer's slow-start, applied to a recovering dependency.
2. Prioritize new over backlog. New deliveries have someone waiting; a 6-hour-old one does not. Claim with ORDER BY next_attempt_at but reserve a share of each destination's concurrency for deliveries younger than a threshold.
3. A per-destination catch-up budget — the backlog drains at a bounded rate, so it takes a while, and that is correct. A backlog that drains instantly is a backlog that takes the destination down.

Say the general principle: a system recovering from an outage is at its weakest exactly when the load is at its highest. Every recovery path needs a ramp.

8. Failure and Recovery

Deliberately accepted: we deliver duplicates when a response is lost, and we cannot prevent it. I accept it because the alternative — at-most-once — silently drops events, which the brief forbids. The mitigation is the idempotency key, and it requires customer cooperation, which I document as an explicit part of the contract rather than pretending we solved it.

9. Bottlenecks and Evolution

1. The claim query, at ~3× load. SELECT ... FOR UPDATE SKIP LOCKED on (destination_host, next_attempt_at) while next_attempt_at is rewritten on every attempt churns the index badly — under MVCC each update writes a new tuple and vacuum falls behind. Fixes in order: partition deliveries by day and shard so each index is smaller and vacuum parallelizes; then move the hot dispatch path to an in-memory per-destination priority queue backed by periodic checkpoints, keeping the store as the record.

2. Fan-out at ingest, at any large subscription count. Inserting 10 delivery rows per event in the ingest transaction is fine at 10; at 100k subscriptions to one event type it is a 100k-row transaction holding locks. Fix: lazy fan-out — write the event plus a fan-out job, and have workers materialize delivery rows in batches of ~1,000. The cost is a second hop before the first delivery, so I would do it only above a threshold, keeping eager fan-out for the common case.

3. Delivery-record storage, immediately. 43 TB/day. Partition by day and drop partitions rather than DELETE (which is vastly more expensive and generates enormous vacuum load). Archive to object storage. Keep a rolled-up counter forever, since that is what customers actually query.

4. The subscription-matching path. Matching an event to subscriptions must not scan. An inverted index from event_type → sub_ids, cached at the ingest tier and invalidated on change. At 100k events/s this must be a memory lookup.

At 100×: the design becomes a per-destination streaming problem rather than a database problem — each destination gets a durable log and a dedicated consumer with its own offset. That is a rewrite, not a scaling, and I would say so.

10. Tradeoffs Explicitly Rejected

Rejected: a message queue as the system of record. Attractive — SQS/Kafka already do visibility timeouts and retries. Rejected because per-destination isolation needs mutable per-destination state (circuit state, rate budget, backlog) that a queue does not model; cancellation of an enqueued message is unsupported; and delayed delivery caps (SQS: 15 minutes) do not span a 24-hour retry window. Flip condition: if retries were bounded to minutes and isolation were not a requirement, the queue alone is simpler and I would use it.

Rejected: sharding deliveries by event. The obvious choice. Rejected because it spreads each destination's failures across every shard, so retry load and circuit state become global and the containment property disappears. Flip condition: if destinations were uniformly reliable — i.e. if the hard part were not there — event sharding gives better ingest distribution.

Rejected: global ordering. Rejected because it forces per-destination concurrency 1, capping throughput at 1/latency. Offered as an opt-in per subscription, so the customers who need it pay for it and nobody else does. Flip condition: if the product were a change-data-capture feed where order is semantically required, ordered would be the default and the design would be a per-destination log with offsets.

Rejected: fencing tokens for the zombie case. The textbook answer, and it does not apply: fencing requires the resource to check the token, and the resource is a customer's HTTP endpoint we do not control. Rejected honestly rather than cargo-culted. Mitigated with short leases and a stable idempotency key.

Rejected: retrying 4xx. Rejected because a 400 is deterministic — retrying 15 times over 24 hours wastes our capacity and theirs to reach the same answer. Exceptions: 408, 425, 429. Flip condition: if a customer's gateway returned 403 during a token refresh, a bounded retry on 403 would be worth it — which is why retryable status codes are per-subscription configurable.

Rejected: at-most-once. Simpler; no idempotency requirement on the customer. Rejected because the brief says do not lose events. Flip condition: an event class where a duplicate is worse than a miss — a payment notification — would justify a per-subscription at-most-once mode, with the drop documented.

The Hostile Critique

C1. "Your circuit breaker is per destination and shared across 600 workers via a coordination store. That's a read on every delivery attempt — a million reads a second to check circuit state. What does that cost, and what happens when that store is slow?"

C2. "A customer has 40,000 subscriptions pointed at the same host — they're multiplexing by path. Your concurrency cap is keyed on destination_host. So all 40,000 subscriptions share one cap of 20. Is that what you meant?"

C3. "You dead-letter after 24 hours. A customer is down for 26 hours — a bad weekend deploy. They lose everything, and 'we must not lose events' was requirement one. What do you actually tell them?"

C4. "You prioritize new deliveries over backlog by reserving concurrency. Under sustained overload for one destination, the backlog never drains — new work keeps arriving and keeps winning. Walk me through what that queue looks like after a day."

C5. "Sharding by destination. One customer is 40% of your traffic. What does that shard look like, and what happens when it needs to split?"

C6. "You sign with HMAC over timestamp.body. Customer's clock is 10 minutes off, so every delivery fails their signature check with a timestamp-tolerance error. From your side, what does that look like, and what does your system do about it?"

The Revision

R1 — Circuit state must be local with async propagation (answers C1)

The critique is right and I had not costed it: 1M reads/s against a coordination store, on the hot path, to read a boolean.

Change: circuit state is local per worker, with gossip.

• Each worker maintains its own per-destination failure counters and its own circuit state, in memory. Zero reads on the hot path.
• Workers publish state transitions only (not counts) to a lightweight pub/sub — a few messages/s cluster-wide, not a million.
• A worker receiving "destination X opened" adopts the open state immediately. Opening propagates fast; closing does not — each worker must independently see a successful probe before closing, so a single lucky probe cannot reopen the gates fleet-wide.

Cost: during the first seconds of a destination's failure, workers that have not yet seen the gossip keep trying — a bounded burst of wasted attempts, versus 1M reads/s forever. And the asymmetry (fast to open, slow to close) is deliberate: false-open costs a little latency, false-close costs the destination.

R2 — Isolate on the subscription's effective concurrency key (answers C2)

The critique found a genuine modelling error. destination_host is right for politeness — we should not overload a host — but wrong for fairness between subscriptions on that host.

Change: two keys, two purposes.

So 40,000 subscriptions on one host still share a host cap of 20 (correct — it is one server), but no single subscription can monopolize it, and the fair share is computed from active subscriptions rather than registered ones.

Cost: a second semaphore and a periodically-recomputed active-subscription count per host. And a customer with 40,000 subscriptions on one host genuinely gets low per-subscription throughput — which is correct, because their server is the constraint, and it is exactly the conversation to have with them via the /health endpoint.

R3 — Dead-letter is not deletion (answers C3)

The critique exposes a contradiction between requirement 1 and my retention policy.

Change: separate stopping delivery attempts from discarding the event.

• At 24 hours, delivery attempts stop and the delivery is marked dead. That is a resource decision, not a data decision.
• The delivery record and the payload reference survive for the full 7/30-day retention.
• Bulk replay (POST /subscriptions/{id}/replay {since, until}) lets a recovered customer request everything they missed — rate-limited, as a job, with progress.
• The /health endpoint shows dead_count and oldest_dead_at, and we proactively notify the customer when deliveries start dead-lettering.

So the honest statement of the guarantee becomes: we attempt delivery for 24 hours; we retain the event for 30 days and you can replay it. That satisfies "do not lose events" without retrying forever, and it is a contract a customer can plan around.

Cost: storage (already accounted), and a replay path that must itself be rate-limited — see R4.

R4 — Backlog needs a guaranteed floor, not just a reservation (answers C4)

The critique is correct: reserving concurrency for new work means that under sustained overload, backlog starvation is the stable state.

Change: invert it — reserve a floor for backlog, not a share for new work.

per-destination concurrency C:
   ≥ 20%  reserved for the OLDEST pending deliveries   (guaranteed drain)
   ≤ 80%  for new deliveries
   unused reservation spills to new work

Now the backlog drains at ≥20% of the destination's capacity regardless of incoming rate — so its drain time is bounded — while new work still gets most of the capacity when there is no backlog.

And the escape valve: if the backlog exceeds a threshold and is not shrinking, the system sheds at ingest for that subscription with a 429 and tells the customer via /health. We cannot accept an unbounded liability for a destination that cannot keep up; accepting it and never delivering is worse than refusing it.

Cost: a customer whose endpoint is persistently under-provisioned starts getting rejected at ingest. That is the correct outcome and it must be visible, not silent.

R5 — Shard splitting must be by subscription, not host (answers C5)

The critique identifies a hot-shard problem I created by sharding on destination.

Change: the shard key is hash(destination_host) for placement, but a shard that exceeds a load threshold splits by (destination_host, sub_id), so one host's subscriptions can span shards.

• Circuit state and the host concurrency cap stay per host, coordinated via the same gossip as R1 — they are host properties and must not fragment.
• Only the storage and claim load splits.

Cost: a host whose subscriptions span shards needs its cap enforced across shards, which is the gossip path again — so a brief window where the cap is exceeded during a split. Bounded, and far better than a shard that cannot be split.

And the guard: never split automatically during a failure. A shard that is hot because its destinations are failing must not trigger a rebalance, which would add load during an incident. Split on sustained healthy load only.

R6 — Signature failures need a distinct signal (answers C6)

The critique is good because it names a failure that looks like success to a naive design: the customer returns a 4xx, we do not retry (correct for 4xx), and the customer silently receives nothing while their dashboard says "delivered: 0, failed: everything" with no useful reason.

Change: treat authentication failures as their own class.

• We already send X-Timestamp. Customers rejecting on timestamp tolerance typically return a specific status; we cannot rely on that, so instead: track the per-subscription 4xx rate by status code, and when a subscription's failures are ≥95% a single 4xx code, surface it as a distinct likely_configuration_error state on /health rather than a generic failure.
• Proactively notify on that state — this is a customer misconfiguration, and the only thing that fixes it is telling them.
• Support two active secrets with a grace window so rotation is never the cause.
• Publish our clock in the X-Timestamp header (already) and document the tolerance, so a customer with a skewed clock can diagnose it.

Cost: a heuristic that can misfire — a genuinely broken endpoint returning uniform 500s is not a config error. Mitigated by scoping the classification to 4xx only, and by making it advisory rather than changing delivery behaviour.

The general lesson worth stating: a failure mode where the system is working correctly and the customer still gets nothing is the worst kind, because no internal alarm fires. Those need customer-facing observability, not better internal handling.

References

• ../../take-home/WARMUP.md — the same system as a 48-hour build, with the decision log and the 40-question interrogation
• ../WARMUP.md#chapter-9-delivery-semantics-and-the-outbox — outbox, DLQ, exactly-once
• ../WARMUP.md#410-load-control — circuit breakers, retry budgets, the recovery ramp
• d03-rate-limiter.md — the per-destination token bucket, in depth
• ../../coding/harness/problems/event_dedupe/ — idempotency and reordering as a timed problem
• Stripe. Webhooks and Idempotent Requests. https://docs.stripe.com/webhooks · https://docs.stripe.com/api/idempotent_requests
• GitHub. Validating webhook deliveries. https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
• Richardson, C. Pattern: Transactional outbox. https://microservices.io/patterns/data/transactional-outbox.html
• Brooker, M. Exponential Backoff and Jitter. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Amazon Builders' Library. Avoiding insurmountable queue backlogs. https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/