« Phase 10 · Warmup · Track Overview

Deep Dive — Mechanisms and Failure Modes

The warmup established what the pieces are. This takes them apart: how each mechanism actually works, what breaks, and what the fix costs.

Table of Contents

• 1. The idempotency store as a state machine
• 2. Concurrency: the conditional write
• 3. The crash window, exhaustively
• 4. What the request hash covers
• 5. Retry, backoff, jitter
• 6. The breaker's window
• 7. Half-open, and the thundering probe
• 8. Timeouts, and the budget that composes
• 9. Saga durability
• 10. Compensation ordering and the dependency graph
• 11. Validation: the subset that matters
• 12. Redaction failure modes
• 13. The audit write path
• 14. Merkle trees: the industrial version
• 15. Performance
• 16. Failure modes

1. The idempotency store as a state machine

                    begin(key, hash)
                          │
              ┌───────────┴────────────┐
         no record                  record exists
              │                        │
              ▼               ┌────────┴─────────┐
         IN_FLIGHT       hash differs        hash matches
         (execute)            │                  │
              │            CONFLICT      ┌───────┴────────┐
      ┌───────┴──────┐      (409)    IN_FLIGHT       COMPLETED
   complete()      fail()             (409+RA)     (replay stored
      │               │                             response)
      ▼               ▼
  COMPLETED      record removed

The transition worth arguing about is fail(). Two defensible designs:

Release the key (what the lab does). The caller may retry. Correct only because the downstream is itself keyed — if the gateway's key is the only key in the system, releasing it is how you get a double payment.

Mark it FAILED and keep it. Safer, and it strands every caller whose network blipped: they can never retry that key, so they must generate a new one, which defeats the purpose.

The real answer depends on whether the downstream is keyed, which is why "does this API accept an idempotency key?" belongs in integration design rather than in an incident. Whatever you choose, document it — this is the kind of decision that is invisible until it is expensive.

2. Concurrency: the conditional write

The in-memory store in the lab is single-threaded, so the in-flight case is demonstrated rather than raced. In production, begin() is where the concurrency lives:

-- The whole mechanism in one statement.
INSERT INTO idempotency (key, state, request_hash, created_at)
VALUES ($1, 'in_flight', $2, now())
ON CONFLICT (key) DO NOTHING
RETURNING key;

Rows returned means we won and should execute. No rows means somebody else has the key, and we read the existing record to decide between replay, conflict and in-flight.

The failure mode of the naive version:

# BROKEN under concurrency: two callers both read None, both insert, both execute.
if store.get(key) is None:
    store.put(key, IN_FLIGHT)
    execute()

Check-then-act across a network is not atomic. The database has to arbitrate, which means the unique index is the control. A Redis equivalent is SET key value NX PX ttl — same property, same reason.

And the operational detail: the TTL must exceed the longest plausible retry window. 24 hours is typical. Too short and a legitimate retry after a long outage re-executes; too long and the table grows without bound, which is a capacity problem rather than a correctness one — so err long.

3. The crash window, exhaustively

Six places the gateway can die, and what each leaves behind:

Rows 2–4 are the same problem: we do not know whether it happened. Three resolutions:

Reconcile. Query the downstream by our key. This is the correct answer and it requires the downstream to support it — a GET /payments?client_reference=.... Ask for this during integration design; it is nearly free to add then and impossible to add during an incident.

Lease timeout. Treat an IN_FLIGHT record older than N minutes as abandoned and allow a retry. Safe if and only if the downstream is keyed. If it is not, you have chosen to risk a double payment to avoid a stuck record, which is the wrong direction for a bank.

Manual resolution. Alert, and a human queries the downstream. Correct for irreversible actions and unworkable at volume.

The last row is worth its own note: an audit record that was never written leaves a gap in the sequence, which the chain verifier detects as a sequence mismatch. That is not a repair, but "we know a record is missing" is a much better position than "we do not know whether a record is missing".

4. What the request hash covers

material = {"tool": ..., "args": ..., "agent": ..., "user": ..., "tenant": ...}

Include and exclude are both deliberate:

sort_keys=True is not cosmetic: {"a":1,"b":2} and {"b":2,"a":1} are the same request, and a digest that says otherwise turns a legitimate retry into a conflict.

The subtler question is where the key comes from. A key derived from the business event — the payment id plus the operation — is correct: two attempts to release the same payment are the same action however many times the agent decides to do it. A key generated per HTTP attempt is useless: every retry is a new key, so there is no idempotency at all. This is the most common way the whole mechanism is silently disabled.

5. Retry, backoff, jitter

Retries are the mechanism that turns one dependency's bad minute into an outage. Three properties:

Bounded. max_attempts from the side-effect class. Not "until it works."

Exponential. 100 ms, 200 ms, 400 ms. Linear backoff barely reduces load; exponential gives the dependency room to recover.

Jittered. This is the one that gets left out and it is the one that matters:

# Without jitter: every client that failed at t retries at t+100ms, together, forever.
delay = base * (2 ** attempt)

# With full jitter: the herd is spread across the window.
delay = random.uniform(0, base * (2 ** attempt))

Without jitter, a downstream that fails a thousand requests receives a thousand retries in the same millisecond, fails them all, and receives them again 200 ms later. The synchronized herd is self-sustaining, and it is why a dependency that was briefly sick stays sick.

And retry budgets, which are the modern refinement: cap retries at a fraction of total traffic (say 10%), globally. Per-request retry limits still permit a 3× traffic amplification exactly when the system is least able to absorb it; a budget bounds the amplification regardless of how many individual requests are failing.

6. The breaker's window

Two window types, and the choice has real consequences:

Count-based — the last N calls. Simple; the failure is that on a low-traffic endpoint the window may span hours, so the breaker reacts to failures that are long over.

Time-based — calls in the last T seconds. Correct for most things, and it is why minimum_throughput exists: a time-based window can legitimately contain one call.

def _should_open(self) -> bool:
    total = len(self._events)
    if total < self.minimum_throughput:
        return False                       # ← the whole point
    failures = sum(1 for _, ok in self._events if not ok)
    return failures / total >= self.failure_threshold

Drop the guard and the breaker opens on the first failure of a quiet night. It gets tuned off within a month, and its absence is discovered during the next real outage.

Closing must clear the window. After a successful recovery the old failures are still inside the rolling window, so the first new failure re-opens the breaker immediately and you oscillate. Clear on close.

Slow calls count as failures. resilience4j's slowCallRateThreshold exists because a dependency answering in 30 s is not returning errors — the breaker sees successes while every thread blocks. It is not a substitute for a bulkhead (§8), but a breaker that only counts errors misses the more common outage.

7. Half-open, and the thundering probe

Half-open admits a limited number of calls. The failure mode of admitting all of them:

   t=0    breaker opens, 1000 rps queued behind it
   t=30   breaker half-opens
   t=30   1000 requests hit a downstream that has just restarted
   t=30   it dies again
   t=30   breaker re-opens
   ...

The dependency never gets a quiet moment to warm caches, refill pools and JIT its hot paths. Admit one to three, and hold the rest.

Two more details:

A single failed probe re-opens. Not a rate, not a threshold. The downstream just told you it is still sick; there is nothing to average.

The probe should be cheap and representative. A health-check endpoint is cheap and not representative — it can pass while the real path is broken. A real request is representative and may be expensive. For reads, use a real read. For writes, this is genuinely hard, and using a read as the probe for a write path is a compromise you should name rather than hide.

8. Timeouts, and the budget that composes

The control people forget entirely, and the one without which the other two do nothing:

A call with no timeout has an infinite one.

Timeouts must compose. If the caller's budget is 5 s and there are three sequential downstream calls, each cannot be 5 s. The pattern is a deadline propagated through the call chain:

   request arrives, deadline = now + 5000ms
     ├─ call A: timeout = min(A_default, deadline - now)   → 5000ms
     ├─ call B: timeout = min(B_default, deadline - now)   → 3200ms
     └─ call C: timeout = min(C_default, deadline - now)   →  900ms

Without deadline propagation, a caller that has already given up leaves work running downstream — work that consumes a connection, a thread and a database lock on behalf of nobody. At scale this is a significant fraction of a struggling system's load: effort spent on requests whose callers are gone.

gRPC propagates deadlines natively. HTTP does not, which is why most enterprise stacks pass one in a header and honour it explicitly.

9. Saga durability

The lab's saga is in-memory: a process restart mid-saga loses it, leaving the bank half-updated with no record of what should happen next. Durability is the entire reason Temporal and Durable Functions exist.

What durability requires:

Temporal's approach is worth understanding even if you buy it: workflow code is replayed from an event history, and side effects go through activities whose results are recorded in that history. On restart, the workflow re-runs from the top, but every activity call returns its recorded result instead of executing — so the code reaches its previous state without repeating any effects. That is why workflow code must be deterministic (no random, no now(), no unordered map iteration) and why every real side effect must be an activity.

Building this yourself is a year of work and it is a solved problem. Use it.

10. Compensation ordering and the dependency graph

Reverse order is the default and it is right most of the time. The reasoning:

    step 1: place-hold      → creates hold H
    step 2: open-case       → references H
    step 3: post-refund     → FAILS

Compensate 2 then 1. Compensating 1 first releases H, and then close-case operates on a case that references a hold that no longer exists — which may fail, or may succeed and leave a dangling reference.

Where reverse order is not sufficient: when steps have a dependency graph rather than a chain. If steps 2 and 3 are independent and step 4 depends on both, their compensations can run in parallel — but only if you have modelled the graph. Most sagas are chains, reverse order is correct, and the graph case is worth knowing exists rather than building speculatively.

The uncompensable step goes last. This is the sequencing rule the pattern hands you: order the saga so that everything that could fail happens before the step that cannot be undone. A saga that releases a payment at step 2 of 5 has thrown away its own safety property.

Compensations are actions. Same contract validation, same authorization, same audit, same idempotency. A compensation that bypasses the gateway is an unaudited write to the bank — which is precisely the thing this phase exists to prevent, arriving through the back door.

11. Validation: the subset that matters

The lab implements type, required, properties, additionalProperties, items, pattern, enum, minimum, maximum — which covers the overwhelming majority of real tool schemas.

The two that bite:

bool is a subclass of int.

isinstance(True, int)        # True

So {"amount": True} passes a naive integer check. Check bool explicitly, first.

re.match is not re.fullmatch.

re.match(r"PMT-\d+", "xxPMT-123yy")      # matches from position 0? No — but
re.match(r"PMT-\d+", "PMT-123-EVIL")     # MATCHES. Anchor it.

A pattern intended as a format check that only anchors at the start accepts a suffix. For an id that becomes a path segment or a downstream lookup key, that is an injection vector.

What the subset omits, and when you will want it: $ref (shared definitions across tools), oneOf / anyOf (polymorphic payloads), format (date-time, iban), and dependentRequired ("if payment_type is SWIFT then bic is required"). The last is the one you will reach for first in a bank, and the honest answer is that it lives in an invariant.

12. Redaction failure modes

The last row is the design rule and the one worth repeating: redaction that happens anywhere other than before serialization is a report, not a defence.

And the error-path rows are where real leaks happen. Teams redact the happy path carefully and then log f"failed: {request}" in an exception handler.

13. The audit write path

Is the audit write blocking?

For a bank the answer is usually: write an "attempting" record synchronously before, and a "result" record after. Two records, and their pairing is itself checkable — an "attempting" with no "result" is exactly the crash window from §3, and now it is visible.

The outbox pattern (Phase 12) is the rigorous version where the downstream shares your database: write the effect and the audit record in one transaction, and publish from the outbox afterwards. It is unavailable across a bank's estate, which is why the two-record approach is the practical answer.

And the audit write is not best-effort. If it cannot be written, the action does not proceed. That is a real availability cost and the right trade: an action nobody can prove happened is worse than an action that did not happen.

14. Merkle trees: the industrial version

A hash chain is O(n) to verify: to prove record 5 is in the log, you must have every record.

A Merkle tree gives O(log n) inclusion proofs:

                    root
                  /      \
              h(01)      h(23)
              /   \      /   \
            h0    h1   h2    h3
            │     │    │     │
           r0    r1   r2    r3

To prove r2 is in the tree you present h3 and h(01) — two hashes, not four records. At a million records that is 20 hashes instead of a million.

Two properties that matter operationally:

• Consistency proofs. Prove that the log at size N is a prefix of the log at size M — i.e. that nothing was retroactively inserted or removed. A hash chain gives this only by replaying everything.
• Third-party auditability. Someone can verify a specific record without being given the whole log, which matters when the log contains other customers' transactions.

This is what Certificate Transparency (RFC 6962) and Sigstore's Rekor implement, and both are worth reading. For a bank's action log, a hash chain plus hourly external anchoring is usually sufficient; reach for a Merkle tree when a third party needs to verify individual records without seeing the rest.

15. Performance

The audit write dominates, which is what makes the "two records, before and after" design a real decision rather than an obvious one. Options if it hurts: batch the "attempting" records (accepting a small loss window), or write to a fast local WAL and ship asynchronously (accepting the WAL as a new failure domain).

What is not worth optimizing: schema validation and hashing. They are three orders of magnitude below the downstream call, and somebody will propose caching them.

16. Failure modes