« Phase 10 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a new tool
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line: buy every mechanism, build the policy and the composition. A circuit breaker is a state machine with a literature; what is unique to you is which tools it protects and what open does. Getting this backwards — writing your own breaker and using a library's opinion about idempotency — is a recognizable pattern in platforms that later need rework.

And a note on Temporal: the resistance is always "it's a big dependency for four workflows." The counter is not a feature comparison. It is: "which engineer is on call for the state machine when it strands a half-completed payment saga at 2 a.m.?"

2. A decision framework for a new tool

Ten questions, in order. Half get answered wrong the first time, and that is the point.

1. What is the side-effect class? No default. If the honest answer is "it depends on the arguments", it is two tools.
2. What is the idempotency key derived from? The business event. If they cannot say, or the answer is "we generate a UUID per call", idempotency is not implemented.
3. Does the downstream honour a key? If not, the gateway is the only defence, and no retry is safe at any layer.
4. Can we query it by our key? The reconciliation question. Nearly free to add during integration design; impossible during an incident.
5. What is the compensation? If there is none, the class is irreversible, and every saga using it must place it last.
6. Which business invariants can a schema not express? "None" means nobody asked the business.
7. What are the limits? Per action, per day, per agent, per tenant.
8. What is the measured p99, and therefore the timeout? Not the SLA. The measurement.
9. What must be redacted? By key name and by value shape.
10. Who owns this tool, and who approves contract changes? A named human, not a team alias.

Question 5 changes designs more than any other. "What is the compensation?" is frequently answered with "there isn't one", which reclassifies the tool and reorders every saga that touches it.

3. Review red flags

In a design document

• The gateway is a library inside the agent runtime.
• No side-effect classification, or a default.
• "We retry on failure" with no distinction between action classes.
• Idempotency keys generated per HTTP attempt.
• No mention of what happens when the gateway crashes mid-call.
• "Exactly-once delivery."
• A saga with no durability story.
• Compensations described as rollbacks.
• An uncompensable step in the middle of a saga.
• A circuit breaker with no minimum throughput.
• A breaker whose open behaviour is "return 503" and nothing else.
• No bulkhead. No timeout.
• Approvals as strings, unauthenticated.
• A dual-control threshold with > instead of >=, or with nobody named as its owner.
• An audit log with no policy version, no model version, or no actor chain.
• "Tamper-proof" claimed with no external anchor.
• Redaction described as a log-pipeline step.
• Refusals not audited.
• A second gateway "because core banking is different".

In code

# Red flag: check-then-act
if store.get(key) is None:
    store.put(key, IN_FLIGHT); execute()        # two callers, two executions

# Red flag: a per-attempt key
idempotency_key = str(uuid.uuid4())             # every retry is a new action

# Red flag: the trace id in the hash
material = {"trace": request.trace_id, ...}     # every retry is a 409

# Red flag: retrying everything
for attempt in range(3): call()                 # including the payment release

# Red flag: retry outside the breaker
breaker.call(lambda: retry(lambda: downstream()))   # 3 failures register as 1

# Red flag: no jitter
delay = base * (2 ** attempt)                   # synchronized herd

# Red flag: a breaker with no throughput floor
if failures / total >= threshold: open()        # 1/1 = 1.0 at 3am

# Red flag: not clearing the window on close
self._state = CLOSED                            # old failures re-open it instantly

# Red flag: a swallowed compensation
try: step.compensate(ctx)
except Exception: pass                          # the worst line in the phase

# Red flag: compensating forwards
for step in completed: step.compensate(ctx)     # must be reversed

# Red flag: bool as int
if isinstance(value, int): ...                  # True is an amount now

# Red flag: an unanchored pattern
re.match(r"PMT-\d+", payment_id)                # "PMT-1-EVIL" matches

# Red flag: invariants before the schema
if args["debit_account"] not in accounts: ...   # KeyError, hiding the real error

# Red flag: redaction after
logger.info("request=%s", request)              # already left the process
audit.append(redact(request))

# Red flag: PII in the error path
raise ValueError(f"bad account {account}")      # and it lands in the log

# Red flag: best-effort audit
try: audit.append(...)
except Exception: pass                          # the action proceeded unprovably

# Red flag: verifying only the content hash
if record.digest() != record.this_hash: ...     # a rewritten chain passes

In an incident review

• "The customer was debited twice" → per-attempt key, or check-then-act.
• "We don't know if it went through" → no reconciliation path.
• "The retries made it worse" → no jitter, no budget.
• "Everything was slow, but nothing was failing" → no bulkhead.
• "It's stuck half-done" → non-durable saga.
• "We couldn't prove what the agent did" → best-effort audit, or a missing field.
• "The breaker never opened" → it was tuned off after false positives.

4. Production war stories

The per-attempt key. Idempotency was implemented, reviewed, tested and documented. The key was str(uuid.uuid4()), generated inside the retry loop. Every retry was a new key. A three-second network blip during a payment run produced eleven duplicate payments, and the code passed every test because a single call works perfectly.

Check-then-act. The store read None, both callers inserted, both executed. It happened twice in eighteen months, both times during a traffic spike, and both times it was attributed to "the client double-submitted". The fix was one ON CONFLICT DO NOTHING.

The trace id in the hash. Every legitimate retry became a 409. Callers, reasonably, worked around it by generating a fresh key on conflict — which disabled idempotency completely and left the 409s in the dashboard looking like the control working.

The synchronized herd. No jitter. Core banking had a two-second hiccup; four hundred agent calls failed together and retried together, 100 ms later, together. The hiccup became a forty-minute outage. The dashboard showed a perfect sawtooth that nobody recognized for the first twenty minutes.

Slow, not failing. A downstream degraded from 200 ms to 25 s. Zero errors, so the breaker saw only successes. Every worker blocked on it; requests to healthy dependencies could not get a thread. The platform was down for eleven minutes because a dependency was slow. There was no bulkhead.

The breaker that was tuned off. Opened four times in the first month, all at low traffic, all false positives (no minimum throughput). Threshold raised from 50% to 90%. Then to "effectively never". Six months later, during a real outage, it did not fire, and the postmortem action was to add the minimum throughput that should have been there originally.

Half a saga. A deploy restarted the pods mid-flow. The hold was placed, the case was opened, and the process holding the saga state was gone. Fourteen customers had holds on their accounts with no corresponding case activity. Found by customer complaints over four days.

The swallowed compensation. except Exception: pass around a compensation call, added during a demo. Compensations failed silently for three weeks. Discovered during reconciliation: 217 orphaned holds.

The uncompensable step in the middle. A saga released the payment at step 2 of 5, then failed at step 4. There was nothing to compensate with. The pattern's entire safety property had been thrown away by the step ordering, and the reordering fix took ten minutes once someone saw it.

PII in the exception. The happy path was redacted meticulously. except Exception as e: logger.error(f"failed: {request}") was not. Nine months of full account numbers in the log aggregator, retained and indexed, discovered by a routine search.

The gateway inside the runtime. "It's the same team, and it saves 20 ms." A prompt-injection proof-of-concept reached a tool that could write to the runtime's memory, and from there the gateway's credentials. The blast radius was the entire estate, and the architecture diagram had shown two boxes.

The audit log with no policy version. Everything was recorded except which policy version allowed it. During an examination, "under what rules was this permitted?" could not be answered for any action older than the current bundle. The remediation was a field addition; the finding was that eight months of records lacked it and always would.

The unauthenticated approver. approvals: ["ahmed", "sara"] arrived as strings in the request body. Dual control was implemented, tested and completely bypassable by anyone who could construct a request. It survived two reviews because the code was correct.

The second gateway. Core banking "needed a special path" for one integration. Six months later there were two audit logs, two idempotency stores, and the special path had neither dual control nor hash chaining. The answer to "what did the agent do?" now depended on which log you read.

5. The interview signal

Signal 1 — process isolation, argued from threat model. Not "we have a gateway service" but "the gateway exists because the kernel's input is attacker-influenced; sharing a process removes the property we built it for."

Signal 2 — the side-effect class with no default. And the reason: a default here is a default retry policy, and both candidates are wrong.

Signal 3 — a different hash executes never. Most candidates say "reject it". The ones who have run one say "and it must never execute, because the caller's key generation is broken, which means their model of what they've already done is broken too."

Signal 4 — exactly-once, reframed. "Exactly-once delivery is impossible — that's two generals. At-least-once plus idempotent handling gives exactly-once effects, which is what you want."

Signal 5 — you volunteer the crash window. Reserved the key, called the downstream, died before storing. This is the question that separates people who have operated one, and very few raise it unprompted.

Signal 6 — compensation is not rollback, with a visible-state example. "The hold was placed, the customer saw a reduced balance, the fraud system scored it. The reversal is its own line on the statement."

Signal 7 — minimum throughput. Naming the breaker parameter everyone omits, with the 3 a.m. consequence and the tuned-off endgame.

Signal 8 — you ask what open does. "A breaker that only fails faster has converted a slow error into a quick one." And the best answer: degrade the capability, so the agent plans without the tool rather than around it.

Signal 9 — bulkhead before breaker. "The breaker handles a failing dependency; the bulkhead handles a slow one, and slow is the more common outage."

Signal 10 — tamper-evident, not tamper-proof. With the external anchor as the thing that closes the gap, and the admission that the code cannot do it alone.

Anti-signals:

• The gateway as a library in the runtime.
• "We retry on failure" with no class distinction.
• Exactly-once delivery claimed.
• Compensation described as rollback.
• A saga with no durability story, described confidently.
• A breaker with no minimum throughput.
• No bulkhead, no timeout.
• "Tamper-proof."
• Redaction as a log-pipeline concern.
• No answer to "what happens if you crash mid-call?"

The question to ask them: "Your gateway reserved an idempotency key, called core banking, core banking applied the payment, and your process died before it stored the response. The caller retries. What happens?" There is no way to answer well without having thought about it, and the best answers get to reconciliation and to "can I query this downstream by my own key?"

6. Mentoring notes

Three exercises, in order of how much they change behaviour:

1. Make them find the per-attempt key. Give them a code sample with uuid4() inside the retry loop and ask what it protects against. Most people take two minutes and never write it again. It is the highest-value five minutes in this phase.
2. Kill the process mid-call. Have them build the smallest possible gateway, then kill -9 it between reserving the key and storing the response, then retry. The design gap is visible rather than theoretical, and the conversation about reconciliation happens naturally.
3. Draw the saga and ask where the uncompensable step is. Then ask what happens if it is in the middle. The moment someone reorders the saga without being told to, they have the pattern.

And the framing for the platform team: this is the phase where the cost of deferring is measured in incidents rather than in effort. Every write tool that ships without an idempotency key is a duplicate waiting for a network blip; every direct network path that survives is a bypass of the whole design. The retrofit is not just code — it is auditing every existing call site for double-execution, which is slow work done under pressure.

The argument that gets it funded is not engineering rigour. It is: "today, a three-second network blip during a payment run produces duplicate payments, and we would find out from the customer. The control that prevents it costs one field and one table."