« Phase 08 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 08 — Staff Engineer Notes: Durable Agent Execution

The person who uses Temporal can annotate a function with @workflow and wire up an activity. The person trusted to own durable execution can tell you, before writing a line, which of the system's problems are yours to solve and which the platform solves for you — and can look at an existing "durable agent" and name the three ways it will silently corrupt state in production. This doc is about that judgment: the decisions a staff engineer owns here, when to reach for durable execution and when not to, and the exact signal an interviewer is listening for.

The decisions you own

  1. The workflow/activity boundary. This is the single most consequential design call, and it is yours. Everything with a side effect or a non-deterministic input goes in an activity; everything that is a pure decision goes in the workflow. Get it wrong — a datetime.now(), a config read, a dict iteration, an LLM call in the workflow body — and you have planted a latent replay bug that detonates on your next deploy, not today. The test is mechanical: "is this a decision or an effect on the world?" Effects are activities.

  2. Idempotency of every activity. The platform gives you exactly-once for recorded effects and at-least-once for in-flight ones. Closing that gap — idempotency keys the downstream dedupes on, upserts instead of inserts, "charge with key abc" not "charge" — is engineering you must do. "The engine makes it exactly-once" is the junior answer; "the engine makes recorded effects exactly-once and I make activities idempotent to cover the at-least-once in-flight window" is the staff answer.

  3. History-size budget and continue-as-new points. For any long-horizon or looping agent, you own the decision of where to truncate history. Nobody warns you when you are trending toward the cap; you have to have designed the checkpoint.

  4. The versioning strategy for a live system. You own the rule that says "we only append steps, we gate risky changes behind a patch marker, we drain old worker builds." This is the difference between a durable system you can deploy weekly and one nobody dares touch.

  5. What is a signal vs a query vs an activity. Human approval is a signal (durable, resumes the workflow). Reading current state without advancing it is a query. Fetching data from a system is an activity. Conflating these is a common design smell.

When to reach for it — and when not

Durable execution is not free; the failure mode of over-adopting it is real. A decision framework:

  • Reach for it when the work is long-running (minutes to days), spans multiple side-effecting steps, must survive process death, waits on humans or external events, or must be exactly-once on money/irreversible actions. Agents that move money, edit production, or run multi-step tool sequences over a long horizon are the sweet spot — this is precisely why Temporal stood up an "AI Foundations" team and why Citi/Cohere phrase JDs as "reliable execution of agent workflows."

  • Do not reach for it when the work is a single short request/response (a 2-second chatbot turn — just retry the whole thing), when a plain queue + idempotent consumer already suffices, or when the team cannot absorb the operational model (determinism discipline, versioning, a stateful cluster or a managed service bill). A state machine in a database with a cron poller is a legitimate, lower-ceremony alternative for simple flows; AWS Step Functions is the managed-JSON-DSL point on the spectrum (less code-native, less flexible, less to operate); an event-driven saga on a message bus is the choice when you want loose coupling over a central orchestrator. The staff move is knowing this spectrum exists and placing the workload on it deliberately, not defaulting to "Temporal for everything."

The strongest anti-pattern to name out loud: reaching for durable execution to paper over a non-idempotent downstream. Durability does not make a non-idempotent effect safe; it makes an idempotent effect reliable. Fix the idempotency first.

Code-review red flags

  • time.time(), datetime.now(), random, uuid4(), or any I/O in workflow code. Instant block. These are replay bombs.
  • Iterating a dict/set and branching on order in workflow code.
  • An activity with a side effect and no idempotency key.
  • A retry policy with unbounded attempts and no jitter (retry storm risk).
  • A large payload (a full model response, a file blob) recorded directly into history instead of stored out-of-band with a reference.
  • A workflow edited in place — steps reordered or inserted in the middle — with no version marker, while executions are in flight.
  • A "wait for approval" implemented as a polling loop or a held thread instead of a signal.
  • Catching a NonDeterministicError and retrying — it is deterministic; retrying reproduces it.

War stories that teach the lesson

  • The deadline that expired instantly. A workflow set a timeout with datetime.now() + delta in workflow code. It ran fine for weeks. Then an unrelated deploy evicted the worker cache, the workflow replayed, now() returned the current (much later) time, the computed deadline was in the past, and the workflow "expired" the instant it recovered. The fix is one line (ctx.now()); the lesson is that the clock bug does not fail at write time, it fails at replay time, which can be weeks later.

  • The double refund. A non-durable batch job refunded a customer, then crashed before marking the refund done, then re-ran from scratch and refunded again. Durable memoization would have replayed the recorded refund instead of re-issuing it. The customer got paid twice; finance noticed, not monitoring.

  • The Tuesday workflow that died on Friday's deploy. A workflow parked on a human approval since Tuesday. Friday's release inserted a validation step before the charge. When the human finally approved, the week-old history no longer matched the new code → NonDeterministicError, stuck workflow. Nothing was wrong at deploy time; the failure was latent in every parked execution and surfaced one at a time. This is the story that justifies the entire versioning discipline.

The interview signal

When an interviewer asks "how do you make a long-running agent survive a crash," they are listening for a specific chain, and most candidates deliver at most the first link:

"Event sourcing plus deterministic replay: every side effect is an activity recorded to an append-only history; on recovery the engine re-runs the workflow and each recorded activity returns its saved result instead of re-executing, so it fast-forwards to the crash point. That requires the workflow to be deterministic — time, randomness, and I/O go through the context or into activities — because replay must reproduce the same command sequence to line up with history. Recorded effects are exactly-once; in-flight effects are at-least-once, so activities carry idempotency keys. Human waits are signals: the workflow suspends holding no resources and resumes when the signal is delivered."

Delivering that whole chain, unprompted, is the Staff/Principal signal. It shows you understand the mechanism, its precondition, its guarantee boundary, and its killer feature — not just the marketing. The follow-up that separates further: "what breaks when you deploy new workflow code?" If you reach for versioning/patching and additive-only changes without prompting, you have demonstrated you have actually operated one of these, not just read about it.

Closing takeaways

  1. The workflow must be deterministic — that is the whole model in five words; time, randomness, and I/O live in activities or the context.
  2. Exactly-once is for recorded effects; make activities idempotent to cover the at-least-once in-flight window. Durability plus idempotency, never durability alone.
  3. The dangerous failures are delayed — the clock bug and the versioning bug both detonate on replay, hours or days after the code that caused them shipped.
  4. Signals make "wait for a human" free — no thread, no poll, no held connection; this is the feature enterprise agents (money, production, approvals) actually need.
  5. Know the spectrum — cron+DB state machine, Step Functions, event-driven saga, code-native Temporal — and place the workload deliberately. Defaulting to the heaviest tool is a smell.
  6. The agent is just an orchestration of durable, idempotent activities, and the LLM is one of them. Say that sentence and you have reframed "agent reliability" as the distributed-systems problem it actually is.