« Phase 08 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 08 Warmup — Durable Agent Execution

Who this is for: you did the earlier phases and can write Python. You've never built a system that survives a crash and resumes. By the end you'll have built a miniature Temporal — event sourcing, deterministic replay, retries, and human-approval signals — and you'll understand the one constraint (determinism) that the whole model rests on.

Table of Contents

  1. Why a for-loop agent dies
  2. Durable execution: the core idea
  3. Workflows vs activities
  4. Event sourcing: state is a log
  5. Deterministic replay: fast-forward through history
  6. The determinism constraint (and the #1 bug)
  7. Memoized activities = crash-safety
  8. Retries, backoff, and idempotency
  9. Signals: durable human-in-the-loop
  10. Sagas and compensation
  11. Why agents especially need this
  12. Common misconceptions
  13. Lab walkthrough
  14. Success criteria
  15. Interview Q&A
  16. References

1. Why a for-loop agent dies

Picture the agent from Phase 01: while not done: proposal = model(scratchpad); execute. It lives entirely in a process's memory — the scratchpad, the plan, the partial results. Now the process dies: a deploy rolls the pod, the machine is preempted, an OOM kills it, the region blips. Everything in memory is gone. If the agent had already spent $3 on tool calls and was about to send the final answer, you've lost the $3 and the answer, and on restart it begins from scratch (possibly re-doing side effects — re-charging a card, re-sending an email).

For a two-second chatbot turn, who cares — retry the whole thing. For an agent that runs for minutes, calls many tools, or waits for a human for hours, this is fatal. The longer and more side-effecting the agent, the more you need it to survive interruption. That is durable execution.

2. Durable execution: the core idea

Durable execution makes a program resume exactly where it stopped after a crash, as if the crash never happened. The trick is not "checkpoint all the variables" (fragile, and doesn't help with in-flight side effects). It's subtler and more robust:

Record every side effect with its result. To recover, re-run the program from the top, but replace each recorded side effect with its saved result instead of doing it again.

The program replays its own history — deterministically re-deriving all its local variables and control flow — until it reaches the point past the last recorded event, where it continues live. From the program's perspective it never stopped. This is the model behind Temporal, Azure Durable Functions, AWS Step Functions, DBOS, and Vercel's Workflow DevKit, and it's what you build in the lab.

3. Workflows vs activities

The model splits your code into two kinds, and the split is the whole discipline:

  • Workflow — the orchestration. It decides what to do in what order: call this, wait for that, branch on the result. Workflow code must be deterministic — given the same history it must make the same decisions, because it will be replayed. It does no I/O directly.
  • Activity — a single side-effecting step: call an API, write a DB row, charge a card. An activity is where non-determinism and the outside world live. Activities are recorded (their results saved) and retried on failure.

Think of the workflow as a conductor who never touches an instrument and the activities as the musicians. The conductor can be asked to "start the piece over from bar 40" (replay) and will wave the baton identically; the musicians already played bars 1–39 and don't replay them (their "notes" are recorded). In the lab, the workflow is a plain Python function that receives a Context; every effect goes through ctx.call_activity(...), ctx.now(), or ctx.wait_signal(...).

4. Event sourcing: state is a log

Event sourcing means you don't store current state directly — you store the sequence of events that produced it, and state is a function of the log. For a workflow, the log (history) is the ordered list of: activity results, timer fires, and signals received. To know the workflow's state, you replay the log.

Why store events instead of state? Because events are an append-only, complete record. You can reconstruct state at any point, audit exactly what happened, and — crucially — resume by replaying. It's the same idea behind a database's write-ahead log and Kafka's log-as-source-of- truth. In the lab, WorkflowState holds commands (recorded activity/now results in issue order) and signals (delivered external events) — that's the history.

5. Deterministic replay: fast-forward through history

Here's replay concretely. The engine calls your workflow function. Each time the workflow issues a command (say the i-th call_activity), the engine checks the history:

  • If position i is already recorded → return the recorded result without running the activity. (Replay: fast-forward.)
  • If position i is not recorded → run the activity for real, append its result to history, and return it. (Live: new work.)

So on a fresh run, everything is "not recorded" and runs live, filling the history. After a crash, you reload the history and call the workflow again: the first k commands replay instantly from history (no side effects), and command k+1 — the one that hadn't finished — runs live. The workflow re-derived all its local state by re-executing its deterministic logic, and it continues seamlessly. The correlation is by issue order, which is why the workflow must issue commands in the same order every time — i.e., be deterministic.

6. The determinism constraint (and the #1 bug)

This is the crux, and the most common interview question about durable execution. Workflow code must be deterministic: replayed with the same history, it must produce the same sequence of commands and the same result. That forbids, in workflow code:

  • Wall-clock time (time.time(), datetime.now()) — it returns a different value on replay, so any branch on it diverges. Use ctx.now(), which records the time on first read and replays it. This is the #1 durability bug: a workflow that reads the clock directly "works" until it replays, then computes a different value and corrupts its state.
  • Randomness (random.random(), UUIDs) — same problem. Use a recorded/seeded source.
  • Direct I/O (network, disk, DB) — non-deterministic and un-replayable. Put it in an activity.
  • Iterating a dict/set with unstable order, reading env/config that changes, threads/timing.

Temporal ships a determinism checker that detects when a replay diverges from history and fails loudly rather than silently corrupting. The lab implements a miniature of it: if the workflow issues a different command than history recorded at that position, call_activity raises a non-determinism error and the run fails with a clear message — far better than silent corruption. A test asserts this. Say "the workflow must be deterministic; side effects and time go through the context or activities" in an interview and you've shown you understand the model.

7. Memoized activities = crash-safety

The payoff of memoization: recorded activities are never re-executed on replay, so a crash cannot cause a double side effect. If the workflow charged a card (an activity) and then crashed, recovery replays the charge from history — it does not charge again. No double charges, no double emails, no double refunds. The lab's headline test proves it: an activity increments a side-effect counter; running to completion moves it once per activity; replaying the completed history moves it zero more times. That single property — exactly-once side effects across crashes — is why banks and payment systems (Citi!) reach for durable execution.

(Subtlety: an activity that crashes mid-execution, before its result is recorded, will be retried — so activities should be idempotent to be safe under at-least-once execution. That's §8.)

8. Retries, backoff, and idempotency

Activities talk to the flaky outside world, so they retry. The engine runs an activity, and on failure retries up to a limit with backoff (waiting longer between attempts to avoid hammering a struggling dependency). This is Phase 00's retry lever (1-(1-p)^(r+1)) made operational — retries turn a flaky activity into a reliable one.

The catch, from Phase 00: retries are only safe for idempotent activities — ones that produce the same effect whether run once or three times. "Charge card with idempotency key abc" is safe to retry (the payment processor dedupes); "charge card" without a key is not (three retries = three charges). Making activities idempotent (idempotency keys, upserts, dedupe) is core durable- execution craft. In the lab, call_activity retries with a simulated backoff (consulting the injected clock so time advances deterministically) and records only the final outcome.

9. Signals: durable human-in-the-loop

The most powerful feature for agents: a workflow can wait for an external event — a human approval, a webhook, another service — for as long as it takes, holding no resources while it waits. That event is a signal.

When the workflow calls ctx.wait_signal("approval") and no such signal has arrived, it suspends: the engine catches a WorkflowBlocked, persists the state, and returns a "blocked" result. The process can now do other work or shut down entirely. Hours or days later, someone delivers the signal (engine.signal(state, "approval", {...})) and re-runs the workflow; replay fast-forwards to the wait point, the signal is now in history, and the workflow continues. This is how durable human-in-the-loop works (Phase 10): the agent pauses for approval without a thread blocked, a connection held, or a dollar spent. The lab implements signals with per-name, in-order consumption and a suspend/resume test.

10. Sagas and compensation

When a multi-step workflow fails partway — three activities succeeded, the fourth failed permanently — you often can't just stop; you must undo the completed steps. A saga is a sequence of steps each paired with a compensating action (book flight → cancel flight; charge card → refund card). On failure, the workflow runs the compensations for the completed steps in reverse order, leaving the system consistent. Durable execution makes sagas practical because the history tells you exactly which steps completed and thus which to compensate. (The lab's engine supports this pattern via a workflow that catches ActivityError and runs compensating activities — a suggested extension.)

11. Why agents especially need this

Everything about agents amplifies the need for durability:

  • They're long-running — planning + many tool calls + waits can span minutes to days.
  • They spend money per step — losing a run loses real tokens and tool costs.
  • They have side effects — sending emails, moving money, editing files; double-execution is a real incident, not a cosmetic one.
  • They wait for humans — approvals on high-impact actions (Phase 10) are inherently long pauses.
  • They're unreliable per step (Phase 00) — so retries and checkpoints, which durability provides for free, are exactly the reliability levers you need.

This is why Temporal built an "AI Foundations" team to connect durable execution to agent frameworks (Pydantic AI, OpenAI Agents SDK, ADK), and why "durable agent" is becoming a standard architecture. A raw agent loop is a demo; a durable agent is a product.

12. Common misconceptions

  • "Durability means checkpointing all my variables." No — it means recording side effects and replaying deterministic logic to re-derive the variables. Replay is the mechanism.
  • "I can read the clock in workflow code." That's the #1 bug — it breaks replay. Use ctx.now().
  • "Retries make everything safe." Only for idempotent activities; otherwise retries multiply side effects.
  • "The workflow does the work." The workflow orchestrates; activities do the work and hold all the I/O and non-determinism.
  • "A blocked workflow holds a thread/resources." A well-designed durable wait holds nothing — it's persisted and re-run on the signal.

13. Lab walkthrough

Open lab-01-durable-workflow-engine/ and fill the TODOs:

  1. Context.call_activity — replay a recorded result at position i (no re-run); else execute with retries + backoff and record. Detect non-determinism (recorded command ≠ issued).
  2. Context.now — record on first read, replay thereafter.
  3. Context.wait_signal — consume in-order per name, or raise WorkflowBlocked.
  4. DurableEngine.run — copy state, build the context, run the workflow, and map WorkflowBlocked→"blocked", ActivityError/RuntimeError→"failed", else "completed".
  5. DurableEngine.signal — append a Signal to a copy of the state.

Run LAB_MODULE=solution pytest -v first, then match it. Read solution.py's main(): it runs a refund workflow that blocks on approval, resumes, and then replays with zero new side effects.

14. Success criteria

  • You can explain event sourcing + replay and why activities are memoized (no double effects).
  • You can state the determinism constraint and what breaks without ctx.now().
  • Your engine suspends on a missing signal and resumes when it's delivered.
  • Retries succeed a flaky activity and exhaustion fails cleanly; a workflow can compensate.
  • The non-determinism guard fires on a changed replay.
  • All 14 tests pass under lab and solution.

15. Interview Q&A

Q: How does durable execution survive a crash? A: Event sourcing plus deterministic replay. Every side effect is an activity whose result is recorded to a persistent history. On recovery the engine re-runs the workflow function; each recorded activity returns its saved result instead of executing again, so the workflow fast-forwards through everything it already did — re-deriving its local state deterministically — and continues from the point past the last recorded event. From the workflow's view it never stopped.

Q: Why must workflow code be deterministic, and what's the classic bug? A: Because it's replayed, and replay must reproduce the same commands and result to line up with history. Reading the wall clock (or random, or doing I/O) in workflow code is the classic bug: it returns a different value on replay, so a branch diverges and state corrupts. You put time behind ctx.now() (recorded once, replayed after) and side effects in activities. Temporal's determinism checker catches divergences; my lab has a miniature that fails the run on a non-deterministic replay.

Q: How do you avoid double-charging a card if the process crashes mid-workflow? A: The charge is an activity, recorded once with its result. On replay it's memoized — returned from history, not re-executed — so no second charge. For an activity that crashes before recording its result, it's retried, so I make it idempotent (an idempotency key the processor dedupes on) so the retry is safe. Recorded → exactly once; in-flight → at-least-once + idempotency.

Q: How does a workflow wait for a human approval for two days? A: A signal. The workflow calls wait_signal("approval"); if it hasn't arrived it suspends — the engine persists the state and the process holds no resources. When the human approves, you deliver the signal and re-run; replay fast-forwards to the wait, the signal is now in history, and the workflow continues. Durable, resource-free, resumable across restarts.

Q: Workflow vs activity — where does each piece of my agent go? A: The orchestration — decide the next tool, branch on results, the loop — is the workflow, and must be deterministic. Every actual tool call, model call, DB write, email — anything with a side effect or non-determinism — is an activity, recorded and retried. If I'm unsure, the test is "is it a pure decision (workflow) or an effect on the world (activity)?"

16. References

  • Temporal — docs & the durable-execution model. https://docs.temporal.io/
  • Temdal/Temporal blog — What is durable execution? https://temporal.io/blog
  • Pat Helland, Life Beyond Distributed Transactions (sagas / compensation).
  • Fowler, Event Sourcing. https://martinfowler.com/eaaDev/EventSourcing.html
  • Garcia-Molina & Salem, Sagas (1987) — the original saga paper.
  • Azure Durable Functions (the same replay model). https://learn.microsoft.com/azure/azure-functions/durable/
  • DBOS & Vercel Workflow DevKit (modern durable-execution takes). https://www.dbos.dev/ · https://vercel.com/docs/workflow