« Phase 08 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 08 — Deep Dive: Durable Agent Execution

The load-bearing idea of this whole phase is one sentence: workflow state is not stored, it is re-derived by replaying an append-only log of side-effect results through deterministic code. Everything else — memoization, crash-safety, exactly-once, durable waits — falls out of that as a consequence. This doc dissects the mechanism at the level of the actual data structures and control flow in solution.py, because the subtle bugs live in the mechanism, not in the concept.

The two logs and the cursor

The persisted state is deliberately tiny. WorkflowState holds two ordered lists:

commands: list[CommandResult] — the outcome of every deterministic command the workflow issued, in issue order. A CommandResult is (kind, name, ok, value, error) where kind is "activity" or "now". Note what is recorded: not the input arguments, only the position, the identity, and the result. That is the event log.
signals: list[Signal] — external events delivered from the outside world (name, payload), in delivery order.

The engine reconstructs state by handing this to a fresh Context and re-invoking the workflow function. The Context carries the machinery that makes replay work:

_cmd_index — a monotonically increasing cursor. Every deterministic command (call_activity, now) reads the current index, then increments it. This is the correlation key: the i-th command the workflow issues maps to commands[i].
_signal_cursor: dict[str, int] — per-name consumption counter for signals, so two wait_signal("approval") calls consume the first and second delivered approval respectively.
replayed_commands / executed_commands — bookkeeping the tests assert on, and the clearest window into whether a given call replayed (returned from history) or executed (touched the world).

The invariant that makes the entire scheme sound: the workflow must issue the same sequence of commands, in the same order, on every replay. The cursor correlates purely by position, so if the workflow issues activity:fetch_invoice where history recorded activity:charge_card, the positions no longer mean the same thing and the reconstructed state is garbage. This is why determinism is not a style guideline — it is the precondition for the correlation-by-index to be meaningful at all.

`call_activity`: the three-way branch

call_activity is where memoization, replay, retries, and the non-determinism guard all meet. Trace the branches:

i = self._cmd_index; self._cmd_index += 1
if i < len(self._state.commands):        # this slot is in history → REPLAY
    rec = self._state.commands[i]
    if rec.kind != "activity" or rec.name != name:
        raise RuntimeError("non-determinism at command i ...")   # the guard
    self.replayed_commands += 1
    return rec.value           if rec.ok else raise ActivityError(rec.error)
else:                                    # slot not in history → EXECUTE LIVE
    self.executed_commands += 1
    for attempt in range(max_retries):
        try: value = activity(*args); append CommandResult(ok=True, value); return value
        except: last_err = ...; self._clock()   # simulated backoff
    append CommandResult(ok=False, error="retries exhausted")
    raise ActivityError(...)

Three properties are worth naming precisely:

Memoization is defined by i < len(commands), nothing else. There is no timestamp, no TTL, no "is this fresh" check. If the slot exists, it replays; if it doesn't, it runs. That is the whole crash-safety guarantee: a recorded activity is structurally unable to run a second time, because the only path that invokes the real function is the else branch, and that branch only executes when the slot is absent.
Failures are recorded, not just successes. ok=False with an error string is written to history when retries exhaust. On replay, that slot re-raises ActivityError identically. This is the difference between "the activity hasn't run yet" and "the activity ran and permanently failed" — without recording failure, a replay would re-run a permanently-failing activity and possibly get a different answer, breaking determinism. A durable engine must memoize the failure verdict as carefully as it memoizes success.
Retries happen only in the live branch, and the result of retrying is a single recorded outcome. History records "fetch_invoice → 700", never "fetch_invoice attempt 1 failed, attempt 2 failed, attempt 3 → 700". The retry loop is an implementation detail of becoming durable; once recorded, the outcome is a single event. This matters: on replay you fast-forward past the whole retry saga in O(1), because it collapsed to one CommandResult.

`now()`: why the clock is a recorded command

ctx.now() uses the same cursor as call_activity. It reads commands[i]; if present and kind == "now", it returns the recorded value; if absent, it calls the injected _clock(), records the value, and returns it. This is the mechanically important point most people miss: time is not special — it is just another command occupying a slot in the same ordered log. The wall clock is non-deterministic, so it cannot be read in workflow code; instead the first read is captured as an event and every replay returns that captured event. time.time() in workflow code is the canonical durability bug precisely because it bypasses the cursor: it returns a fresh value on replay, diverges a branch, and the command sequence after that branch no longer lines up with history.

The non-determinism guard, mechanically

The guard is the two-line check if rec.kind != "activity" or rec.name != name. It fires when the command the workflow issues at position i disagrees with what history recorded there. This converts silent state corruption into a loud, positioned error ("non-determinism at command 3: history has activity:'charge_card', workflow issued activity:'fetch_invoice'"). The design choice in the engine is to map that RuntimeError to a failed result rather than let it crash — an operator sees corruption as a failed workflow with a diagnostic, not a stack trace in a log. Real engines call this a NonDeterministicError and treat it as non-retryable: retrying deterministically reproduces it, so the only fix is a human correcting the workflow code or versioning it.

A worked trace: the refund that survives a restart

Follow main()'s refund workflow across three engine invocations. The workflow: amount = call_activity("fetch_invoice", 7) → approval = wait_signal("approval") → ts = now() → receipt = call_activity("charge_card", -amount).

Run 1 (fresh, no state): fetch_invoice slot absent → executes → 700 recorded at commands[0]. wait_signal("approval"): _signal_cursor["approval"]=0, no matching signals → raises WorkflowBlocked. Engine catches it, returns status="blocked", blocked_on="approval", and persists state with one command. side_effects["charges"] == 0 — nothing was charged, and crucially no thread is parked; the process is free.
Signal + Run 2 (recovery): engine.signal(state, "approval", {approved:True}) appends a Signal to a copy of the state. Re-run: fetch_invoice slot commands[0] present → replays 700, does not re-call the activity (replayed_commands=1, charges still 0). wait_signal: cursor 0, one matching signal → returns {approved:True}, cursor→1. now() slot absent → records 1001. charge_card slot absent → executes → charges becomes 1, recorded. Returns completed.
Run 3 (replay the completed history — the crash-safety proof): every slot is present. fetch_invoice, now, charge_card all replay from history; wait_signal finds its signal. charges stays 1. The headline invariant made observable: replaying a completed history produces zero new side effects, because every real call fell into the replay branch.

Complexity, and why naive alternatives fail

Replay is O(k) in history length k: each of the first k commands is an O(1) list lookup, and only the (k+1)-th does real work. The cost model is "recovery is linear in what already happened," which is why real engines cap history size and offer continue-as-new to start a fresh history.

Two naive alternatives fail at the mechanism level, not merely in practice:

Checkpoint all variables. You cannot atomically checkpoint the moment between "card charged" and "charge recorded" — the in-flight side effect has no representation in a variable snapshot. Replay sidesteps this by never trying to snapshot mid-effect: the effect is either recorded (replayed) or not (re-run under idempotency). The unit of durability is the completed side effect, not the variable.
Just retry the whole job. Re-running a non-idempotent side-effecting job double-charges. The whole point of memoization is that the already-completed effects are excluded from the retry; only the tail past the last recorded event re-executes. Retrying the job is retrying with an empty history — exactly the thing durability prevents.

The one residual gap the mechanism cannot close by itself: an activity that crashes after its real side effect but before append(CommandResult) will be re-run (its slot is absent). That is the at-least-once boundary, and it is why activities — not workflows — must be idempotent. The log gives you exactly-once for recorded effects and at-least-once for in-flight effects; idempotency keys close the remaining gap.

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems