« Phase 08 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 08 — Principal Deep Dive: Durable Agent Execution

The lab is an in-process, single-threaded miniature. The WorkflowState is a dataclass you pass by value; "crash and recover" is you calling engine.run(..., state=r.state) again. That is the right teaching artifact, but it hides the architecture that makes durable execution a platform. This doc is about that architecture: what a real durable-execution system looks like, where it scales, where it breaks, and which of the lab's simplifications become the hardest problems in production.

The service decomposition

A production durable-execution system (Temporal is the canonical shape) is not a library — it is a stateful cluster plus stateless workers. Four responsibilities separate cleanly:

• The workflow code runs in your worker processes. Workers are stateless and horizontally scalable; they hold no durable state, only a cache of in-flight executions. This is where the lab's Context and workflow function live.
• The history service is the durable core. It owns the append-only event history per workflow execution, persists it (Cassandra, an RDBMS, or the like), and is sharded so no single node owns all executions. This is the lab's WorkflowState, promoted to a replicated, sharded database.
• The matching service is the queue broker. Workers long-poll task queues; the matching service hands a workflow task or activity task to whichever worker polls. This is the piece the lab has no analog for — the lab calls the workflow function directly.
• The frontend is the API gateway: start-workflow, signal, query, describe.

The load-bearing consequence of this split: the durable state and the code that mutates it live in different processes, connected only by a log. A worker can die, be deployed over, or be scaled to zero, and the history is untouched. Recovery is "some worker picks up the task queue and replays the history." The lab collapses all four services into one call stack, which is exactly why it looks so simple — and why the interesting failure modes are invisible in it.

The event-history budget is the master constraint

Every design decision downstream flows from one number: the size of a single workflow's event history. History is replayed on every recovery and every time a worker without a warm cache picks up the execution, so history length is directly a latency and memory cost. Real systems impose hard caps (on the order of tens of thousands of events, and a payload ceiling measured in low megabytes).

This is why the shape of a durable workflow is constrained in ways the lab never forces you to feel:

• Continue-as-new. A workflow that would loop forever (a per-tenant agent that runs indefinitely, a nightly-batch driver) must periodically truncate its own history by atomically finishing the current run and starting a fresh execution with carried-over state. In the lab, a workflow that issued a million commands would build a million-element commands list and replay all of them; in production you would have blown the history cap long before. Agents are especially exposed here: a long-horizon agent that calls a tool every few seconds for hours accumulates history fast. "How does your durable agent bound its history?" is a real system-design question and the answer is continue-as-new at a sensible checkpoint (per task, per N steps).

• Payloads are events too. Recording a 2 MB tool result into history N times is how you turn a working agent into an OOMing one. The discipline is to store large blobs out-of-band (object store) and keep a reference in history. The lab records value directly into CommandResult; fine for 700, catastrophic for a 50k-token model response replayed on every recovery.

Where the bodies are buried: determinism across deploys

The lab's non-determinism guard fires when the code changes between two runs of the same process. In production the code changes between runs because you deployed a new version while workflows were mid-flight. A refund workflow that started last Tuesday is still parked on an approval signal when you ship a new build that inserts a step before the charge. On the next replay, the new code issues a command sequence that no longer matches the week-old history → NonDeterministicError on a workflow that was working fine.

This is the single most important operational fact about durable execution, and it is invisible in a single-process lab. The mitigations are architectural:

• Versioning / patching. Real SDKs give you a GetVersion/patched primitive: the workflow records which branch it took the first time, so old executions keep taking the old branch and new executions take the new one. Changes are gated behind a version marker rather than edited in place.
• Worker versioning / build IDs. Pin an execution to the worker build that started it, and drain old builds gracefully rather than forcing week-old histories onto new code.
• Additive-only discipline. The safe change is appending steps at the end; reordering or removing steps in the middle is the dangerous one.

The blast radius of getting this wrong is large and delayed: nothing breaks at deploy time; it breaks hours or days later when parked workflows wake up, and it breaks per-execution, so you get a slow drip of stuck workflows rather than a clean outage. That delayed, partial failure mode is why this belongs in a principal-level discussion.

Failure modes and their blast radius

• Poison workflow task. A workflow that deterministically throws (a bug, or a NonDeterministicError) will be retried forever by the platform and never make progress. Blast radius: one execution stuck, plus retry load. Detection is a "workflow task failing repeatedly" metric; remediation is reset-to-a-prior-event or a code fix + redeploy.
• Activity retry storm. A downstream dependency is down, so thousands of activities across thousands of workflows all retry with backoff simultaneously. Blast radius: you can DDoS your own recovering dependency. Mitigations: jittered backoff, retry caps, and circuit-breaking at the activity level — the lab's fixed retry loop has none of these.
• Signal to a completed/terminated workflow. The human approves after the workflow already timed out and compensated. Blast radius: a lost signal or a resurrection bug. Real systems make this an explicit error; your design must decide the policy.
• History corruption / cap breach. Rare but fatal — the execution can no longer make progress. This is why continue-as-new and payload hygiene are not optional at scale.

Cross-cutting concerns

Multi-tenancy. Real platforms isolate tenants with namespaces: separate task queues, separate retention, separate rate limits, separate authz. A noisy tenant's retry storm should not starve another tenant's workflows. The lab has no notion of tenant; a platform team building "durable agents as a service" (Phase 13, Phase 17) spends most of its time here.

Cost. The dominant cost of a durable agent is usually not the durability engine — it is the model and tool calls, which are activities. Durability's own cost is storage (history) plus the replay compute on recovery. The napkin math worth internalizing: replay cost per recovery scales with history length, and recoveries happen on every worker cache miss, so a hot workflow bouncing across cold workers pays replay repeatedly. Sticky execution (keep an execution on the worker that has its warm cache) is the optimization that makes this affordable.

Observability. The event history is the audit log — you get "exactly what happened, in order, forever" for free, which is a genuine gift for agent debugging and compliance (Citi). The production add-on is correlating that history with traces and spans (Phase 14) so a single agent run is legible across the model calls, tool calls, and durable steps.

Security. Durable state is durable — a recorded tool result containing a secret or PII is now persisted, replayed, and retained. Payload encryption (a codec that encrypts before the event hits the history store) is the standard control, and it interacts with the "store big blobs out of band" rule: you are encrypting references, not just values.

The one-sentence architecture

A durable execution platform is a replicated, sharded, append-only log service (history) fronted by a queue broker (matching), driven by stateless, horizontally-scaled workers that replay the log to reconstruct state — and every hard problem (history caps, cross-deploy determinism, retry storms, multi-tenant isolation, payload hygiene) is a consequence of the log being the source of truth and the code being separable from it. The lab teaches the log-and-replay core honestly; production is that core plus everything required to run it for thousands of tenants and survive your own deploys.