« Phase 08 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 08 — Core Contributor Notes: Durable Agent Execution

This is the maintainer's-eye view of the real system the lab mirrors: Temporal (and its lineage — it grew out of Uber's Cadence, which grew out of AWS Simple Workflow). If you were reading the Temporal server and SDK source, here is what you would find under the hood, where it differs from the lab's dataclass-and-cursor model, and which simplifications the lab makes on purpose.

Commands vs events: the two vocabularies

The lab conflates two things the real system keeps strictly separate. In the lab, call_activity both decides to run an activity and records the result into WorkflowState.commands. Temporal splits this across a client/server boundary:

• Commands are what the SDK (your worker, running your workflow code) emits. When your workflow calls ExecuteActivity, the SDK does not run anything — it produces a ScheduleActivityTask command and adds it to a list. The workflow function runs to the point where it can make no more progress without new input, then hands the server a batch of commands.
• Events are what the server records into history in response to commands. The server turns ScheduleActivityTask into an ActivityTaskScheduled event, later ActivityTaskStarted, then ActivityTaskCompleted (or ...Failed/...TimedOut). The history is a log of events, not commands.

So the real replay is: server sends the worker the event history → SDK re-runs the workflow → workflow re-issues commands → SDK matches each new command against the corresponding recorded event to decide "replay the recorded result" vs "this is new, tell the server." The lab's single commands list is really a fusion of the SDK's command buffer and the server's event history. When you read historyEvent types in the Temporal source (EVENT_TYPE_WORKFLOW_EXECUTION_STARTED, ACTIVITY_TASK_SCHEDULED, TIMER_STARTED, WORKFLOW_EXECUTION_SIGNALED, MARKER_RECORDED), you are looking at the thing the lab compresses into CommandResult and Signal.

Workflow tasks vs activity tasks

The lab has one kind of thing: a function call. Temporal has two distinct task types on two distinct queues, and the distinction is the core of the execution model:

• A workflow task is "advance this workflow's code by replaying its history and collecting the next batch of commands." It is short, deterministic, side-effect-free, and CPU-bound. When you signal a workflow or an activity completes, the server schedules a new workflow task so the code can react.
• An activity task is "actually go do the side-effecting work." It runs your activity function, can take arbitrarily long, can heartbeat, and is retried by the server per a retry policy.

This is why "the workflow orchestrates, activities do the work" is not a metaphor — they are physically different task types with different queues, different timeouts, and different determinism rules. The lab's call_activity fuses "issue a workflow command" and "execute the activity" into one synchronous call; Temporal decouples them across the matching service so the activity can run on a different worker, at a different time, with its own retry lifecycle.

Blocking without blocking: coroutines, not exceptions

The lab suspends a workflow by raising WorkflowBlocked and having the engine catch it. That is a clean pedagogical trick, but it is not how a real SDK parks a workflow, and the difference matters. Real SDKs run the workflow on a cooperative-scheduled coroutine runtime. When your workflow awaits an activity or a signal, it yields — the coroutine parks, the SDK collects the commands produced so far, completes the workflow task, and the worker moves on. There is no exception and no lost stack; the coroutine's state is implicit in "we replay to this await point next time."

Because the runtime is a custom deterministic scheduler, the SDKs also intercept the primitives the lab forbids by convention. In the Go, Java, TypeScript, and Python SDKs, workflow.Now(), workflow.NewTimer, workflow.SideEffect, and the SDK's random are all deterministic replacements for the language's native versions — the runtime feeds them recorded values on replay. The lab's ctx.now() is exactly this pattern in miniature; the real thing extends it to timers, UUIDs, sleep, and even a deadlock detector that trips if a workflow task runs too long (a sign someone did blocking I/O or a time.sleep inside workflow code).

The determinism checker is a diff, and versioning is how you live with it

The lab's guard compares rec.name to the issued name at one position. The real NonDeterministicError is generated by the SDK diffing the stream of commands the replayed code produces against the stream of events in history: a ScheduleActivityTask where history has a StartTimer, a different activity type, a different count. It is non-retryable by design — replaying the same bad code reproduces it.

The part the lab cannot teach because it is single-process: how you change workflow code without breaking in-flight executions. The real answer is the patching API — GetVersion (Go/Java) / patched and deprecatePatch (TS/Python). It writes a MARKER_RECORDED event the first time an execution reaches the change, so old executions replay down the old branch and new ones take the new branch; the marker is what makes the branch choice itself deterministic and durable. Temporal later layered Worker Versioning / Build IDs on top so you can pin executions to compatible worker builds and drain old code. A committer's mental model: "you can never edit the past, you can only record a marker that says which future this execution chose." Every real durable-execution bug report that starts with "it worked, then we deployed" ends in this area.

Markers, local activities, and the optimizations the lab omits

• SideEffect / mutable-side-effect record a one-off non-deterministic value (a UUID, a config read) as a MARKER_RECORDED event without the full cost of an activity round-trip. The lab's now() is a hand-rolled version of exactly this marker mechanism.
• Local activities run in the workflow worker process and record a single marker instead of the full Scheduled/Started/Completed event trio — an optimization for short, low-value calls where the task-queue round-trip would dominate. It trades some of the activity's independent retry/timeout semantics for latency.
• Sticky execution keeps a workflow's replayed state cached on one worker so the next workflow task does not replay the entire history from scratch — it applies only the new events. Without it, every workflow task is a full O(history) replay. The lab always does a full replay because it has no cache; production could not afford that.

What our miniature deliberately simplifies

• No separate services. The lab is one process; Temporal is frontend + history + matching + workers, with the durable state in Cassandra/SQL and executions sharded across history nodes.
• State is a pickled dataclass, not sharded replicated storage. WorkflowState.copy() stands in for "persist the history durably and replicate it."
• Exceptions, not a coroutine scheduler. WorkflowBlocked is a stand-in for cooperative yielding; there is no real await, no timers-as-events, no heartbeats.
• Fused command/event model. One commands list instead of SDK-commands vs server-events.
• Retries are a naive in-loop counter. No retry policy object (initial interval, backoff coefficient, max interval, max attempts, non-retryable error types), no server-driven scheduling of the next attempt, no jitter, no per-activity timeouts (schedule-to-start, start-to-close, heartbeat).
• No continue-as-new, no history cap, no query/update, no child workflows, no cron schedules, no visibility store. Each of these is a substantial subsystem in the real product.

None of these omissions are wrong — they are the difference between "understand the mechanism" and "operate the platform." When you read the Temporal source, map each subsystem back to the lab: the history store is commands, the deterministic runtime is Context, the NonDeterministicError is the two-line guard, and the patching API is the thing the lab could not show you because it never had a second deploy.