« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 08 — Durable Agent Execution: Workflows, Retries & Crash-Safety (Temporal-class)

Answers these JD lines: Temporal's entire "AI Foundations / durable execution" role — "the durable execution model with AI frameworks (Pydantic AI, Vercel AI SDK, Google ADK, OpenAI Agents SDK)," "reliable agent execution," "long-running tasks," "retries, orchestration." Also Cohere/Citi "reliable execution of agent workflows" and every human-in-the-loop requirement.

Why this phase exists

Phases 00–07 built an agent that runs in a process. But an agent that plans for ten minutes, calls twenty tools, and waits a day for a human approval cannot live in a process — a deploy, a crash, or a preemption wipes it, and you've lost the plan, the partial results, and the money already spent on tool calls. Production agents run on durable execution: an engine that makes a long-running workflow survive process death and resume exactly where it stopped.

The mechanism is event sourcing + deterministic replay, and it's genuinely beautiful. Every side effect the workflow performs is an activity, recorded with its result in a persistent history. To recover, the engine re-runs the workflow function from the top — but each activity call returns its recorded result instead of executing again, so the workflow fast-forwards through everything it already did and continues from the crash point. For that to work the workflow code must be deterministic (no wall-clock, no random, no direct I/O) — those live in activities. This is exactly Temporal's model, and this phase builds a working miniature of it.

Concept map

• Workflow vs activity: the workflow is deterministic orchestration; activities are the side-effecting, retryable steps. The workflow coordinates; activities do.
• Event sourcing: persist a log of what happened (activity results, signals); state is a function of the log.
• Deterministic replay: re-run the workflow, feeding recorded results, to reconstruct state. Requires determinism — the #1 durability bug is reading the wall clock in workflow code.
• Memoized activities = crash-safety: recorded activities are not re-executed on replay, so no double charges / double emails.
• Retries + idempotency: activities retry with backoff; idempotency makes retries safe.
• Signals = durable human-in-the-loop: an external event that resumes a suspended workflow; the workflow parks holding no resources.

The lab

Integrated scenario

An agent processes a refund: look up the invoice, wait for a human to approve, then charge the card. Between "wait" and "approve" a deploy restarts the service. In a naive agent, the run is gone. In this phase's engine, the workflow was suspended on the approval signal holding no resources; when the human approves hours later, the engine replays the recorded invoice lookup (without re-calling it), delivers the signal, and charges the card exactly once. The customer is refunded once, the card is charged once, and no engineer got paged. That is what "reliable agent execution" means, and it's why Temporal pays $224k–$302k for it.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py.
• You can explain event sourcing + replay and why activities are memoized.
• You can explain the determinism constraint and what breaks without it.
• You can describe signals and durable human-in-the-loop.

Key takeaways

• Long-running agents need durable execution; a for-loop in a process is not it.
• The workflow must be deterministic; side effects live in activities that are recorded and memoized — that's what makes replay crash-safe.
• Retries buy reliability (Phase 00) but only for idempotent activities.
• Signals turn "wait for a human" into a durable, resource-free pause — the production shape of human-in-the-loop (Phase 10).