« Phase 01 · Warmup · Track Overview
Core Contributor Notes — How the Real Runtimes Do This
How LangGraph, Google ADK, AWS Bedrock AgentCore, the OpenAI Agents SDK and Temporal implement the mechanisms in this phase — the non-obvious decisions, the sharp edges, and what our miniature simplifies.
Table of Contents
- 1. LangGraph: the graph is the state machine
- 2. Checkpointers and the resume contract
- 3. Interrupts: how HITL is actually implemented
- 4. ADK: state scopes as a first-class idea
- 5. AgentCore: isolation as the product
- 6. Temporal: what "durable" actually costs
- 7. Consistent hashing in the mesh
- 8. Sharp edges
- 9. What the miniature simplifies
- 10. References
1. LangGraph: the graph is the state machine
Our TRANSITIONS table is explicit. LangGraph's is implicit in the graph topology: you declare
nodes and edges, and the runtime executes them in super-steps (a Pregel-style
bulk-synchronous-parallel model — all nodes scheduled in a step run, then the state is merged,
then the next step is scheduled).
The consequences are worth understanding because they explain most LangGraph behaviour that surprises people:
- State is a typed dict with reducers. Each key declares how concurrent writes merge
(
operator.addfor message lists, last-write for scalars). This exists because two nodes in the same super-step can both write. Our kernel is single-threaded, so it needs no reducers — and that is exactly what makes parallel tool calls a non-trivial extension rather than a loop tweak. recursion_limitis the step budget, and it counts super-steps, not tool calls. A graph with a fan-out node burns one super-step for many calls. Teams set it as if it were a tool-call count and are surprised.- Conditional edges are the transition table. A routing function returns the next node name. Nothing prevents a routing function returning a node that makes no sense from the current state — the graph has no notion of "illegal from here." That is precisely the guarantee our explicit table buys, and it is why building the table once by hand is worth doing even if you then adopt a graph runtime.
2. Checkpointers and the resume contract
LangGraph's BaseCheckpointSaver (with MemorySaver, SqliteSaver, PostgresSaver
implementations) is our SessionStore. The interface is richer in two ways that matter:
It stores pending tasks, not just state. A checkpoint records the channel values and the tasks scheduled but not yet executed. That is what allows resumption mid-super-step rather than only at step boundaries. Our kernel resumes at step boundaries, which is why a crash during a tool call re-dispatches.
Checkpoints form a chain, and you can branch it. Each has a parent_config, so
get_state_history() walks backwards and update_state() forks a new branch from an old
checkpoint. This is "time travel," and it is genuinely useful in production for two things people
underuse: replaying a bad run with a fixed prompt, and letting a reviewer edit an agent's proposed
action before resuming. Our snapshot has a linear version counter, which cannot branch.
The thread_id is the session id, and checkpoint_ns namespaces sub-graphs. The important
detail: the thread is the concurrency unit, and LangGraph does not provide cross-writer
conflict detection out of the box the way our CAS does — two concurrent invocations on one
thread_id interleave writes into the same channels. Production deployments serialize per thread
themselves (a queue, a lease, or a database lock). If you take one thing from our lab into a
LangGraph deployment, take the version check.
3. Interrupts: how HITL is actually implemented
Our ask decision transitions to WAITING_INPUT and returns. LangGraph's interrupt() does
something more surprising: it raises a special exception inside the node, the runtime
checkpoints, and the invocation returns with an __interrupt__ payload. On resume with
Command(resume=value), the node is re-executed from the top, and the interrupt() call
returns the supplied value instead of raising.
The sharp edge follows immediately and bites everyone once: any side effect before the
interrupt() call in that node happens twice. The rule is to put interrupt() at the top of
the node, or to isolate side effects in their own node. Our kernel avoids this by making the
pause a state transition rather than a re-executed function — a simpler model that costs the
ability to pause mid-node.
interrupt_before / interrupt_after on node names are the static version: pause at a named
boundary regardless of the agent's decision. For a bank, that static form is often what you want
for money-moving nodes, because it does not depend on the model choosing to ask.
4. ADK: state scopes as a first-class idea
Google's Agent Development Kit gets one thing very right that most runtimes leave to the developer: state keys carry a scope prefix.
| Prefix | Meaning |
|---|---|
| (none) | session-scoped: this conversation |
user: | this user, across sessions |
app: | the whole application |
temp: | this invocation only, never persisted |
SessionService (in-memory, database, or Vertex AI managed) enforces the persistence behaviour
per prefix. This is our SemanticMemory scope, promoted into the state API itself — so a
developer writing user:language_preference has already made the partition decision, rather
than deciding later where a fact belongs.
Two lessons for a platform design:
- Make the partition syntactically unavoidable. Our lab requires a
scopeandowneron everyFact; ADK requires a prefix. Both beat an API where the scope is an optional argument. temp:is underrated. An explicit "this is scratch, never persist it" scope prevents a large class of accidental data retention, which in a bank is a compliance question, not a storage one.
ADK also exposes output_key on an agent, which writes its final response into session state
under a name — the mechanism that makes SequentialAgent/ParallelAgent/LoopAgent composition
work without bespoke glue.
5. AgentCore: isolation as the product
AWS Bedrock AgentCore Runtime's headline property is that each session gets its own microVM, with dedicated CPU, memory and filesystem, torn down when the session ends. Our lab's isolation is logical (a session id and a partition key); AgentCore's is physical.
This matters for a bank in a specific way: a code-executing agent (data analysis, document processing) is running model-generated code, and logical isolation is not a defensible control against it. The design consequence is that isolation strength should be a property of the agent class, not of the platform: conversational agents get logical isolation and share workers; code-executing agents get a sandbox per session. Building one kernel that supports both placements is a Phase 13 concern; recognizing that you need it is a Phase 01 concern.
AgentCore Memory splits short-term (raw session events) from long-term (extracted, consolidated strategies — semantic facts, user preferences, summaries), with extraction running asynchronously after a session. That asynchronous extraction is a pattern worth stealing: it keeps the hot path free of memory-write latency, and it gives you a natural place to run the guardrails and classification checks that memory writes need (see PRINCIPAL-DEEP-DIVE §6).
6. Temporal: what "durable" actually costs
Temporal-class durable execution is the strongest form of what our checkpointing gestures at. Workflow code is re-executed from the beginning on every resume, with completed activity results served from an event history instead of being re-run. The result is exactly-once activity execution semantics from the workflow's point of view.
The price is a determinism constraint on workflow code: no wall clock, no random, no direct I/O, no iteration over non-deterministic collections — because replay must produce the same sequence of commands. This is the same discipline our LAB-STANDARD imposes, and it is not a coincidence: a runtime that can replay is a runtime whose code is a pure function of its history.
Two things teams get wrong when they reach for it:
- Determinism applies to workflow code, not activity code. Activities may do anything; they are recorded by result. Putting agent logic in a workflow and model calls in activities is the correct split, and putting model calls in the workflow is the classic mistake.
- Versioning is the hard part. Changing workflow code changes the command sequence, which breaks replay for in-flight runs. Temporal's patching API exists for this. Any durable agent runtime inherits the problem: you cannot freely change an agent's graph while runs are in flight. For long-running banking workflows measured in days, this is a first-order operational constraint, not a footnote.
7. Consistent hashing in the mesh
You will rarely implement a ring in application code — Envoy already has one. Configure the load
balancing policy to RING_HASH or MAGLEV, and a hash policy on a header
(x-session-id) or cookie:
RING_HASHis the classic Karger ring;minimum_ring_sizeis ourvirtual_nodes(Envoy's default is large — thousands — because imbalance shrinks like \( 1/\sqrt{V} \)).MAGLEVbuilds a fixed-size lookup table instead of a sorted ring: \( O(1) \) lookup and better balance, at the cost of slightly more disruption on backend changes than a ring.- Draining is
HealthCheck+drain_connections_on_host_removal, plus the endpoint being markedDRAININGin EDS. Kubernetes surfaces this asterminationGracePeriodSecondsplus a readiness probe that starts failing before the pod stops — the same drain-then-remove discipline as ourdrain().
The application-side thing you still own: emitting a stable session header, and making sure it is present on every request including retries. A missing header falls back to round-robin, and the symptom is a mysterious cache-hit-rate cliff on some fraction of traffic.
8. Sharp edges
Salted hash(). Covered in the WARMUP; it belongs here too because it is a real bug in real
code. Python salts string hashing per process unless PYTHONHASHSEED is fixed. Never build a
ring, a shard key, or a stable id on it.
Message-history reducers are append-only by default. In LangGraph, add_messages appends. A
node that "replaces" history by returning a new list appends it instead, and the context silently
doubles. Trimming requires RemoveMessage, which people find only after a cost spike.
Checkpoint size grows with the run. Every checkpointer stores the full channel values. A
message list that grows to 100 k tokens is written on every super-step. PostgresSaver will do
this happily until your write throughput or your storage bill notices. Trim or summarize inside
the state, not just at render time.
Session TTLs are a correctness feature. Without one, waiting_input sessions accumulate
forever, holding memory in the store and skewing every "active sessions" metric. Every real
runtime has a session expiry; our lab does not, and adding one is a five-line extension with a
large operational payoff.
Re-entrancy on resume. Whatever the runtime, ask: if I resume, does anything before the pause run again? LangGraph: yes, the whole node. Temporal: no, activities are replayed from history. Ours: no, but a crash mid-dispatch re-dispatches. The answer determines where side effects may safely live, and it is the first question to ask of any agent runtime.
9. What the miniature simplifies
| Miniature | Reality |
|---|---|
| Explicit transition table | implicit in a graph topology; no notion of "illegal from here" |
| Single-threaded loop | super-steps with parallel nodes and state reducers |
| Snapshot with a linear version | checkpoint chains with parents, history and branching (time travel) |
| Pause as a state transition | interrupt() raising inside a node, with node re-execution on resume |
SemanticMemory scopes | ADK state prefixes, AgentCore memory strategies with async extraction |
| Logical session isolation | microVM-per-session for code-executing agents |
| Checkpoint after observation | pending-task checkpoints, or full event-history replay |
| Ring in application code | Envoy RING_HASH/MAGLEV with EDS draining |
| No session TTL | expiry, archival and retention policy |
| Deterministic summarizer | a model call, tuned against a quality eval |
The mechanisms are the same; the reasoning transfers directly. What the real runtimes add is concurrency, durability and scale — and each of those additions brings the sharp edge listed above, which you can now recognize rather than discover.
10. References
- LangGraph — persistence and checkpointers,
interrupt()andCommand(resume=...),recursion_limit,get_state_history()/update_state(),add_messagesandRemoveMessage. - Google ADK —
SessionService, state scopes (user:,app:,temp:),output_key, workflow agents, the callback chain. - AWS Bedrock AgentCore — Runtime session isolation (microVM per session), Memory short-term vs long-term strategies, Gateway and Identity.
- OpenAI Agents SDK —
Runner, sessions (including the SQLite session store), handoffs, guardrails and tripwires. - Temporal — durable execution, determinism constraints on workflow code, activity replay, workflow versioning/patching.
- Envoy —
RING_HASHandMAGLEVload balancers, hash policies, endpoint draining. - Karger et al., Consistent Hashing and Random Trees, STOC 1997; DeCandia et al., Dynamo, SOSP 2007; Google, Maglev, NSDI 2016.