« Phase 01 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 01 — Deep Dive: The Agent Loop — ReAct, ReWOO & Plan-Execute-Replan

The load-bearing idea of this phase is that an agent is a state machine driven by an untrusted oracle, and every hard part is a consequence of that one sentence. The oracle (the model) can emit anything; the loop must transform "anything" into either progress or a bounded, recoverable failure. This document is about the machine — the state it carries, the transitions it allows, the invariants it never violates, and where the three modes diverge in their control and data flow.

The ReAct loop: state, transitions, invariants

The ReAct machine carries exactly one piece of mutable state: the scratchpad, a string that starts as f"Task: {task}" and grows by one Thought/Action/Action Input/Observation block per accepted step. There is no hidden state. Every decision the model makes is a pure function of that string, which is precisely why an injected Policy (a str -> str function) can stand in for the model and make the whole run deterministic.

The transition per iteration is: raw = policy(scratchpad); increment llm_calls; parsed = parse_react_step(raw); branch on the parse result. Three terminal outcomes and one continuation exist, and the guard bounds them all:

1. Continuation — parsed is an Action. Dispatch the tool, append the block, loop.
2. Final — parsed is a FinalAnswer. Set final_answer, stopped_reason = "final", return.
3. Parse error — parse_react_step raised ValueError. Record a PARSE_ERROR step, stopped_reason = "error", return.
4. Budget exhausted — the for _ in range(max_steps) falls through. stopped_reason = "max_steps", return a partial trace.

The termination invariant is the whole point: the loop always terminates in at most max_steps iterations, regardless of what the oracle emits. A model that proposes the same search forever does not hang the process — it hits case 4 at exactly max_steps. Remove that for-bound and case 4 becomes an infinite loop that pins a CPU and drains the token budget. The guard is not a tuning knob; it is the only thing standing between a confused oracle and a production incident.

The cost invariant is the second: llm_calls == steps. Because llm_calls is incremented once per iteration and each iteration is one policy(scratchpad) call, the counter equals the number of steps taken. This is asserted by a test so the cost model is not abstract. And because the entire scratchpad is resent every iteration — the model needs history to reason — the token cost is the sum of a growing prefix. If step (k) resends roughly (k) blocks, total input tokens over (n) steps scale as (\sum_{k=1}^{n} k = \tfrac{n(n+1)}{2}), i.e. (O(n^2)). That quadratic is not an implementation defect; it is inherent to resending an interleaved history, and it is exactly the number Phase 00 asked you to calculate.

Parsing across the trust boundary

parse_react_step is where untrusted text becomes structured data, and its contract is liberal-in, strict-out. Liberal: it tolerates missing Thought:, extra whitespace, multi-line inputs (the regexes use re.DOTALL where needed), and it takes only the first line of the Action: name via action_m.group(1).splitlines()[0]. Strict: what it returns is either a well-formed Action(thought, name, args) with args a real dict, or a well-formed FinalAnswer, or nothing at all — it raises ValueError. There is no third, half-parsed thing that leaks downstream.

The order of checks matters. Final Answer: is tested before Action:, so a step that contains both terminates rather than acting. If neither anchor is present, ValueError("no 'Action:' or 'Final Answer:'"). If the action has no input, ValueError. If the input is not JSON, json.JSONDecodeError is caught and re-raised as ValueError. If the JSON parses but is not an object (e.g. a bare list or number), ValueError("Action Input must be a JSON object"). Every malformed shape converges to one exception type, and the loop's except ValueError catches exactly that. This is why a garbage step becomes a PARSE_ERROR observation and not a stack trace: the boundary raises a typed, expected error, and the caller is written to expect it.

Errors as observations: dispatch never raises

ToolRegistry.dispatch is the second trust boundary, and it upholds a stronger contract than the parser: it never raises at all. An unknown name returns ToolError(name, "unknown tool ..."). Any exception inside the tool — TypeError from wrong kwargs, ZeroDivisionError, a network error in a real tool — is caught by except Exception and returned as ToolError(name, f"{type(e).__name__}: {e}"). The caller then does result.as_observation() if isinstance(result, ToolError) else str(result) and feeds the string back into the scratchpad.

The consequence is architectural. If a tool raised, the run's reliability would be capped at the reliability of its flakiest tool: one HTTP 429 kills the whole agent. By converting failure into an observation, a recoverable failure stops being a failure — the model sees ERROR[divide]: ZeroDivisionError: division by zero and can choose another path on the next step. The naive version (let tools raise) is not simpler; it is a different, worse machine whose termination and reliability properties are set by its dependencies rather than by its own design.

ReWOO: three modules, two LLM calls

ReWOO restructures the machine to kill the quadratic. The control flow is strictly sequential across three phases with no loop-back to the model:

1. Planner — one call, llm_calls += 1, produces JSON {"plan": [{"id","tool","args"}, ...]}, parsed by parse_plan into PlanSteps. The planner reasons without observation — it never sees a tool result.
2. Workers — a for step in plan loop. Per step: args = substitute(step.args, evidence); result = registry.dispatch(step.tool, args); evidence[step.id] = result. No policy call here at all.
3. Solver — one call over task plus the stringified evidence, llm_calls += 1, producing the answer.

The invariant is llm_calls == 2, independent of plan length — asserted against a 3-tool plan. Because the worker loop holds no model call, tokens are linear in tool count and, critically, steps with no #E dependency between them are data-parallel: their execution order is unconstrained, so a fan-out that ReAct runs as a latency sum ReWOO can run as a latency max.

Variable substitution: the type-preserving wire

substitute is the mechanism that lets the model leave the execution loop. For each arg value that is a string, it distinguishes two cases via _REF_RE.fullmatch(val):

• Exact reference — the entire value is "#E2". Return evidence["E2"] raw, preserving type. If E2 produced the int 6, the next tool receives the int 6, not the string "6". This is the single subtlety the lab forces you to get right: multiply wants numbers, and x * y on "6" silently produces "66" or a TypeError. Stringifying an exact reference is the classic substitution bug.
• Embedded reference — "#E1" inside a larger string. Here _REF_RE.sub(_rep, val) interpolates str(evidence[ref]), because you cannot splice a typed value into the middle of a string.

An unresolved reference (#E9 with no E9 in evidence) raises ValueError("unresolved reference #E9") — a structured error, not a KeyError leaking through. In ReWOO that aborts with stopped_reason = "error"; in plan-execute it triggers a replan.

Plan-execute-replan: bounded recovery

This mode keeps ReWOO's plan/execute structure but wraps it in while replans <= max_replans. Execution tracks failed_step; a SUBST_ERROR or a ToolError sets it and breaks the inner loop. If failed_step is None, the last evidence value is the answer and it returns. Otherwise replans += 1 and the context is rebuilt to include A previous plan FAILED at {failed_step} plus the evidence gathered so far, and the planner is called again. The invariant is llm_calls == 1 + replans: you pay for the first plan, and one more call per recovery, bounded so a permanently-failing task terminates with stopped_reason = "max_replans" rather than replanning forever.

Three worked traces of one task

Task: "How many words are in the France fact, doubled?" Tools: search, word_count, multiply. The fact has 8 words, so the answer is 16.

ReAct. Step 1: scratchpad has 0 Observation:s → Action: search {"q":"france"} → observation is the fact; llm_calls=1. Step 2: 1 observation → word_count on the fact → 8; llm_calls=2. Step 3: 2 observations → multiply {"x":8,"y":2} → 16; llm_calls=3. Step 4: Final Answer: 16; llm_calls=4, tool_calls=3, stopped_reason="final". Four LLM calls for three tools — one per step plus the finalizer.

ReWOO. Planner (call 1) emits the 3-step plan. Workers: E1 = search("france") → fact; E2 = word_count("#E1") → the exact reference passes the string fact, returns 8; E3 = multiply(x="#E2", y=2) → the exact reference passes the int 8, returns 16. Solver (call 2) reads E1=..., E2=8, E3=16 and answers 16. llm_calls=2, tool_calls=3. Same three tools, half the model calls, and no quadratic resend.

Plan-execute-replan. First plan ends in no_such_tool → dispatch returns ToolError, failed_step = "E2: unknown tool ...", inner loop breaks. replans=1; context now contains FAILED at E2. Second plan (the pure planner returns the corrected plan once it sees FAILED) runs search then word_count cleanly; failed_step is None; answer is the last evidence value. llm_calls=2 (1 + 1 replan), stopped_reason="final". ReAct would have paid a model call on every step of both attempts; this paid two, total.

Why the naive machines fail

Three "simplifications" each break a specific invariant. No max_steps breaks termination: the machine is no longer guaranteed to halt, and a looping oracle becomes an unbounded resource burn. Raising on tool error breaks reliability isolation: the run's success probability collapses to the product of its tools' success probabilities, so one flaky dependency caps the whole agent. String-only substitution breaks type integrity across the wire: an exact #E1 reference stringifies a number, and the next tool either throws or, worse, silently computes the wrong thing. Each is not a smaller version of the loop — it is a machine with a different, broken contract. The discipline of this phase is the guards, the boundary, and the typed wire; the loop itself is five lines.