« Phase 07 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 07 — Deep Dive: Multi-Agent Orchestration

The load-bearing mechanism in this phase is not "several LLMs." It is a shared, totally-ordered message log plus a mutable blackboard dict, driven by an orchestrator loop that reads state, routes by name, and terminates on a quiescence condition. Strip the word "agent" and what remains is a tiny deterministic message-passing runtime. This doc traces that runtime at the level a person implementing it has to reason about — the envelope schema, the routing loop, the ordering and consistency invariants, and where they break — because that is exactly what solution.py makes you build.

The message envelope and the two data structures

Every interaction is one Message: a five-field record (sender, recipient, kind, content, seq). Four fields are semantic; seq is infrastructure. sender and recipient are agent names (strings, resolved through a registry, never object references — this is what lets a handoff cross a process boundary in the real thing). kind is a small closed vocabulary — control, handoff, result, draft, critique — that tags the protocol slot a message occupies, which is what makes the transcript self-describing: history(kind="handoff") reconstructs the routing graph without parsing content. content is the opaque payload (a string, or a typed Handoff).

Behind the messages sit two structures with different jobs:

  • The message log — an append-only list[Message]. post(msg) stamps msg.seq from a monotonic counter, increments the counter, appends, and returns. That counter is the entire ordering guarantee: because it is an integer bumped on a single code path, history() is a total order, and two identical runs produce byte-identical transcripts. Order by a wall clock instead and you have lost determinism the instant two events land in the same millisecond — untestable, unreplayable.
  • The blackboard — a shared state dict: {task, completed: [...], results: {...}}. This is the 1970s blackboard architecture: agents coordinate by reading and writing shared state, not by calling each other. The supervisor routes off completed; workers append their output to results. The distinction from the log matters — the log is the history (immutable, ordered, an audit tape); the blackboard is the current world (mutable, overwritten). Confusing the two is how people build systems they cannot replay.

The orchestrator loop, field by field

Supervisor.run(task, workers, max_turns, bus) is Phase 01's agent loop lifted one level: the "tools" are whole agents. Setup builds a registry — {w.name: w for w in workers} plus the supervisor itself — the name-to-agent map that makes routing a dictionary lookup and makes an unknown target a detectable error rather than a crash. Then a bounded for _turn in range(max_turns):

  1. Decide. decision = self.act(render_state(state)).strip(). render_state projects the blackboard to a small deterministic string — TASK: ... and COMPLETED: researcher writer. The supervisor routes on progress, not on the content of results, which keeps its context tiny and its policy a pure function of a stable rendering.
  2. Record the decision. post(Message(self.name, self.name, "control", decision)). The routing decision is itself a first-class message — a self-addressed control message — so the why of the next step is in the tape, not hidden in a variable.
  3. Terminate or route. If decision == "done", set reason = "done" and break — the quiescence condition. Otherwise build a least-context handoff: prior = list(results.values())[-1] (the most recent result, not the whole history), then make_handoff(self.name, decision, task, {task, prior}, registry), which validates decision is a known agent and raises ValueError if not.
  4. Execute. worker = registry[decision]; result = worker.act(render_handoff(handoff)). The worker sees only render_handoff output — TASK: and optionally PRIOR: — never the bus, never COMPLETED:, never another worker's chatter.
  5. Commit. Post the result message, then mutate the blackboard: results[name] = result; completed.append(name). State advances; the loop re-reads it next turn.

If the loop exits by exhausting range(max_turns) without a done, reason stays "max_turns" and the Transcript returns a partial result with stopped_reason == "max_turns" — a bounded failure, never a hang. The termination condition is therefore two-pronged: success is the policy emitting done (progress-driven quiescence); safety is the budget firing (a hard ceiling). Both are explicit; neither depends on a model choosing to stop.

A worked trace

Task: "Summarize recent work on X with citations", workers researcher and writer, a supervisor that routes the first unfinished worker then done. Watch the seq counter and the blackboard:

state = {task, completed: [], results: {}}
turn 0: render_state → "COMPLETED: "        → policy → "researcher"
        #0 supervisor→supervisor [control]  "researcher"
        #1 supervisor→researcher [handoff]  ctx={task, prior:""}
        researcher.act("TASK: ...\n")       → "sources: Smith 2021, Lee 2022, Ada 2023"
        #2 researcher→supervisor [result]   commit: completed=[researcher]
turn 1: render_state → "COMPLETED: researcher" → policy → "writer"
        #3 supervisor→supervisor [control]  "writer"
        #4 supervisor→writer     [handoff]  ctx={task, prior:"sources: ..."}
        writer.act("TASK: ...\nPRIOR: sources: ...\n") → "Report drafted from [sources: ...]"
        #5 writer→supervisor     [result]   commit: completed=[researcher, writer]
turn 2: render_state → "COMPLETED: researcher writer" → policy → "done"
        #6 supervisor→supervisor [control]  "done"        → stop (reason="done")

Seven messages, two routed workers, turns=2, stopped_reason="done". Note the mechanism that makes this auditable: reading #0..#6 in order tells you exactly who decided what, who was handed what, and what came back — no call stack to reconstruct. And note the least-context property is visible in #4: the writer's handoff carries prior (the researcher's findings) and the task, and nothing from #0..#3. The critique-revise loop is the same shape with kind toggling draft/critique and the budget being max_rounds: a good draft is one round (draft, approve); revise-then-approve is two; a never-approved draft stops at max_rounds with approved=False.

Ordering, consistency, and the races the miniature hides

The miniature is single-threaded, so several distributed-systems hazards are latent — invisible here, lethal the moment you add real concurrency. Name them, because they are the whole difficulty of the production version:

  • Message ordering. A single monotonic counter gives a total order for free only because there is one writer. Fan out to parallel workers on separate threads or processes and "the next seq" is a contended resource; you need a single-writer bus, a per-topic sequence, or logical (Lamport) clocks to recover a deterministic order. Order is not a given — it is a design.
  • Livelock (ping-pong). No deadlock is possible in the miniature — nobody blocks — but livelock is trivial: a supervisor whose policy oscillates researcher → writer → researcher makes progress in the log while making no progress toward done. The max_turns budget is precisely the livelock breaker; without it the loop burns tokens forever. In a real parallel topology you also get deadlock: a join step that waits for N worker results blocks forever if one worker never posts — which is why real fan-outs need per-branch timeouts, not just a global budget.
  • Duplicate delivery. A real network bus is at-least-once; handlers must be idempotent. Here it bites at commit: results[name] = result is idempotent by key (a redelivery overwrites with the same value), but completed.append(name) is not — a redelivered result double-appends and the supervisor's progress view corrupts. Idempotency is a per-write property you have to design, not a bus feature you can assume.
  • Shared-state races. The blackboard is a shared mutable dict. Single-threaded, reads and writes are trivially serialized. Under parallel workers writing the same key, last-writer-wins is a race, and you need either per-worker keys or a reducer — a merge function that combines concurrent writes deterministically (exactly LangGraph's channel model, Phase 07's Core Contributor doc).

Why the naive alternatives fail at the mechanism level

Two shortcuts tempt everyone; both fail structurally, not stylistically.

"One big agent." Fold research, writing, and checking into a single prompt-and-toolset and the failure is context blowup: every turn re-feeds the entire growing transcript, so cost and latency scale with turns squared, and past a point retrieval degrades (lost-in-the-middle) so the model gets worse as it accumulates its own history. There is also no verification seam — the same policy that wrote the draft judges it, sharing its blind spots. Roles exist to keep each policy's context small, single-purpose, and independently checkable.

"Unstructured chatter." Let agents call each other directly, passing whatever they like, and you lose the total order (no seq, no replay), the audit tape (hidden side channels), and — worst — convergence: with no blackboard to route off and no budget to bound the exchange, there is no defined quiescence condition, so "when does it stop?" has no answer. The bus-plus-blackboard-plus- budget is not ceremony; it is the minimum machinery that makes a multi-agent run ordered, replayable, and guaranteed to terminate.