Lab 01 — Multi-Agent Orchestrator: Blackboard, Roles, Scheduling & Consensus

Phase: 13 — Multi-Agent Orchestration Difficulty: ⭐⭐⭐☆☆ (the code is plumbing; the judgment — when multi-agent helps and how to stop it looping — is ⭐⭐⭐⭐⭐) Time: 4–5 hours

A single agent is an LLM in a loop (Phase 12). A multi-agent system is several such agents sharing state and taking turns — so genuinely distinct roles (a planner, executors, a critic) can collaborate and check each other. This lab builds that machinery from scratch, minus the LLM: every agent's behaviour is an injected deterministic policy so the whole system is reproducible and testable. You build the blackboard (shared, append-only, authored state), the scheduler (deterministic turn-taking), the orchestrator with its critique-revise loop (the cheapest reliable quality boost in agentic systems), consensus voting, and stall detection — the guard that makes the difference between a system that halts and one that bills you forever.

What you build

• Blackboard — write(key, value, author) / read(key) / history(key): a shared store with an append-only, authored, monotonically-versioned log. The audit trail and the basis of stall detection. Optional subscribe for write notifications.
• Agent — name, role (planner / executor / critic / router — just data), and an injected policy(view) -> action. Behaviour comes entirely from the policy; the scheduler never knows there's no LLM behind it.
• RoundRobinScheduler / PriorityScheduler — next_agent() / step() / run(max_rounds): deterministic turn-taking. Round-robin is fair; the priority scheduler is opinionated (planner before executors), with a stable tie-break.
• Orchestrator — wires a planner (decompose a task into subtasks), executors (claim & complete subtasks), and a critic (review) into a working team, including the critique-revise loop: executor drafts → critic accepts or returns revision notes → executor revises, until accepted or a hard max_revisions cap.
• consensus_vote / weighted_vote — plurality and weighted aggregation with a deterministic (earliest-ballot) tie-break — the debate-to-agreement primitive.
• detect_stall — no-progress detection (the last window writes are all no-op rewrites) so the orchestrator halts instead of spinning.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why the blackboard's history is append-only and authored and how the monotone seq makes the run replayable and stall-detectable.
• You can explain why test_orchestrator_decomposes_and_completes is the soul test: a scripted planner/executor/critic team must turn a task into a correct final blackboard state.
• You can explain why test_critique_revise_caps_at_max_iterations is the safety soul test: a critic that never accepts must terminate at the cap, not loop forever.
• You can explain why detect_stall watches for no-op rewrites rather than just "did anyone write," and why the orchestrator halts on stall instead of burning the whole budget.
• You can state when multi-agent actually helps (distinct roles, parallelizable subtasks, cross-checking) and when it just adds latency, cost, and coordination bugs.

How this maps to the real stack

Limits of the miniature (be honest in the interview): real agents are non-deterministic (temperature > 0), so production needs retries, schema-validated tool calls, and eval harnesses the offline lab elides; the shared blackboard here is free, but in production every agent re-reads the shared context as tokens — coordination has a real, super-linear cost; and consensus over LLM samples can amplify a shared bias (groupthink), which a majority vote does not fix.

Extensions (build these toward a real system)

• Add a router role: an agent whose policy reads the task and dynamically hands off to the right specialist (the Swarm/handoff pattern) instead of a fixed plan.
• Make executors claim subtasks from the board (write claim:<task> with the author) so two executors never duplicate work — model the race and the deterministic claim resolution.
• Add a debate mode: two executors argue, a judge agent reads both and calls consensus_vote or weighted_vote to decide — and measure when debate beats a single critic.
• Add a token/cost budget alongside max_rounds: charge each turn and halt on budget exhaustion, then chart cost vs. quality as you add agents (the cost-explosion failure mode).
• Swap one policy for a real LLM call behind the same policy(view) -> action signature and show that the scheduler, blackboard, and orchestrator are unchanged.

Interview / resume

• Talking points: "When does multi-agent actually beat one good agent, and when is it just latency and coordination bugs?" "How do you stop an agent system from looping forever?" "Why is the critique-revise loop the cheapest reliable quality win?" "How do you make a multi-agent run reproducible?"
• Resume bullet: Built a deterministic multi-agent orchestrator from scratch — a shared authored blackboard, role-specialized agents with injectable policies, round-robin/priority scheduling, a capped critique-revise consensus loop, weighted voting, and stall/deadlock detection — modeling the coordination, reflection, and termination patterns behind CrewAI / AutoGen / LangGraph.