« Phase 22 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 22 — Principal Deep Dive: Google Agent Development Kit (ADK)

The mechanism doc treats ADK as a fold over an Event stream. This doc treats it as a component in a production system on Google Cloud, and asks the questions a principal engineer owns: where does this run at scale, what breaks under load, what is the blast radius when a tenant's data leaks, and what does each design choice cost in dollars and milliseconds. ADK's one-sentence thesis — deterministic structure around LLM reasoning — is not a slogan; it is a bet about where to spend reliability, and the architecture is the shape of that bet.

Dev/prod parity as a first-class property

The single most important architectural decision in ADK is that the agent is decoupled from its services through interfaces. An agent talks to a SessionService, a MemoryService, and a model provider — never to a concrete store. This is the seam that makes dev/prod parity real rather than aspirational:

• Local dev: InMemorySessionService, an injected model policy, adk web/adk run to watch the Event stream.
• Production: VertexAiSessionService (managed, persistent, autoscaled), Gemini on Vertex, deployed to Vertex AI Agent Engine.

The agent code does not change. You swap the service implementation and the same orchestration, the same scoped-state semantics, the same callbacks run against managed infrastructure. The principal-level consequence: your test suite exercises the real control flow because the control flow lives in the agent, not the platform. Contrast the anti-pattern where orchestration lives in the deployment substrate — then your local tests prove nothing about production behavior. ADK puts the determinism where you can test it.

Scaling: turning a sum into a max, and killing per-run planning cost

Two performance levers dominate, and both come from choosing structure over LLM improvisation.

ParallelAgent turns a sum of latencies into a max. A research assistant that gathers from three sources serially costs t₁ + t₂ + t₃. Run them as a ParallelAgent and the wall-clock cost is max(t₁, t₂, t₃) plus a merge. With three ~2s retrievals, that is ~6s → ~2s — a 3× latency cut for free, because the branches are independent (an antichain in the dependency DAG). This is the ReWOO plan-then-parallel-execute insight, made a first-class agent. The moment a principal sees three independent evidence-gathering steps chained sequentially, the review comment writes itself.

Deterministic workflow agents eliminate per-run re-planning. An LLM coordinator with sub_agents pays for a model call every run just to decide "do A, then B, then C" — a decision that was knowable at design time. That is tokens spent, latency added, and a non-deterministic failure mode introduced, all to re-derive a fixed plan. A SequentialAgent encodes the plan once, in code, and pays zero model calls to route. Across millions of runs, replacing a routing LLM call with a workflow agent is both a cost line item and a reliability win. The capacity math: if coordination is ~1 model call at ~500 tokens per run and you serve 10M runs/day, that is 5B tokens/ day of pure routing overhead you delete by using a workflow agent where the flow is known.

Scoped state as a multi-tenancy boundary

The user:/app:/temp:/unprefixed scope prefixes are not ergonomics — they are tenancy boundaries, and a principal reads them as such:

• unprefixed = per-conversation. Blast radius of a bad write: one session.
• user: = per-user, shared across all of that user's sessions. Blast radius: one user's entire history. This is where preferences and profile live.
• app: = global, shared across every session of every user. Blast radius: the whole tenant population. A wrong app: write is a config change that hits everyone at once.
• temp: = this turn only, never persisted. Blast radius: nil — which is exactly why secrets and scratch values belong here.

The design forces you to name the sharing scope at the write site. state["user:tier"] = "gold" is a different, deliberate act from state["tier"] = "gold". The failure mode a principal guards against: an unprefixed key that should have been user:, so a preference silently fails to persist across sessions; or its inverse, a per-request value written to app:, leaking one request's context into every user's next turn. The VertexAiSessionService persists these tiers to managed storage with the same prefix routing — so a scoping bug written against the in-memory store is a cross-tenant data leak in production. The prefixes are the multi-tenant contract, and they are enforced by the State view's dispatch-on-prefix, not by convention.

Cross-cutting concerns ride the callback and Event rails

Because state changes and control flow all pass through Events and callbacks, the platform-level concerns compose cleanly onto them:

• Security lives in before_tool_callback. It sits on the trust boundary: the model proposes a tool call, the callback inspects the proposed arguments, and returning a rejection means the tool function provably never executes. A hard block on any production-write tool is one callback, testable by asserting the function ran zero times. Authorization does not belong in the agent's prompt; it belongs on this rail.
• Cost lives in before_model_callback. Hash the request; on a cache hit, return the stored response and the paid model call is skipped entirely. For a support agent seeing repeated questions, this is a direct, measurable cut in Gemini spend — the short-circuit contract doing exactly what it is for.
• Observability lives in the Event stream. Because every step is an Event with an author, content, and state_delta, the stream maps directly onto traces and spans in Vertex AI Agent Engine. You get per-step token attribution, latency per agent, and a replayable transcript without instrumenting anything — the observability was a byproduct of the execution model.

The principal insight: these are not three subsystems bolted on. They are three interpretations of the same interception points. Add a new cross-cutting concern (rate limiting, redaction, audit) and it is another callback or another Event consumer — the architecture already has a place for it.

Failure modes and blast radius

• A LoopAgent that never escalates spins to max_iterations — bounded cost, but if the bound is high and the loop calls a model each iteration, it is a quiet cost amplifier. Blast radius: one run's budget, multiplied by however long it takes someone to notice the bill.
• An LLM coordinator that mis-routes sends a request down the wrong sub-agent path. Blast radius: one request's correctness, and — because it is non-deterministic — an intermittent failure that is hard to reproduce. This is the argument for workflow agents wherever the flow is known.
• A ParallelAgent branch reading a sibling's write is a race that passes in dev and fails in prod. Blast radius: silent wrong answers, correlated with load. The structural fix (branches are an antichain) is cheaper than any amount of retry logic.
• A scoping bug (app: where you meant session) is the highest-blast-radius failure: one request's data written to global scope contaminates every tenant. This is why the scope prefix is a review-gate item, not a stylistic preference.

Why the "looks wrong but is intentional" parts are right

Two ADK choices look like limitations and are load-bearing decisions. First, the model is injectable and the loop is model-agnostic — the framework refuses to hardwire Gemini, so the same orchestration runs against any provider via LiteLlm. That is what lets a Google-Cloud shop adopt ADK without betting the whole stack on one model's availability. Second, workflow agents are "just" BaseAgents with no LLM — they look underpowered next to an autonomous coordinator, but that is the point: the powerful, non-deterministic part is confined to the leaves, and the structure around them is ordinary code you can unit-test, replay, and defend in a design review. ADK's entire value proposition is knowing which parts deserve determinism and building the API so the deterministic choice is the easy one. A principal's job is to keep the LLM in the leaves and the control flow in code — and to recognize, at review time, every place someone let it drift the other way.