« Phase 22 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 22 — Staff Engineer Notes: Google Agent Development Kit (ADK)

Anyone can wire up an LlmAgent, hand it three FunctionTools, and call runner.run. That is using ADK. Being trusted to own it is a different job: it is knowing which coordination decisions belong to code and which to the model, defending those choices in a design review, and catching the one line in a diff that turns a reliable system into a flaky one. This doc is about that gap — the judgment, the red flags, and the signal an interviewer is actually listening for.

The decisions a staff engineer owns

1. Workflow agent vs sub_agent transfer — the defining call. This is the ADK decision, and it has exactly one right framing: is the control flow knowable at design time? If yes, encode it as a SequentialAgent/ParallelAgent/LoopAgent — the order is guaranteed, testable, replayable, and costs zero model calls to route. If the routing genuinely needs to read the input at runtime (triage a support message into billing vs technical vs sales), use sub_agents + transfer_to_agent and accept the non-determinism because you're buying real adaptivity. The junior instinct is that model- driven routing is "more advanced." The staff instinct is the opposite: reliability compounds (Phase 00's 0.95ⁿ), so you spend determinism everywhere you can and reserve LLM routing for the places that actually need it. Most real systems are a shallow LLM coordinator over deterministic workflow subtrees — model at the top where routing is real, code underneath where the flow is known.

2. ADK vs LangGraph vs OpenAI Agents SDK — pick by constraint, not taste. Reach for ADK when you're on Google Cloud / Gemini and want first-class Vertex AI Agent Engine deployment (managed sessions, autoscaling, tracing) and the opinionated workflow-agent split. Reach for LangGraph when you need an arbitrary, complex control-flow graph — custom reducers, intricate cycles, fine- grained checkpoint/resume — because it hands you the graph directly. Reach for the OpenAI Agents SDK when you're OpenAI-centric and want the minimal Agent + Runner + handoffs surface. The staff signal is that you pick by the team's actual constraints (ecosystem, deployment target, orchestration complexity), and you know the ideas are portable — an agent as (model + instruction

  • tools), a runner that emits events, deterministic structure around fuzzy leaves — so the framework is a substitution, not a rewrite.

3. State scope choices. Every state write is a tenancy decision. Unprefixed = this conversation. user: = this user's preferences across sessions. app: = global config for everyone. temp: = scratch, never persisted. Choosing the scope wrong is not a style nit; an app: write where you meant session is a cross-tenant leak, and an unprefixed write where you meant user: is a preference that silently fails to persist. Owning ADK means treating the prefix as a boundary you name deliberately at the write site.

Code-review red flags

These are the diffs a staff engineer blocks on sight:

  • An LLM coordinator where the flow is known. A top-level LlmAgent with five sub_agents routing a fixed three-stage pipeline. The model will occasionally transfer to the wrong child. Comment: "This is a SequentialAgent. The routing never needs the model — you're paying tokens and buying flakiness to re-derive a fixed plan."
  • A ParallelAgent branch reading a sibling's output_key. The instruction of branch B references branch A's key. This passes in dev (A finishes first) and races in prod. Comment: "Branches must be independent. Each writes its own key; a downstream sequential step combines them."
  • Unprefixed state that should be user: (or vice versa). A preference written without a prefix won't persist across sessions; a per-request value written to app: leaks into everyone's context. Comment: "Name the scope. user: for preferences, session for this turn's working set."
  • A LoopAgent with no max_iterations relying on a model to escalate. It can spin. Comment: "Cap it. Escalation is the model's judgment; the bound is your safety net."
  • A tool with a vague or missing docstring. The docstring is the model's interface; a bad one is silent mis-calling. Same for an overlapping set of descriptions on sub_agents — that's a routing bug wearing a model-failure costume.
  • A mutating tool with no before_tool_callback. Anything that writes to production should have a guardrail on the trust boundary, and the guardrail should be testable: assert the function ran zero times when blocked.

Production war stories

  • The AI dispatcher that flaked 1-in-8. A five-sub_agent coordinator routing a known pipeline; the model occasionally mis-routed. Rewriting it as a SequentialAgent killed the flakiness overnight. The lesson isn't "the model is bad" — it's "we made the model responsible for a decision code already knew."
  • The surprising Gemini bill. Repeated support questions hit Gemini every time. A ten-line before_model_callback cache (hash the request, return the stored response on a hit) cut model calls ~40%. The short-circuit contract is a cost lever, not just an observability hook.
  • The LoopAgent that never escalated. A critic sub-agent that could never quite decide "good enough" ran to a high max_iterations every time — each iteration a paid model call. The bill was the symptom; the missing escalation condition was the cause. Watch the Event stream's step count.
  • The parallel research heisenbug. Branch B referenced branch A's output; it worked until it didn't, correlated with load. The fix was structural (branches are independent), not more retries.

The signal an interviewer is listening for

When someone asks about ADK, the tell that separates a user from an owner is whether they reach for the deterministic-vs-LLM-driven distinction unprompted and can defend it with the reliability argument. Weak answers recite the API ("there's LlmAgent, FunctionTool, Runner"). Strong answers say: "ADK's whole philosophy is deterministic structure around LLM reasoning. I encode known flows as workflow agents so they're testable and replayable, and I reserve sub_agent transfer for runtime routing — because reliability compounds and I won't spend it on a decision I could hard-code." The follow-up signal: can they explain a guardrail as a provable property (before_tool_callback returns a rejection → the tool ran zero times, and you can assert it), and can they name the ParallelAgent independence race before you lead them to it. Those two — testable guardrails and the concurrency trap — are the concrete details that only someone who's built or owned this actually has.

Closing takeaways

  1. The distinction is the phase. Deterministic workflow agents put guaranteed structure around LLM leaves; sub_agents + transfer let the model route. Know the flow → workflow agent. Reliability compounds; don't spend it on a routing decision you could hard-code.
  2. Every state write is a tenancy decision. user:/app:/temp:/session are boundaries, not ergonomics. The wrong prefix is a leak or a lost preference, not a style nit.
  3. Guardrails and caches are before_* short-circuits, and their value is that they're testable. "The blocked tool provably never ran" is the difference between a guardrail and a hope.
  4. The Event stream is your observability, your replay log, and your bill. Step count, observation sizes, and cache hit rate all live there; read it before you add retries.
  5. Own the boundaries, not the API. ADK vs LangGraph vs OpenAI SDK is an ecosystem-and-constraint call; the ideas are portable, so master the mechanism and the framework is a substitution.