« Agentic AI Engineer · Cheat Sheet · Glossary

System-Design Walkthroughs — The Agent-Platform Design Framework

Who this is for: you did the phases, you can build each mechanism, and now an interviewer says "Design an enterprise agent platform" and gives you 45 minutes and a whiteboard. This section is the reusable structure you run every time, plus five worked instances of it — the archetypes the target roles actually hire for. Read this page first; it is the spine every walkthrough hangs off.

The phases teach you to build the mechanism (the ReAct loop, the MCP server, the durable workflow, the sandbox, the eval harness). System design is the other half of the Staff bar: composing those mechanisms into a platform, naming the tradeoffs, quoting the numbers, and predicting the failure modes before the incident. A junior lists boxes. A Staff engineer runs a repeatable framework, defends every decision with a number, and pre-mortems the design in the same breath. This page is that framework.

Table of Contents

1. The five walkthroughs
2. What makes agent system design different
3. The framework (the 8-step spine)
4. Step 1 — Clarify requirements & scale
5. Step 2 — Define the API / contract
6. Step 3 — The reference architecture (components)
7. Step 4 — Data & storage
8. Step 5 — The five cross-cutting concerns
9. Step 6 — Tradeoffs (the dials)
10. Step 7 — Failure modes (the pre-mortem)
11. Step 8 — The delivery script
12. The numbers to quote
13. How each walkthrough instantiates the framework

The five walkthroughs

Each walkthrough is a complete run of the framework below for one archetype. They deliberately overlap at the seams (an enterprise platform contains an MCP layer, a RAG subsystem, a durable engine, and an eval gate) so you see the same primitive from five angles. Companion drills live in interview-prep/03-system-design-cases.md.

What makes agent system design different

Ninety percent of an agent platform is a normal distributed system — a gateway, queues, databases, caches, workers, quotas, traces. If you can design a rate-limited multi-tenant SaaS backend, you already own that ninety percent. The other ten percent is what the interview is actually probing, and it inverts four instincts a good backend engineer has:

1. The compute unit is a stochastic oracle, not a function. A deterministic service returns the same answer for the same input; an LLM returns a distribution. So you cannot reason about correctness per-call — you reason about it statistically (\(0.95^n\)), and you design verification, retries, and human gates as first-class components, not error handling.
2. Input and instructions share one channel. A SQL injection is stopped by parameterized queries because code and data are separable. An LLM cannot reliably separate the two — any text it reads (a document, a tool result, a memory) can become a command. Security is therefore architectural (trust boundary, least privilege, isolation), never a validated input. (See Phase 10.)
3. The dominant cost is tokens, and tokens scale super-linearly with autonomy. A ReAct loop resends its whole scratchpad every step, so an n-step agent is roughly \(O(n^2)\) input tokens. Cost and latency are designed (routing, caching, chain length), not observed on the invoice. (See Phase 00 / Phase 14.)
4. "It worked in the demo" is a sample of one. Because the oracle is stochastic, the only proof a change is safe is a golden-set + regression gate run. Evaluation is part of the platform, not a QA afterthought. (See Phase 11.)

Hold those four in mind and every design decision below has an obvious reason.

The framework (the 8-step spine)

Run these in order, out loud, every time. Steps 1–4 build the happy path; step 5 is where Staff candidates separate from seniors; steps 6–7 are where you show judgment; step 8 is delivery.

1. CLARIFY   requirements & scale        → turn vibes into numbers
2. CONTRACT  the API / interface         → what a caller sends & gets, sync vs async
3. COMPONENTS the reference architecture  → gateway · runtime · tools/MCP · retrieval ·
                                            orchestration · durability · state/cache
4. DATA      storage & data model        → what's stored where, and the tenancy key
5. CROSS-CUT the five non-negotiables    → RELIABILITY · SECURITY · EVAL · COST/LATENCY · OBSERVABILITY
6. TRADEOFFS the dials                    → name the axis, pick a point, say why
7. FAILURE   the pre-mortem               → how it breaks, how you detect & contain
8. DELIVER   the script                   → drive the conversation, don't get driven

The single most common failure in an agent design interview is skipping straight to step 3 (boxes and arrows) without step 1 (numbers) or step 5 (the cross-cutting concerns). The framework exists to stop you doing that.

Step 1 — Clarify requirements & scale

Never draw a box before you have numbers. Agent architecture is determined by scale and reliability targets — a 2-step 100-QPS assistant and a 40-step 5-QPS research agent are different machines. Ask these, and write the answers on the board:

Then do the arithmetic before the architecture. Example you can say out loud: "You want 10 steps at 95% per step — that's \(0.95^{10} \approx 0.60\) end-to-end, unacceptable. So either I cut the chain (ReWOO, fewer tools), buy steps back with a per-step retry (1 − (1−p)^(r+1)), add a critic, or checkpoint durably so a human can approve the risky step. I'll do three of those four." That sentence alone signals Staff. It comes straight from Phase 00 and its reliability/cost calculator lab.

Step 2 — Define the API / contract

Before internals, define what a caller sends and receives — it forces the sync-vs-async decision that shapes everything downstream.

• Synchronous (POST /v1/agents/{id}/invoke returns the answer) works only for short tasks inside the latency budget. Streaming (SSE / chunked) keeps a sync connection alive while the agent thinks — the right default for chat.
• Asynchronous (POST /runs returns a run_id; poll GET /runs/{id} or subscribe to a webhook/stream) is mandatory for multi-minute or human-gated tasks. This is the seam where durable execution (Phase 08) enters: the run_id is a durable workflow handle that survives a crash.
• Idempotency: every mutating call takes an Idempotency-Key; the platform dedupes so a client retry (or an at-least-once queue redelivery) does not double-charge or double-send.
• The envelope: requests carry tenant_id (from the auth token, never the body), a principal/scopes, a budget (token/step/dollar caps), and a trace_id. Responses carry the answer, a usage block (tokens, cost, latency), the trajectory (for eval/audit), and a finish_reason (stop / budget_exhausted / guardrail_blocked / needs_approval).

The contract is the trust boundary made concrete: the caller hands you intent; your platform decides, validates, and executes.

Step 3 — The reference architecture (components)

This is the canonical agent-platform diagram. Every walkthrough is a specialization of it — some components collapse to a box, others explode into the whole design.

                        ┌───────────────────────────────────────────────┐
   client ──HTTPS──▶    │                 GATEWAY                        │
                        │  authN (OIDC) · authZ (RBAC/ABAC) · tenant_id  │
                        │  rate-limit / quota (token bucket) · idempotency│
                        │  admission control · request envelope · trace  │
                        └───────────────┬───────────────────────────────┘
                                        │ (validated, tenant-scoped)
                        ┌───────────────▼───────────────────────────────┐
                        │            ORCHESTRATION / CONTROL PLANE        │
                        │  intent router · model router/cascade          │
                        │  durable workflow engine (Phase 08)            │
                        │  supervisor / worker / critic (Phase 07)       │
                        └───┬───────────┬───────────┬──────────┬─────────┘
                            │           │           │          │
              ┌─────────────▼──┐  ┌─────▼──────┐ ┌──▼───────┐ ┌▼──────────────┐
              │  AGENT RUNTIME │  │  TOOL /MCP │ │ RETRIEVAL│ │  MODEL ACCESS │
              │  ReAct/ReWOO   │  │  LAYER     │ │ hybrid + │ │  LLM providers│
              │  loop + budget │  │  registry· │ │ rerank · │ │  + semantic & │
              │  (Phase 01)    │  │ authz·audit│ │ GraphRAG │ │  prefix cache │
              │                │  │ SANDBOX    │ │(P05/P06) │ │  (Phase 14)   │
              │                │  │ (Phase 09) │ │          │ │               │
              └───────┬────────┘  └─────┬──────┘ └────┬─────┘ └──────┬────────┘
                      │                 │             │              │
        ┌─────────────▼─────────────────▼─────────────▼──────────────▼─────────┐
        │                       DATA & STATE PLANE                             │
        │  event log / history (durability) · state store (KV) · vector DB     │
        │  (per-tenant namespace) · object store · secrets vault · audit log   │
        └──────────────────────────────────────────────────────────────────────┘
                      │
        ┌─────────────▼──────────────────────────────────────────────────────┐
        │       CROSS-CUTTING: OTel traces · cost meter · guardrails ·        │
        │       eval harness (offline + online) · red-team (Phases 10/11/14) │
        └────────────────────────────────────────────────────────────────────┘

Component-by-component, the job of each and the phase that builds it:

• Gateway — the front door and the trust boundary in code. Terminates auth (OIDC/OAuth), resolves tenant_id from the token, enforces RBAC/ABAC, applies per-tenant token-bucket quotas, dedupes on the idempotency key, stamps a trace_id, and does admission control (reject or shed load before it hits an expensive model). (Phase 13.)
• Orchestration / control plane — decides how to run this request: which intent/skill (Phase 04), which model tier (Phase 14), and whether it is a request-scoped loop or a durable workflow (Phase 08). For hard tasks it fans out to supervisor/worker/critic roles (Phase 07).
• Agent runtime — the ReAct/ReWOO/plan-execute loop with a hard step budget, error-as- observation, and a trace per step. (Phase 01.)
• Tool / MCP layer — the typed catalog of actions, discovered and called over MCP, permission- gated per tenant/principal, executed inside a sandbox, and audited. (Phase 03 + Phase 09.)
• Retrieval — hybrid (BM25 + dense) search with reranking, ACL/security-trimming, and optional GraphRAG/RAPTOR for global questions. (Phase 05 / Phase 06.)
• Model access — the boundary to LLM providers, fronted by a router/cascade and a semantic + prefix cache, with a cost meter on every call. (Phase 14.)
• Data & state plane — see step 4.
• Cross-cutting — traces, cost, guardrails, evals; see step 5.

Step 4 — Data & storage

The Staff move here is to name what is stored, in which store, keyed by what — and to make tenant_id part of every key. The AI-specific leak channels all live in this plane.

Staff vs junior: a junior stores embeddings in one shared index because "the vector DB handles it." A Staff engineer says: "The shared vector index is the number-one multi-tenant leak channel — one missing filter and tenant A retrieves tenant B's documents. I default-deny with a per-tenant namespace, and I put tenant_id in the prompt-cache key too, or the cache leaks across tenants." That ranking (vector index, then prompt cache, then agent memory, then co-mingled fine-tunes) is straight from Phase 13.

Step 5 — The five cross-cutting concerns (every design must cover)

This is the heart of the framework. Every agent platform design — no exceptions — must explicitly address all five. Interviewers grade on whether you volunteer these unprompted. Each walkthrough has a section per concern; here is the reusable checklist.

5.1 Reliability

• Do the compound-reliability math: \(p_{\text{end}} = p^n\); \(0.95^{10} \approx 0.60\). State the per-step p, the step count n, and the target T, then the reliable step budget \(\lfloor \log T / \log p \rfloor\).
• Buy steps back with: shorter chains (ReWOO), per-step retries on idempotent tools (1 − (1−p)^(r+1)), a critic/verifier (Phase 07), and human gates on high-impact actions.
• Durability: for anything multi-minute or side-effecting, the run is a durable workflow (event-sourced, deterministic replay, idempotent activities) so a crash resumes instead of restarting. (Phase 08.)
• Degradation ladder: define what happens under load/failure — cheaper model → cached answer → queued/deferred → graceful refuse. Never a hard 500.

5.2 Security

• State the trust boundary explicitly: the model proposes, the platform executes after validation. Every control lives on your side of that line.
• Prompt injection (direct / indirect / memory) has no general fix — contain it: least privilege (minimal tools/scopes), allow-lists for tools and egress, output exfil scanning (markdown-image and outbound-URL leaks), and human approval on high-impact actions. (Phase 10, OWASP LLM Top 10.)
• Least privilege & sandboxing: tool code runs in a capability-gated jail (fs/net/exec limits, resource caps, egress control). (Phase 09.)
• Tenancy: tenant_id threaded from trusted auth through every layer; default-deny; per-tenant namespaces, cache keys, and quotas. (Phase 13.)

5.3 Evaluation

• Golden dataset (versioned) per task → run → score (programmatic > calibrated judge > human) → regression gate in CI. You evaluate your task, not public benchmarks.
• Grade the trajectory, not just the final answer — did it call the right tools, in a sane order, without wasted steps?
• LLM-as-judge needs a Cohen's κ agreement check against humans (κ > ~0.6) before you trust it in CI, and controls for position/verbosity/self-preference bias.
• Safety is a gate, not a weighted average — a run that leaks PII fails outright regardless of task score. (Phase 11.)

5.4 Cost & latency

• Unit economics: quote $/resolved-task = $/attempt ÷ success_rate. A cheap model that fails half the time is more expensive per resolved task than an expensive one that succeeds.
• Levers: token budget, semantic + prefix cache (target 30–60% hit-rate on repetitive traffic), model routing/cascade (cheap model first, escalate on low confidence), chain-length reduction, and distillation.
• Design to p95/p99, not the mean. Streaming hides latency (time-to-first-token is the felt metric). (Phase 14.)

5.5 Observability

• Trace every run with OpenTelemetry GenAI spans (one span per LLM call, tool call, retrieval), linked by trace_id; attach tokens, cost, latency, and finish_reason.
• Meter cost per request per tenant from day one — you cannot bill or budget what you do not measure.
• Online eval / drift: sample production traffic into the eval harness; watch success-rate, cost, and guardrail-trigger rates as SLOs, not just CPU/memory.

Step 6 — Tradeoffs (the dials)

Judgment is naming the axis and choosing a point on it out loud. The universal agent-platform dials:

Step 7 — Failure modes (the pre-mortem)

Close every design by breaking it yourself. The recurring agent-platform failure modes and their containment:

• The agent loops (never emits final) → step budget + wall-clock cap + loop-detector; alert on runs hitting the cap.
• Reliability collapse on long chains (0.95^n) → checkpoints, critics, HITL gates; monitor end-to-end success as an SLO.
• Prompt injection → exfiltration (a fetched page tells the agent to email a secret) → egress allow-list, output exfil scan, least privilege, HITL on sends. (Phase 10.)
• Cross-tenant leak (shared vector index / prompt cache) → per-tenant namespaces + cache keys, default-deny, isolation tests in CI. (Phase 13.)
• Cost blowout (a runaway multi-agent fan-out, or a retry storm) → per-tenant token quotas, per-run budget caps, circuit breakers, cost alerts.
• Silent quality regression (a prompt/model change tanks success) → golden-set regression gate in CI; block deploy on a drop. (Phase 11.)
• Provider outage / latency spike → multi-provider routing, timeouts, degradation ladder, cached fallbacks.
• Poisoned tool result / retrieval → treat all tool/retrieval output as untrusted; sanitize; never auto-execute instructions found in data.
• Non-deterministic workflow breaks replay → keep wall-clock/random/IO out of workflow code; all effects via activities. (Phase 08.)

Step 8 — The delivery script

Drive the conversation. A tight 45-minute run:

1. 2 min — restate the problem, ask the step-1 questions, write the numbers on the board.
2. 3 min — do the reliability/cost arithmetic; state whether it is an agent or a workflow.
3. 5 min — the contract (sync/async, envelope, idempotency).
4. 10 min — the reference architecture; walk one request lifecycle end to end.
5. 10 min — the five cross-cutting concerns; volunteer all five, go deep on the two the role cares about (Docker → security+eval; Temporal → reliability+durability; Citi → tenancy+RAG).
6. 5 min — tradeoffs: name three dials and where you put them and why.
7. 5 min — the pre-mortem: break it, then contain it.
8. 5 min — buffer for their deep-dive; follow their lead, keep quoting numbers.

Staff vs junior, in one line: a junior answers the question asked; a Staff engineer frames it (numbers first), volunteers the cross-cutting concerns unprompted, and pre-mortems their own design. Every claim closes with a number.

The numbers to quote

Have these loaded (fuller table in the Cheat Sheet):

How each walkthrough instantiates the framework

• 01 Enterprise Agent Platform runs all eight steps at full depth — it is the reference. If you only read one, read this one.
• 02 MCP Tool Platform explodes the tool/MCP + sandbox component (step 3) and the security concern (5.2) into a whole platform.
• 03 Durable Multi-Agent Workflow explodes the orchestration + durability component and the reliability concern (5.1).
• 04 RAG at Scale explodes the retrieval component and the data/storage plane (step 4), with security-trimming as the tenancy angle.
• 05 Eval & Safety Platform explodes the evaluation and security concerns (5.3 + 5.2) into a platform with its own lifecycle.

Read a walkthrough with this page open; you will see the same eight headings every time. That repetition is the skill — the framework is what lets you design a system you have never seen in a room you have never been in.