« Rapid-Fire Q&A · Interview Prep · Coding Drills »

System-Design Cases

Nine agent-platform design prompts, each with a repeatable structure you can run live on a whiteboard. The structure is the skill: an interviewer scores whether you drive the design, ask the numbers first, name the trust boundary, and make explicit tradeoffs — not whether you recall a reference architecture. Full-length walkthroughs of five of these live in ../system-design/; this doc is the drill version — the shape to reproduce under pressure.

The structure to run every time

Say this order out loud so the interviewer sees the method:

1. Requirements & the six numbers — functional + non-functional. Then extract the six that decide the architecture (from Q72): step count, per-step reliability, target reliability, latency budget (p95), tenancy, cost target. Turn vibes into numbers.
2. API / interface — the contract first. What does a caller send, what streams back, what are the idempotency and auth semantics.
3. Components — the boxes, and where the trust boundary runs through them (model proposes, app executes).
4. Data — what's stored, keyed how, isolated how (especially tenant_id).
5. The four cross-cutting properties — reliability, security, cost, evaluation. Every platform answer must hit all four; most candidates forget eval.
6. Tradeoffs & "least-agentic that works" — what you deliberately did not build, and why.

Open every case with: "Before I draw boxes — what's the step count, per-step reliability, latency budget, tenancy model, and cost target?" That one sentence is worth a level.

Case 1 — Multi-Tenant Enterprise Agent Platform

Targets: Citi (both), RBC, Wolters Kluwer. Full walkthrough: ../system-design/01-enterprise-agent-platform.md.

Prompt. "Design an internal platform where many business teams build and run agents over shared enterprise data, safely, at Citi scale."

Requirements & numbers. Multi-tenant (many internal teams = tenants); agents call enterprise APIs + retrieve enterprise data; auditable; regulated. Ask: how many tenants, QPS/tenant, p95 budget (say 5s interactive), per-step reliability (~0.95), acceptable autonomy (high-impact actions need human gate). These set the isolation model and the step budget.

API. POST /v1/agents/{id}/invoke with tenant_id derived from the auth token, never the body; returns an SSE stream of steps + a final result; idempotency key on mutating invocations. Admin API to register agents, tools, and per-tenant policies.

Components. Gateway (authn/authz, rate limit, tenant_id injection) → agent runtime (the loop, P01) → tool layer via MCP servers (P03) → retrieval (P05/P06) → durable engine for long/high-impact flows (P08) → observability (P14). The trust boundary runs at the tool layer: every tools/call is validated + authz'd on the app side.

Data. Per-tenant vector namespaces (the #1 leak channel — never a shared index), tenant-scoped memory and cache keys, secret custody per tenant, audit log of every tool invocation. Everything keyed by tenant_id from trusted auth.

Cross-cutting. Reliability: 0.95^n sets a step budget; durable workflow + retries + idempotency for multi-step. Security (P13): default-deny, least privilege, HITL on high-impact (wire/deploy), egress control. Cost: model router + prefix cache, $/resolved-task per tenant for chargeback. Eval (P11): golden set per critical workflow, regression gate in CI, safety as a hard gate.

Tradeoffs. Most "agents" here are workflows (auditable, cheaper) — reserve autonomy for genuinely open tasks. Pool tenancy for cost but silo the vector store for regulated data. Don't build one god-agent; use intent routing (P04) to many small workflows to avoid collisions.

Case 2 — MCP Tool Platform

Targets: Docker, Citi, Anthropic. Full walkthrough: ../system-design/02-mcp-tool-platform.md.

Prompt. "Design a platform that hosts MCP servers so hundreds of agents can safely use hundreds of tools." (This is the M×N problem made real.)

Requirements & numbers. N tools × M agents, each server is untrusted code, tools must be discoverable, permission-gated, versioned, and isolated. Ask: how many servers, call volume, which tools are high-impact (mutating), latency budget per tool call.

API. MCP over the wire: initialize capability handshake (protocolVersion 2025-06-18), tools/list for discovery, tools/call for use; remote servers over Streamable HTTP + OAuth, local over stdio. notifications/tools/list_changed for dynamic tool sets.

Components. Registry/catalog of servers + tool schemas → a per-server sandbox (P09) — containerize each MCP server (Docker's angle) → a permission-gating proxy that validates every tools/call against JSON Schema (P02) and an allow-list → audit log. Trust boundary: the proxy, on every call.

Data. Tool schemas (versioned), per-agent tool allow-lists, invocation audit trail, secrets injected per call (never handed to the model).

Cross-cutting. Reliability: retries + timeouts per tool; circuit-break a flapping server. Security: each server least-privileged + egress-controlled — a malicious server can't pivot to the host or exfiltrate; scan tool results for indirect injection before they re-enter the model. Cost: cache idempotent tool results. Eval: contract-test every tool schema; behavioral regression on tool-selection accuracy.

Tradeoffs. Sandbox strength (microVM vs container) vs cold-start latency. Auto-approve read-only tools; require HITL for mutating ones. Few well-scoped tools beat many overlapping (Q14).

Case 3 — Durable Multi-Agent Research System

Targets: Temporal, OpenAI Infra, Citi. Full walkthrough: ../system-design/03-durable-multi-agent-workflow.md.

Prompt. "Design a system where a supervisor agent farms out a long (30–60 min) research task to worker agents, survives worker crashes, and lets a human approve the final report."

Requirements & numbers. Long-running (must survive process restart), multi-agent (supervisor/workers/critic), human approval gate. Ask: task duration, fan-out width, acceptable cost per report, how often a worker fails.

API. POST /research returns a workflow_id; GET /research/{id} streams progress; POST /research/{id}/approve sends the human-approval signal. Idempotency key on submit.

Components. A durable workflow engine (P08) is the spine: the workflow is deterministic orchestration; each LLM/tool call is an activity (retried, idempotent). Supervisor = workflow logic; workers = child workflows/activities; critic = a verify activity (P07). Human approval = a signal the workflow durably waits on.

Data. Event-sourced history (the replay log), per-activity idempotency keys, worker results on a message bus/blackboard.

Cross-cutting. Reliability: no LLM call, time.now(), or random() in workflow code — those are non-deterministic and break replay; they live in activities whose results are logged (Q37). Crash at minute 30 → replay to minute 30, re-run nothing committed. Security: workers sandboxed; the human gate stops a bad report from auto-publishing. Cost: fan out with ReWOO-style workers (linear tokens) where the path is predictable; $/resolved-report. Eval: trajectory eval on the research path + a critic pass before human review.

Tradeoffs. Durable engine adds ops complexity — justified only because the task is long and failure-prone (for a 5-second task a plain loop is fine — least-agentic that works). At-least-once delivery + idempotency = effective exactly-once.

Case 4 — Agent Evaluation & Safety Platform

Targets: Docker, Juniper Square, Anthropic, Wolters Kluwer. Full walkthrough: ../system-design/05-eval-and-safety-platform.md.

Prompt. "Design the platform that decides, automatically, whether a change to an agent shipped an improvement or a regression — and blocks unsafe releases."

Requirements & numbers. Every agent change runs against golden datasets in CI; catch quality regressions and block safety failures. Ask: how many agents/workflows, how big the golden sets, what's the human-labeling budget, what's the acceptable false-block rate.

API. POST /eval/run with {agent_version, dataset_version} → returns per-metric scores + a pass/fail gate decision; a CI webhook that blocks merge on regression or safety-gate failure.

Components. Golden dataset store (versioned) → runner (executes the agent over the set) → scorers: programmatic first, then a calibrated LLM-judge, then human spot-check (P11) → trajectory evaluator (tools, order) → regression gate + safety gate. Judge calibration harness computes Cohen's κ vs human labels.

Data. Versioned datasets + labels, per-run scores, κ history for judge trust, a leaderboard of agent versions.

Cross-cutting. Reliability: deterministic scorers where possible; judge only where not. Security: safety is a hard gate, not a weighted average (Q55) — injection/exfil/PII failures block regardless of quality. Cost: cheap programmatic checks gate first; run the expensive judge only on what passes. Eval-of-eval: monitor judge biases (position, verbosity, self-preference); require κ > ~0.6 to trust a judge in CI.

Tradeoffs. Judge speed/cost vs fidelity; golden-set coverage vs labeling cost. Evaluate your task, not public benchmarks. Feedback loop: production failures become new golden cases.

Case 5 — Customer-Support Voice Agent

Targets: LiveKit, Wolters Kluwer, Redcan. Related depth: P16.

Prompt. "Design a real-time voice agent that answers customer calls, looks up account data, and escalates to a human when needed — under a conversational latency budget."

Requirements & numbers. Real-time (sub-second perceived latency), streaming STT→LLM→TTS, tool calls to account systems, human handoff. Ask: target end-to-end p95 (~800ms), concurrent calls, languages, which actions need a human.

API. A streaming session (WebRTC/LiveKit) carrying audio in/out; server-side agent with a turn-taking state machine; tool calls to backend APIs; a transfer signal for escalation.

Components. VAD/endpointing → streaming STT → agent loop with tools (P01/P02) → streaming TTS → barge-in handling (interrupt cancels playback, re-opens mic). The turn-taking state machine (P16) is the core.

Data. Session state, conversation memory (tiered, P04), account lookups (tenant-scoped), a call transcript for audit.

Cross-cutting. Reliability: partial-failure handling (STT mis-hears → confirm, don't guess). Security: authenticate the caller before account tools (P13); never let the model self-assert identity; HITL/human transfer for high-impact (refunds). Cost/latency (P14): stream every stage, cut time-to-first-audio, route simple intents to a small model. Eval: transcript-based quality + task-success + a barge-in responsiveness metric.

Tradeoffs. Latency vs accuracy (start LLM before final STT when confident). The tail is the product — p95, not mean, or the conversation stutters.

Case 6 — RAG-at-Scale with GraphRAG

Targets: Citi, Wolters Kluwer, RBC. Full walkthrough: ../system-design/04-rag-at-scale.md.

Prompt. "Design retrieval for a large enterprise corpus that answers both pinpoint factual questions and broad thematic ones, with citations, and stays current as documents change."

Requirements & numbers. Millions of docs, mixed query types (local facts + global themes), citations required (regulated), incremental updates. Ask: corpus size, update frequency, recall target, latency budget, citation/audit requirement.

API. POST /retrieve → ranked, cited chunks + optional graph-community summary; POST /answer → grounded answer with sources; ingest pipeline API for new/changed docs.

Components. Ingest → chunk → embed → hybrid index (dense ANN + BM25) → RRF (k≈60) → cross-encoder rerank (P05) for local questions. For global/thematic: a GraphRAG entity/relation graph + community summaries, and a RAPTOR tree for multi-level abstraction (P06). A router (P04) sends local queries to hybrid retrieval, global ones to graph/tree.

Data. Vector index (HNSW), BM25 index, Neo4j graph + community summaries, RAPTOR tree nodes, source metadata for citations. LightRAG-style incremental updates so a few changed docs don't force a full re-index.

Cross-cutting. Reliability: evaluate retrieval (recall@k, MRR, nDCG) separately from generation (faithfulness). Security: tenant-scoped namespaces; citation = auditability. Cost: rerank only the top ~100 candidates; cache embeddings; graph ingest is expensive — justify it by the thematic-query need. Eval (P11): grounding/faithfulness gate; "I don't know" over hallucination in regulated domains.

Tradeoffs. GraphRAG ingest cost vs thematic-answer quality — don't build it if all queries are local (least-agentic that works). Hybrid always beats pure-vector because BM25 catches IDs/codes dense misses (Q26).

Case 7 — Secure Sandbox for AI-Generated Code

Targets: Docker, OpenAI Security. Related depth: P09, P10.

Prompt. "An agent writes code and you must run it. Design the execution environment so malicious or buggy generated code can't harm the host, other tenants, or exfiltrate data."

Requirements & numbers. Untrusted code, per-execution isolation, resource + time limits, controlled egress, fast enough for interactive use. Ask: languages, execution volume, acceptable cold-start, what the code legitimately needs to touch.

API. POST /exec with code + a capability policy (allowed fs paths, egress hosts, cpu/mem/ time) → returns stdout/stderr/exit + resource usage; hard-killed on limit breach.

Components. A capability-gated sandbox (P09): microVM (Firecracker/Kata) or gVisor per execution — not a bare container (shared kernel is not a trust boundary, Q41). Policy engine enforces fs/net/exec capabilities + resource/step limits + egress allow-list (the capability that matters most — it's the exfil path).

Data. Ephemeral per-execution filesystem (destroyed after), no ambient secrets, egress audit.

Cross-cutting. Reliability: hard timeouts, OOM-kill, no shared mutable state across runs. Security: default-deny everything; least privilege; scan outputs for exfil patterns (P10); assume the code is hostile (indirect injection may have authored it). Cost: microVM cold-start vs pooled warm sandboxes. Eval: red-team the sandbox with escape/exfil attempts as a regression suite.

Tradeoffs. Isolation strength vs latency (warm pool of microVMs). Give the code the minimum it needs — read+egress together is a data-exfiltration path, so break the combination.

Case 8 — Cost/Latency-Optimized Agent Gateway

Targets: Cohere, Citi (VP), OpenAI Infra. Related depth: P14.

Prompt. "Design a gateway in front of an agent platform that cuts cost and tail latency without hurting quality, and gives on-call the observability to debug a failed run in minutes."

Requirements & numbers. Cut $/resolved-task, hit a p95 SLO, full tracing. Ask: current $/attempt and success rate, current p95/p99, request mix, margin target (70–80%).

API. A drop-in gateway: same invoke contract, plus response headers for cache-hit, model-used, tokens, and trace-id. Streaming passthrough (SSE) to preserve time-to-first-token.

Components. Gateway with, in order: prefix cache (shared system prompt/few-shots — exact, safe) → semantic cache (tight threshold; unsafe for personalized results) → model router/cascade (cheapest model that clears a confidence bar; escalate on low confidence) → a cost meter (tokens/request) → OTel spans per LLM/tool/retrieval call → a degradation ladder under load (P14).

Data. Cache stores (keyed carefully — never cross-tenant, Q60), per-request token/cost ledger, trace store.

Cross-cutting. Reliability: design to p95/p99, not the mean; the tail pages you. Security: cache keys include tenant_id (a shared prompt cache is a leak channel, Q63). Cost: $/resolved-task = $/attempt ÷ success_rate — a cheap model that fails often is expensive. Eval: A/B the router's quality vs cost; ensure routing-down doesn't regress the golden set.

Tradeoffs. Semantic cache savings vs correctness risk (a near-miss returns a wrong answer). Aggressive down-routing saves money but can drop success rate — optimize $/resolved, not $/attempt.

Case 9 — Agentic Coding Platform (SDD)

Targets: Anthropic (Claude Code), Temporal, Redcan. Related depth: P15.

Prompt. "Design a platform where engineers hand a spec to a coding agent that plans, edits a repo, verifies, and opens a reviewable change — reliably, at team scale."

Requirements & numbers. Spec in → reviewable, tested diff out; must not break the repo; team adoption. Ask: repo size, test-suite runtime, acceptable autonomy (auto-merge vs human review), quality bar.

API. POST /task with a spec (executable/verifiable) → streams plan → applies patches → runs verify → opens a PR-style diff. Reusable commands/skills as first-class parameterized workflows.

Components. SDD loop (P15): spec → plan → apply-patch (reviewable diff, never blind overwrite) → verify (compile + tests) → iterate, bounded. Runs in a sandbox (P09). Optional durability (P08) so a long task survives a crash (Temporal's angle).

Data. Repo snapshot, the diff, test results, a skill/command registry (versioned, reusable).

Cross-cutting. Reliability: verify gate — no diff accepted without passing tests. Security: sandboxed execution, human review before merge on anything high-impact. Cost: bound plan/repair iterations; cache repo context. Eval (P11): task-success on a golden set of specs, trajectory eval on the edit path, and the Anthropic framing — quality raised, not just volume (Q69); evals-as-CI hold the bar automatically at higher velocity.

Tradeoffs. Autonomy vs review overhead — auto-merge only what the eval gate and sandbox make safe. Reusable commands trade upfront design for repeatable quality.

How to practice these

Pick a case, set a 45-minute timer, and run it out loud on a whiteboard: numbers first, API, components (mark the trust boundary), data (mark tenant_id), the four cross-cutting properties, tradeoffs. Record yourself; the tell of a Staff candidate is that they drive — ask the six numbers, name what they'd not build, and close each subsection on a metric. Then read the matching ../system-design/ walkthrough and diff your version against it.

Next: prove you can implement the pieces in 04-coding-drills.md.