« System Design Index · Agentic AI Engineer · Cheat Sheet · Glossary

01 — Design a Secure, Multi-Tenant, Durable Enterprise Agent Platform

The archetype: Citi (Lead Agentic AI Engineer VP; Agentic AI Technical Lead), Docker (Staff SWE, Agentic Platform), Cohere (Senior SWE, Agent Infrastructure). The prompt: "Design the internal platform that lets many product teams ship agents to production — securely, cheaply, reliably — across many business units." This is the reference walkthrough: it runs the Agent-Platform Design Framework end to end at full depth. The other four walkthroughs are zoom-ins on components introduced here.

Table of Contents

1. Step 1 — Clarify requirements & scale
2. Step 2 — The API / contract
3. Step 3 — Architecture
4. The request lifecycle
5. Step 4 — Data & storage
6. Tenancy & isolation
7. Durability
8. The MCP tool layer
9. Reliability
10. Security
11. Evaluation
12. Cost & latency
13. Observability
14. Tradeoffs
15. Failure modes
16. Staff vs junior recap

Step 1 — Clarify requirements & scale

Open by turning the vague ask into numbers. A realistic enterprise brief:

• Tenants: ~40 internal business units (wholesale banking, payments, risk, ops), each with several product teams; a few thousand end users; strict isolation (a payments agent must never read a risk team's documents). Compliance regime: SOC2 + banking controls (GLBA-class).
• Workloads: a mix — short RAG Q&A (2–4 steps), medium tool-using assistants (5–10 steps), and long automations (20–40 steps, minutes-to-hours, some with human approval).
• Scale: peak ~200 QPS of agent invocations, ~30% are multi-step; ~5M tool calls/day; retrieval corpus ~50M chunks across tenants.
• Reliability target: T = 0.95 end-to-end for assistants; higher-stakes automations must be durable and human-gated.
• Latency: chat p95 < 6 s to final token, time-to-first-token < 1.5 s; automations async.
• Cost: target $/resolved-task under budget with 70–80% gross margin; central cost attribution per business unit.

Now the arithmetic, out loud. A 10-step assistant at p = 0.95 per step is \(0.95^{10} \approx 0.60\) — far below T = 0.95. The reliable step budget at T = 0.95, p = 0.95 is \(\lfloor \log 0.95 / \log 0.95 \rfloor = 1\) unverified step; even at p = 0.99, \(\lfloor \log 0.95 / \log 0.99 \rfloor \approx 5\). Conclusion I state immediately: beyond ~5 steps I must verify, checkpoint, or gate — the platform therefore needs critics, durable checkpoints, and human-approval signals as first-class components, not add-ons. This one paragraph frames the entire design. (Math from Phase 00.)

Step 2 — The API / contract

Two entry shapes, one envelope.

# Synchronous / streaming (chat, short RAG)
POST /v1/agents/{agent_id}/invoke        # SSE stream of tokens + tool events
  headers: Authorization: Bearer <OIDC>, Idempotency-Key, X-Trace-Id
  body:    { input, session_id, budget?: {max_steps, max_tokens, max_usd} }

# Asynchronous / durable (automations, human-gated)
POST /v1/runs                            # -> 202 { run_id }
GET  /v1/runs/{run_id}                   # -> status, trajectory, usage, finish_reason
POST /v1/runs/{run_id}/signals/{name}    # human approval / external event (Phase 08)

The request envelope the gateway constructs (never trusting the body for identity):

Envelope {
  tenant_id      # from the validated OIDC token, NOT the request body
  principal      # user/service identity + scopes (RBAC/ABAC)
  budget         # {max_steps, max_tokens, max_usd} clamped to tenant policy
  trace_id       # OTel root span id
  idempotency_key
}

Every response carries usage {input_tok, output_tok, usd, latency_ms}, the trajectory (ordered tool calls + observations, for eval/audit), and a finish_reason (stop / budget_exhausted / guardrail_blocked / needs_approval / error). The contract is the trust boundary: the caller supplies intent; the platform decides and executes.

Staff move: put budget in the contract. It makes cost a caller-visible, enforced property and lets the gateway do admission control (reject a request whose budget can't be met) before spending a dollar on tokens.

Step 3 — Architecture

                          ┌──────────────────────────────────────────────┐
  product-team client ──▶ │                   GATEWAY                      │
   (chat UI / service)    │  OIDC authN · RBAC/ABAC authZ · resolve tenant │
                          │  token-bucket quota · idempotency · admission  │
                          │  build Envelope · start OTel root span         │
                          └───────────────┬────────────────────────────────┘
                                          │ Envelope (tenant-scoped)
                          ┌───────────────▼────────────────────────────────┐
                          │              CONTROL PLANE                       │
                          │  intent router (Phase 04) → skill/agent          │
                          │  model router / cascade (Phase 14)               │
                          │  is-it-durable? ─┬─ short → runtime loop          │
                          │                  └─ long/gated → WORKFLOW ENGINE  │
                          └───┬───────────────────────────┬──────────────────┘
                              │                           │
              ┌───────────────▼─────────┐    ┌────────────▼─────────────────┐
              │      AGENT RUNTIME       │    │   DURABLE WORKFLOW ENGINE     │
              │  ReAct / ReWOO loop      │    │   event-sourced history       │
              │  step budget · critic    │    │   deterministic replay        │
              │  (Phase 01 / 07)         │    │   retries · idempotency        │
              │                          │    │   human-approval signals       │
              └───┬─────────┬────────────┘    │   (Phase 08)                  │
                  │         │                 └───────────┬───────────────────┘
        ┌─────────▼──┐  ┌───▼──────────┐   ┌──────────────▼──────────┐
        │ MCP TOOL   │  │  RETRIEVAL    │   │     MODEL ACCESS         │
        │ LAYER      │  │  hybrid+rerank│   │  provider mux + cascade  │
        │ registry · │  │  ACL-trimmed  │   │  semantic + prefix cache │
        │ authz ·    │  │  GraphRAG     │   │  cost meter (Phase 14)   │
        │ SANDBOX    │  │  (P05/P06)    │   └──────────────────────────┘
        │ (Phase 09) │  └───────────────┘
        └─────┬──────┘
              │
   ┌──────────▼────────────────────────────────────────────────────────────┐
   │                        DATA & STATE PLANE                               │
   │  event log · state KV (Redis) · vector DB [per-tenant namespace] ·      │
   │  object store · secrets vault · audit log · warehouse (traces/cost)     │
   └─────────────────────────────────────────────────────────────────────────┘
              │
   ┌──────────▼────────────────────────────────────────────────────────────┐
   │  CROSS-CUTTING: OTel traces · cost meter · guardrail chain ·           │
   │  eval harness (offline gate + online sampling) · red-team (P10/11/14)  │
   └─────────────────────────────────────────────────────────────────────────┘

The platform is a control plane (routing, durability, orchestration) over a set of shared data planes (tools, retrieval, models, state), with a gateway enforcing the trust boundary and a cross-cutting layer watching everything. Product teams register an agent spec (its tools, its retrieval namespaces, its budget, its guardrail policy) and the platform runs it — they write no runtime, no sandbox, no quota code.

The request lifecycle

Follow one 8-step assistant request from click to answer:

1. Gateway validates the OIDC token, resolves tenant_id = payments, checks the principal's RBAC scopes, checks the token bucket has budget, dedupes on the idempotency key, builds the Envelope, opens the OTel root span.
2. Control plane routes intent ("invoice reconciliation") to the right skill, picks a model tier (start on the cheap model), and — because this is short and non-mutating — chooses a request-scoped runtime loop, not a durable workflow.
3. Runtime enters the ReAct loop with max_steps = 8. Step 1: the model proposes retrieve(query). The platform validates the tool call against JSON Schema, checks the tool allow-list for payments, and calls retrieval.
4. Retrieval runs hybrid BM25 + dense over the payments namespace only (security-trimmed), RRF-fuses, reranks top-50 → top-8, returns chunks. The runtime appends them as an observation.
5. Step 2–3: the model proposes a ledger.lookup tool call. The platform validates, checks least privilege, executes it inside the sandbox (Phase 09), audits the call, returns the result as an observation.
6. A critic pass (cheap model) checks the draft answer is grounded in retrieved chunks; if not, it triggers one re-retrieve + revise (buying reliability back).
7. Output guardrail scans the final answer for PII leaks and exfil patterns (no rogue markdown-image URLs). Clean → stream tokens to the client via SSE.
8. Cross-cutting: every step emitted an OTel span; the cost meter recorded tokens and dollars against tenant_id = payments; the full trajectory was written to the audit log and sampled into the online eval stream.

For an async automation, step 2 instead instantiates a durable workflow: the same tool calls become activities recorded in an event log, so a crash at step 15 resumes at step 15, and a needs_approval step pauses until a human posts a signal.

Step 4 — Data & storage

The single design rule: tenant_id is part of every key, and it comes only from the validated token. Everything in Tenancy & isolation enforces that rule.

Tenancy & isolation

The enterprise differentiator. Three isolation models, chosen per tier (Phase 13):

• Silo — dedicated infra (DB, vector index, VPC) per tenant. Strongest isolation, highest cost; reserve for the most sensitive business units or a "compliance tier."
• Pool — shared infra, isolation enforced in software by tenant_id scoping. Cheapest, densest; the default for most tenants.
• Bridge — shared services but dedicated data stores; a middle ground.

For a bank I would offer pool by default, silo for the compliance tier, and be explicit about the four AI-specific leak channels (ranked):

1. Shared vector index (#1) — a missing tenant_id filter and tenant A retrieves tenant B's chunks. Mitigation: per-tenant namespace/collection, filter applied server-side from the Envelope, and an isolation test in CI that queries as tenant A and asserts zero tenant-B hits.
2. Prompt / semantic cache — a cache keyed only on prompt text serves tenant A's cached answer to tenant B. Mitigation: tenant_id in the cache key.
3. Agent memory — long-term memory recall crossing tenants (or a poisoned memory). Mitigation: memory keyed by tenant_id+principal; treat recalled memory as untrusted.
4. Co-mingled fine-tunes — training a shared adapter on mixed-tenant data. Mitigation: per-tenant adapters or strict data governance.

Quotas are per-tenant token buckets (requests/sec and tokens/day), so one tenant's runaway agent cannot starve another — the noisy-neighbor control.

Staff vs junior: a junior says "we'll add a WHERE tenant_id = ?." A Staff engineer says "isolation is default-deny, enforced server-side from the token, and tested — I run a CI test that tries to cross the boundary and asserts it fails. Defense I can't test is a defense I don't have."

Durability

Short chat turns are request-scoped: if the process dies, retry the turn. But automations that run for minutes, call many side-effecting tools, or wait hours for a human cannot live in process memory — a deploy or preemption would lose the work and possibly re-run side effects (re-charge, re-send). So the control plane routes them to the durable workflow engine (Phase 08, built in the durable-workflow-engine lab):

• Workflow = deterministic orchestration; activities = the side-effecting tool calls (recorded and retried). No wall-clock, random, or IO in workflow code — it must replay identically.
• Event sourcing: state is the append-only history of activity results + signals; a crash reloads the history and replays to the last recorded event, then continues live.
• Idempotency keys on activities make at-least-once retries safe.
• Human-approval signals: a needs_approval step pauses the workflow durably (for hours or days) until a human posts a signal — the durable human-in-the-loop pattern a bank needs for any money-moving action.

This is how the platform buys back reliability: a 30-step automation at p = 0.95 would be \(0.95^{30} \approx 0.21\) if run naively; as a durable workflow with retried idempotent activities and a human gate on the one irreversible step, the effective success rate is dominated by the human decision, not the compounding oracle.

The MCP tool layer

Tools are the agent's hands, and in an enterprise they are the highest-risk surface — a tool moves money or reads a customer record. The platform exposes tools over MCP (Phase 03), which gives clean, versioned integration boundaries between agents and enterprise systems. (Full treatment in Walkthrough 02; the enterprise-relevant essentials:)

• Registry & discovery: an internal catalog of MCP servers (ledger, KYC, docs, email); tools/list per tenant returns only the tools that tenant's policy allows.
• Permission gating: every tools/call is checked against the principal's scopes and the tenant allow-list before execution — least privilege, default-deny.
• Sandboxed execution: tool code runs in a capability-gated jail (Phase 09) with fs/net/exec limits, resource caps, and egress allow-lists (the anti-exfiltration control).
• Audit: every tool call — args, principal, tenant, result hash — is written to the immutable audit log. In a bank, "who did the agent call, on whose behalf, with what inputs" is a regulatory requirement, not a nice-to-have.

Reliability

• Math: assistants target T = 0.95; the reliable step budget at p = 0.95 is ~1 unverified step, so I verify or checkpoint every few steps. Levers used: ReWOO for predictable paths (fewer LLM calls, less compounding), per-step retries on idempotent tools (1 − (1−p)^(r+1) — one retry of a 0.9 tool → 0.99), a critic on grounded answers, and durable checkpoints + HITL for automations.
• Degradation ladder: strong model → cheap model → cached answer → queued/deferred → graceful "I couldn't complete this, escalating to a human." Never a hard 500 to the user.
• SLOs: end-to-end task success rate, not just uptime, is a first-class SLO — watched by the online eval stream.

Security

The trust boundary is the product: the model proposes; the platform executes after validation.

• Prompt injection (Phase 10) — direct (user), indirect (a fetched document or tool result), and memory (poisoned store). No general fix; contained by layers: input guard, tool allow-list, egress allow-list, output exfil scan (block outbound markdown-image and rogue URLs), and human approval on high-impact actions. Every fetched document and tool result is treated as untrusted data, never as instructions.
• Least privilege & sandbox — minimal tools/scopes per agent; tool code jailed (Phase 09, the capability-sandbox lab).
• Tenancy — tenant_id from the token through every layer; default-deny; per-tenant namespaces, cache keys, quotas (see Tenancy & isolation).
• Secrets — tool credentials live in a vault, injected at execution time into the sandbox, never placed in a prompt or returned to the model.

Map this to OWASP LLM Top 10 out loud: LLM01 prompt injection (guardrails + HITL), LLM02 insecure output handling (validate/scan before executing or rendering), LLM06 sensitive-information disclosure (exfil scan + tenancy), LLM08 excessive agency (least privilege + approval gates).

Evaluation

An agent you cannot evaluate, you cannot ship — so eval is part of the platform (Phase 11, full treatment in Walkthrough 05):

• Per-agent golden datasets (versioned) — each product team registers golden tasks with their agent spec. CI runs them on every prompt/model/tool change; a regression gate blocks deploy on a success-rate drop.
• Trajectory evals — grade the path (right tools, sane order, no wasted steps), not just the final answer, because two answers can be equally right but one burned 5× the tokens.
• LLM-as-judge for open-ended answers, gated by a Cohen's κ > ~0.6 agreement check against human labels, with position/verbosity/self-preference bias controls.
• Safety as a gate — any run that leaks PII or crosses a tenant boundary fails outright, regardless of task score.
• Online eval — sample production trajectories back into the harness to catch drift; success rate and guardrail-trigger rate are dashboards, not surprises.

Cost & latency

• Unit economics: report $/resolved-task = $/attempt ÷ success_rate per business unit. A model that's 3× cheaper but succeeds 60% vs 90% is more expensive per resolved task — this reframes "just use the cheap model."
• Levers (Phase 14, the agent-gateway lab):
  • Model router/cascade — cheap model first; escalate to the strong model only on low confidence or critic rejection. Typical mix routes 60–80% of traffic to the cheap tier.
  • Semantic + prefix cache — repeated/similar queries and shared system-prompt prefixes; realistic 30–60% hit-rate on enterprise FAQ-heavy traffic, each hit ~0 marginal token cost.
  • Chain-length reduction — ReWOO over ReAct where the path is predictable (2 LLM calls vs n); remember ReAct is O(n²) input tokens because the scratchpad is resent every step.
• Latency: design to p95/p99, stream tokens so time-to-first-token (< 1.5 s) is the felt latency; run independent tool calls in parallel; keep the cheap-model path on the hot path.

Observability

• Trace every run with OTel GenAI spans — one span per LLM call, tool call, retrieval — linked by trace_id, annotated with tokens, cost, latency, tenant, and finish_reason. When an agent misbehaves at 2 a.m., the trace is how you find the lossy parse, the denied tool, or the bad plan.
• Meter cost per request per tenant from day one — feeds both the invoice/showback and the per-BU $/resolved-task dashboard.
• SLO dashboards: task success rate, p95 latency, guardrail-trigger rate, cache hit-rate, quota-rejection rate, cross-tenant-isolation test status. These are the platform's vitals.

Tradeoffs

Failure modes

• Cross-tenant leak via shared vector index or prompt cache → per-tenant namespaces + cache keys, server-side filter from the token, CI isolation test. (Highest-severity for a bank.)
• Prompt-injection exfiltration — a fetched document instructs the agent to email account data → egress allow-list, output exfil scan, HITL on email.send, treat all fetched content as data.
• Reliability collapse on long automations (0.95^n) → durable checkpoints, critics, human gates; monitor task-success SLO.
• Cost blowout — a runaway loop or retry storm → per-tenant token quotas, per-run budget caps, circuit breakers, cost alerts.
• Silent quality regression from a prompt tweak → golden-set regression gate blocks the deploy.
• Durable-workflow non-determinism (someone put datetime.now() in workflow code) → replay diverges; enforce the workflow/activity split and lint for it.
• Provider outage → multi-provider routing, timeouts, degradation ladder, cached fallbacks.
• Noisy neighbor — one tenant saturates shared workers → per-tenant token buckets + fair scheduling.

Staff vs junior recap

• A junior draws boxes; a Staff engineer opens with numbers (\(0.95^{10} \approx 0.60\), reliable step budget) and lets them dictate the architecture.
• A junior treats security as input validation; a Staff engineer names the trust boundary, ranks the tenant leak channels, and writes a CI test that tries to cross them.
• A junior says "add eval later"; a Staff engineer makes the golden-set regression gate a deploy blocker and samples production for drift.
• A junior quotes one model; a Staff engineer quotes $/resolved-task, a cascade, and a cache hit-rate.
• A junior ends when the happy path works; a Staff engineer pre-mortems the design and contains each failure.

This platform is the composition of Phases 01, 03, 04, 05/06, 07, 08, 09, 10, 11, 13, and 14 — which is exactly why the capstone builds a scored miniature of it. Next: zoom into the tool layer in Walkthrough 02 — MCP Tool Platform.