Template 10 — LLM System Design Doc

Copy and fill to design an LLM system — and the exact structure to follow in a system-design interview. Pairs with interview-prep/, diagrams/.

System Design: <system name>

Author: <name> · Date: <date>

1. Requirements (clarify first — interviewers grade this)

• Functional: <what it must do>
• Scale: <QPS, users, data volume, growth>
• Latency: <p95 target; interactive vs batch>
• Quality bar: <acceptance criteria>
• Constraints: <budget, residency/privacy, compliance, on-prem?>
• Non-goals: <explicitly out of scope>

2. High-level architecture (draw it — diagrams/)

client → gateway (auth/route/guardrails) → app (workflow) → [RAG | agents | model] → providers/self-host
                                          observability + cost + audit

• Components & responsibilities: <...>

3. Key design decisions (with tradeoffs)

4. The cross-cutting four (always address these)

• Cost: cost per resolved task; levers (routing/caching) (Phase 15.05).
• Latency: TTFT/TPOT, p95 under load, streaming (Phase 12.06).
• Reliability: fallback, retries, circuit breakers, timeouts (Phase 7.07).
• Safety/security: trust boundary, guardrails (fail closed), isolation, audit (Phase 14).

5. Evaluation & observability

• Eval: golden set + metric + CI regression gate (template 02).
• Observability: traces (correlation IDs), p95/p99, cost, error rate, audit (Phase 7.08).

6. Scaling & failure modes

• Bottlenecks: <KV-cache/GPU, vector DB, provider limits> → <mitigations>.
• Failure modes: <provider outage, injection, leak, cost spike> → <containment>.

7. Rollout

• Canary + rollback; versioned models/prompts; eval-gated deploys.

Interview tip

Spend the first 5 minutes on requirements + scale, draw the architecture, then proactively address the cross-cutting four (cost/latency/reliability/safety). Naming tradeoffs and failure modes unprompted is the senior signal.

This is the structure for both real design docs and system-design rounds. Practice filling it for the diagrams/ scenarios.