System Design — Reference AI/LLM Architectures

Worked, senior-level designs that tie the whole track together. Each one states the workload numbers first, picks a model strategy (the FLOPs/memory/cost math), designs the serving path and the offline path, then quantifies latency, cost, and quality and names the tradeoff. Use them as templates; the capstone (P19) builds one end-to-end.

The design method (every time)

  1. Workload facts first: task + modality, QPS / peak concurrency, prompt and response token sizes, latency SLO (TTFT and TPOT separately), quality bar + eval, safety/compliance, budget.
  2. Two paths, always:
    • Serving (online) — tokenizer → model (quantized? sharded?) → sampler/constraints → KV-cache/PagedAttention → continuous batching → autoscaling → guardrails → observability.
    • Offline — data → fine-tune/align (PEFT/DPO) → quantize → eval gate → registry → deploy; plus the RAG indexing pipeline if there is retrieval.
  3. Quantify: tokens/sec per GPU, KV-cache bytes at peak concurrency, $/1M tokens, p99 TTFT/TPOT, recall@k, eval score with a confidence interval.
  4. Tradeoff: quality vs latency vs cost vs memory vs safety — name what you trade and write the one-sentence decision record.
  5. Fail it: hallucination, prompt injection, tool misuse, KV-cache OOM, cost blowup, model regression, drift — and the rollback.

Design 1 — High-throughput LLM serving platform (the canonical one)

Requirement: serve a 13B chat model, 2k concurrent users, p99 TTFT < 1s, smooth streaming (TPOT < 50ms), minimize $/1M tokens.

Model strategy

  • Open-weights 13B, int8 (or int4-AWQ) weights ⇒ ~13 GB → ~7–13 GB; fits one 24–48 GB GPU with room for KV-cache. Quantize after measuring eval impact, not before.

Serving path

  • vLLM-class server: PagedAttention KV-cache (blocked, prefix-shared) + continuous batching to keep the GPU busy; tensor-parallel across 2 GPUs only if a single one can't hold weights+KV at target concurrency.
  • Sampler: temperature 0.7 / top_p 0.9 for chat; a separate temperature=0 + constrained path for tool/JSON endpoints.
  • Autoscale on queue depth / KV-cache utilization, not CPU. Admission-control + reject with 429 past the SLO-preserving batch size.

Quantify

  • KV-cache: 2 · L · T · d_model · bytes per sequence — at 2k concurrency this, not the weights, sets the GPU count. Show the number; it's the whole capacity plan.
  • Throughput: batching amortizes the per-token weight read; report tokens/sec/GPU and $/1M.

Tradeoffs

  • Quality vs cost: int4 + draft-model speculative decoding cuts $/token but must pass the eval gate. Latency vs throughput: cap the batch at the largest size that still meets p99.

Design 2 — Production RAG over a 10M-document corpus (P11, P16)

Requirement: grounded Q&A with citations, p99 < 2s end-to-end, "I don't know" instead of hallucination, nightly corpus refresh.

  • Indexing (offline): chunk (semantic, ~512 tokens, overlap), embed, build an HNSW (or IVF-PQ if memory-bound) index in a vector DB (FAISS/Milvus/Pinecone/Weaviate) with metadata filters; version the index so a bad rebuild can roll back.
  • Query (online): embed → ANN top-k (k≈20) → cross-encoder rerank to top-5 → assemble a grounded prompt with citations → constrained answer that must cite or abstain.
  • Eval: retriever (recall@k, nDCG) and generator (faithfulness, citation accuracy) measured separately; gate deploys on both.
  • Tradeoff: bigger k + reranker raises recall and cost/latency; abstention raises trust but lowers coverage — tune to the business cost of a wrong answer.
  • Fail it: stale index, embedding/model version skew, prompt injection inside retrieved docs (treat retrieved text as untrusted), citation hallucination.

Design 3 — Multimodal VLM assistant (P04)

Requirement: answer questions about uploaded images + text, single model, p99 < 3s.

  • Architecture: ViT vision encoder → projector → 13B LLM (LLaVA pattern); images become prepended tokens. Image tokens inflate the context — budget the KV-cache for them.
  • Adaptation: freeze encoders, train the projector + a LoRA on the LLM for the domain; cheap and preserves pretraining.
  • Serving: vision encode is a separate compute-bound stage (cache by image hash); the LLM decode is the memory-bound stage — pipeline them.
  • Tradeoff: more image patches = better grounding, more tokens = more latency/cost; pick resolution to the task.

Design 4 — Production agent with tools (P12–P14, P16)

Requirement: an agent that resolves customer tickets using internal tools (lookup, refund, escalate); must be safe, auditable, and bounded in cost.

  • Loop: ReAct with a hard step cap and a token budget; typed JSON-schema tools with input validation; errors returned as observations so the agent can recover.
  • Safety: refund/irreversible tools require human approval; all tools idempotent so a retry can't double-act; a critic pass (or neurosymbolic verifier) checks the proposed action against business rules before execution.
  • Memory: conversation buffer → summary → semantic recall of similar past tickets.
  • Observability: log every thought/action/observation with a trace ID; eval on a fixed ticket suite (pass@k, tool-call accuracy, cost/ticket).
  • Tradeoff: more autonomy ⇒ more failure surface; constrain with verification + approvals. Multi-agent only if planner/executor/critic roles genuinely separate.

Design 5 — Train/fine-tune & align a domain model (P03, P05, P07, P10)

Requirement: adapt an open 7B model to a domain, align it to a house style, ship it quantized.

  • Pipeline: SFT on curated instructions (LoRA) → preference data → DPO (closed-form, stable) → quantize (int4-AWQ) → eval gate (domain benchmark + safety + calibration) → registry → canary deploy.
  • Compute: if full fine-tune is needed, ZeRO/FSDP across the data-parallel group; otherwise QLoRA on a single GPU. Track every run in W&B/MLflow with the exact data + config.
  • Tradeoff: DPO is simpler/cheaper than PPO-RLHF but less online exploration; LoRA is cheap but caps how far you can move the model — measure the gap.

Design 6 — Edge / on-device deployment (P06, P17)

Requirement: run a small assistant on a device with no cloud round-trip, limited RAM, battery budget.

  • Strategy: distill to a small student → int4 quantize → compile to the target runtime (ONNX Runtime / TensorRT / GGUF); KV-cache is the memory budget, so cap context.
  • Tradeoff: every bit of quantization and every dropped parameter trades quality for battery/RAM/latency; measure on the device, not the dev box. Latency may be a safety property if it drives an actuator (P15).

Anti-patterns the panel listens for

  • Quoting a model size without the KV-cache memory at target concurrency.
  • "We'll fine-tune" when RAG or a better prompt solves it cheaper — or RAG when the knowledge is actually reasoning, not retrieval.
  • Sampling (temperature>0) for a structured/extraction endpoint instead of constrained decoding.
  • One eval number with no confidence interval; testing on the training distribution.
  • An agent with no step cap, no idempotency, no human gate on irreversible actions.
  • Claiming a latency/throughput number without saying whether it's TTFT or TPOT, prefill or decode, batched or single-stream.
  • Multi-agent / multi-GPU / multi-region with no number justifying the added complexity.