Capstone — Architecture (Component 15)
The architecture document for the Startup-Grade LLM Platform. Reuse the diagrams/ and the system-design template.
System overview
users (web / IDE / API)
│
┌─────────────▼──────────────┐
│ PRODUCT APP (owns workflow │ ← not a wrapper [15.03]
│ + system of record) │
└─────────────┬──────────────┘
│ OpenAI-compatible
┌─────────────▼──────────────────────────────┐
│ GATEWAY │
│ auth · rate limit · ROUTER · FALLBACK │
│ provider adapters · model registry │
│ metering · cache · guardrails · audit │
└───┬──────────────┬───────────────┬──────────┘
▼ ▼ ▼
OpenAI/Anthropic vLLM (GPU) local (Ollama)
features that call the gateway:
┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐
│ RAG │ │ AGENT │ │ CODING │ │ EVAL │
└─────────┘ └─────────┘ └──────────┘ └──────────┘
cross-cutting: observability · cost · security/isolation
(See diagrams/18 for the Mermaid version; diagrams/17 for the K8s deployment.)
Components & responsibilities
| Component | Responsibility | Key files |
|---|---|---|
| Gateway | One OpenAI-compatible entry; route/fallback/meter/guard | gateway/server.py, router.py |
| Provider adapters | Normalize OpenAI/Anthropic/vLLM/local to one shape | gateway/adapters/ |
| Model registry | model × provider × price × limits × capabilities | gateway/registry.py |
| RAG | ingest → hybrid retrieve → rerank → grounded cite; per-tenant filter | rag/ |
| Agent | tool-calling loop; trust boundary; approval; tracing | agent/ |
| Coding | index + symbol search + apply-patch + latency tiers | coding/ |
| Eval | golden set + scorers + judge + CI gate | eval/ |
| Product | the MVP workflow owning a system of record | product/ |
| Observability/cost | traces (correlation IDs), p95/p99, cost dashboards | gateway/metering.py, ui/ |
Key design decisions (record as ADRs)
| Decision | Choice | Why |
|---|---|---|
| Interface | OpenAI-compatible | clients/tools interoperate; easy provider swap |
| Routing | difficulty/cost-based + fallback | margin + resilience 8.05 |
| Knowledge | RAG (not fine-tune) | facts change; behavior via prompt 13.00 |
| Serving | hybrid API + self-host | cost/residency cheatsheet 05 |
| State | Postgres + RLS, per-tenant vector namespaces | isolation 14.04 |
| Trust boundary | model proposes, app executes | safety 14.01 |
Data flow (one request, with correlation ID)
request (cid) → gateway auth/route → [RAG retrieve (tenant filter) | agent tool loop]
→ model call (provider/local) → guardrails (out) → response (streamed)
→ metering + audit log (cid) [reconstructable end-to-end]
Cross-cutting concerns (always addressed)
- Cost: per request/resolved-task/tenant; routing + caching (15.05).
- Latency: TTFT/TPOT, p95 under load, streaming (12.06).
- Reliability: fallback, retries, circuit breakers, timeouts (7.07).
- Security: trust boundary, isolation, secrets, fail-closed guardrails, audit (SECURITY.md).
Scaling & failure modes
- Bottlenecks: KV-cache/GPU (serving), vector DB (RAG), provider limits → mitigations in the runbook.
- Failure modes: provider outage (fallback), injection (trust boundary), leak (isolation), cost spike (caps).