Capstone — Build a Startup-Grade LLM Platform

The final project that integrates the entire curriculum. One repository that proves you can interpret terminology, read model cards, choose models, estimate cost/memory, run local models, serve models, build routing/fallback, build RAG, build agents, evaluate models, operate in production, and build a startup product on top. This is the portfolio centerpiece — for every role (interview-prep/).

What you're building

An OpenAI-compatible LLM platform with a product on top — the shape in diagrams/18:

users → product app (owns a workflow) → LLM platform (gateway + RAG + agents + eval + security + cost)
                                            ├─ commercial APIs (hard tail)
                                            └─ self-hosted / local (cheap bulk, private)

Build incrementally — each component maps to a phase you've already learned. It does not need to be huge; it needs to be coherent, measured, and defensible.

The 18 components

Suggested repo layout

capstone/
  gateway/            # 1,2,3,6 — adapters, registry, routing, fallback, OpenAI-compatible API
    adapters/         #   openai_adapter, anthropic_adapter, local_adapter (4), vllm_adapter (5)
    registry.py       #   model × provider × price × limits × capabilities
    router.py         #   cost/latency/difficulty routing + fallback + retries
    server.py         #   /v1/chat/completions (OpenAI-compatible), streaming
    metering.py       #   per-tenant token + cost accounting (8)
    guardrails.py     #   input/output checks, fail closed (16)
  rag/                # 10 — ingest, hybrid search, rerank, grounded generation, tenant isolation
  agent/              # 12 — tool-calling loop, trust boundary, approval gate, tracing
  coding/             # 11 — index, symbol search, apply-patch, latency tiers
  eval/               # 9  — golden set, scorers, judge, regression gate (CI)
  ui/                 # 7,8 — model selection + usage/cost dashboard
  product/            # 13 — the MVP workflow app (owns a system of record)
  deploy/             # 14 — docker-compose / k8s manifests
  docs/
    ARCHITECTURE.md   # 15 — diagrams
    SECURITY.md       # 16 — filled security checklist
    RUNBOOK.md        # 17 — incident runbook
    NARRATIVE.md      # 18 — investor/demo narrative
  README.md           # the numbers (p95, cost/resolved-task, eval scores, apply-rate)

Build phases (incremental)

1. Gateway core (1–6): provider adapters (OpenAI + one more + local) → registry → OpenAI-compatible server with routing + fallback + streaming. Proof: swap providers, survive an outage, route easy→cheap.
2. RAG (10): ingest a corpus → hybrid search + rerank + grounded citations + tenant isolation. Proof: retrieval recall@k + generation faithfulness numbers.
3. Agent (12): tool-calling loop with the trust boundary + approval gate + tracing. Proof: per-step reliability + a passing injection red-team.
4. Eval (9): golden set + scorers + judge + a CI regression gate that fails the build on a drop. Proof: the gate catches a deliberate regression.
5. Cost + UI (7,8): model-selection view + usage/cost dashboard (per tenant, per resolved task). Proof: cost attribution + a routing-savings number.
6. Coding demo (11): index + apply-patch + latency tiers. Proof: apply-rate + latency per tier.
7. Product (13): wrap one workflow as an MVP that owns a system of record (not a wrapper). Proof: the wrapper check passes; a before→after demo.
8. Ops + docs (14–18): deployment, architecture diagrams, security checklist, runbook, and the narrative.

Proof checklist — the capstone must demonstrate you can:

• Interpret LLM terminology (README + narrative use it correctly).
• Read model cards (registry captures context/price/capabilities/license).
• Choose models (selection UI + a memo on your eval).
• Estimate cost (cost dashboard: per request / resolved task / tenant).
• Estimate memory (a KV-cache/VRAM calc for the self-hosted model).
• Run local models (local adapter works).
• Serve models (vLLM adapter / serving path with p95 under load).
• Build routing/fallback (router survives a provider outage; routes by cost/difficulty).
• Build RAG (recall@k + faithfulness numbers; tenant isolation tests).
• Build agents (trust boundary + approval; per-step reliability; injection red-team).
• Evaluate models (golden set + CI regression gate).
• Operate in production (runbook + dashboards + canary/rollback).
• Build a startup product (MVP owning a workflow + unit economics + narrative).

The numbers your README must report

p95 latency · cost per resolved task · gross margin · RAG recall@k + faithfulness · agent per-step reliability · code apply-rate · eval score + a caught regression · routing cost savings · a passing prompt-injection red-team.

Why this matters: in interviews, "I built a startup-grade LLM platform — here's the repo, here are the numbers" is the single most impressive thing you can show. It proves the entire curriculum at once. See the 12-month roadmap Month 10 and your role's portfolio project.

Components & docs

• Architecture (component 15) · Security (16) · Runbook (17) · Investor/Demo Narrative (18)