Diagrams 04 — Eval, Cost, Deployment & Startup

Diagrams 15–18: the evaluation pipeline, cost observability, Kubernetes production deployment, and the full startup product architecture. Pairs with Phase 12, Phase 7, Phase 15.

15. Model evaluation pipeline

   GOLDEN SET (representative + edge + unanswerable; never trained on)
        │
        ▼
   run candidate model(s) ──▶ outputs
        │
        ▼
   SCORE:  programmatic (exact/schema/tests)  >  LLM-judge (calibrated)  >  human
        │
        ├──▶ quality metric (per task)
        ├──▶ latency (p95 under load)
        ├──▶ cost (per resolved task)
        └──▶ SAFETY gate (violation rate → pass/FAIL)   ← hard gate, not weighted
        │
        ▼
   weighted score (quality/latency/cost) + safety gate → DECISION
        │
        ▼
   CI REGRESSION GATE: no change ships unless it passes  [Phase 12.08]

flowchart TD
  GS[(Golden set)] --> RUN[Run candidate models]
  RUN --> SC[Score: programmatic > judge > human]
  SC --> QM[Quality per task]
  SC --> LAT[Latency p95]
  SC --> COST[Cost per resolved task]
  SC --> SAFE{Safety gate}
  QM --> WS[Weighted score]
  LAT --> WS
  COST --> WS
  SAFE -->|fail| BLOCK[Block]
  SAFE -->|pass| WS
  WS --> DEC[Decision]
  DEC --> CI[CI regression gate]

16. Cost observability pipeline

   each request ──▶ [ gateway/app emits: tokens_in, tokens_out, model, tenant, latency ]
        │
        ▼
   [ cost calc: tokens × price per (model,provider) ]
        │
        ▼
   [ metrics store / OTel ] ── tags: tenant, feature, model
        │
        ├──▶ dashboards: cost/request, cost/resolved-task, $/user, gross margin
        ├──▶ alerts: spend spike, budget breach, margin drop
        └──▶ per-tenant budgets / caps (enforce) [Phase 7.09]
   feeds: routing decisions (cheapest that works) + pricing [Phase 15.05]

flowchart TD
  REQ[Each request] --> EM[Emit tokens_in/out, model, tenant, latency]
  EM --> CC[Cost calc: tokens × price]
  CC --> MS[(Metrics / OTel)]
  MS --> DB[Dashboards: cost/req, cost/resolved, $/user, margin]
  MS --> AL[Alerts: spend spike, budget breach]
  MS --> BUD[Per-tenant budgets/caps]
  DB --> FEED[Feeds routing + pricing]

17. Production deployment on Kubernetes

            Internet
               │
        [ Ingress / LB ]
               │
        [ API gateway ]  auth, rate limit, routing, guardrails
          │          │
          ▼          ▼
   [ app pods ]   [ inference: vLLM pods on GPU nodes ]
   (stateless,     (HPA/KEDA autoscale on queue depth/GPU)
    autoscaled)    node pool: GPU; taints/tolerations
          │          │
          ▼          ▼
   [ vector DB ]  [ cache (Redis) ]  [ Postgres (state, RLS per-tenant) ]
          │
          ▼
   [ observability: Prometheus/Grafana + OTel traces ]  [ secrets: Vault ]
   model weights: PVC / object store; rollout: canary + rollback

flowchart TD
  NET[Internet] --> ING[Ingress/LB]
  ING --> GW[API gateway: auth/ratelimit/route/guardrails]
  GW --> APP[App pods stateless autoscaled]
  GW --> INF[vLLM pods on GPU nodes HPA/KEDA]
  APP --> VDB[(Vector DB)]
  APP --> RED[(Redis cache)]
  APP --> PG[(Postgres + RLS per-tenant)]
  INF --> OBS[Prometheus/Grafana + OTel]
  APP --> OBS
  GW --> SEC[Secrets: Vault]

18. Startup product architecture (the capstone shape)

   users (web/mobile/IDE)
        │
   [ product app: owns the WORKFLOW + system of record ]   ← not a wrapper [Phase 15.03]
        │
   ┌───────────────────────────────────────────────────────┐
   │  LLM PLATFORM                                          │
   │  gateway (routing/fallback/metering/keys) [Phase 8]     │
   │  RAG (ingest/retrieve/rerank/cite) [Phase 9]           │
   │  agents (tools, trust boundary) [Phase 10]             │
   │  eval harness (golden set, CI gate) [Phase 12]         │
   │  guardrails + audit + tenant isolation [Phase 14]      │
   │  observability + cost (per resolved task) [Phase 7/15] │
   └───────┬───────────────────────────┬────────────────────┘
           ▼                           ▼
   commercial APIs              self-hosted (vLLM/local)
   (frontier, hard tail)        (cheap bulk, private/residency)
   moat: data flywheel + workflow + domain + distribution [Phase 15.06]

flowchart TD
  U[Users: web/mobile/IDE] --> APPP[Product app: owns workflow + system of record]
  APPP --> PLAT[LLM platform]
  PLAT --> GWY[Gateway: routing/fallback/metering]
  PLAT --> RAGB[RAG]
  PLAT --> AGT[Agents: trust boundary]
  PLAT --> EVAL[Eval harness + CI gate]
  PLAT --> SEC[Guardrails + audit + isolation]
  PLAT --> OBSV[Observability + cost]
  GWY --> COM[Commercial APIs: hard tail]
  GWY --> SELF[Self-hosted vLLM/local: cheap bulk, private]
  APPP --> MOAT[Moat: data flywheel + workflow + domain + distribution]

Back to: Diagrams index · This last diagram is the shape of the Capstone. Concepts: Phase 12.08, Phase 7.10, Phase 15.