Diagrams 02 — Serving & Gateways

Diagrams 7–10: how models are served locally and in production, and how gateways sit in front of providers. Pairs with Phase 6, Phase 7, Phase 8.

7. Local inference stack

        Your app  (OpenAI SDK pointed at localhost)
            │  HTTP (OpenAI-compatible /v1/chat/completions)
            ▼
   ┌────────────────────────────────────────────┐
   │  Local engine: Ollama / llama.cpp / LM Studio / MLX  │
   │  ┌──────────────────────────────────────┐  │
   │  │ model weights (GGUF / safetensors)    │  │
   │  │ quantized (Q4_K_M / Q8 / FP16)        │  │
   │  └──────────────────────────────────────┘  │
   │  KV-cache + activations                     │
   └───────────────┬─────────────────────────────┘
                   ▼
   ┌────────────────────────────────────────────┐
   │ Hardware: CPU + RAM | GPU + VRAM | Apple unified memory │
   │  speed ≈ memory bandwidth ÷ model bytes      │
   └────────────────────────────────────────────┘

flowchart TD
  App[Your app: OpenAI SDK → localhost] --> Eng[Local engine: Ollama/llama.cpp/MLX]
  Eng --> W[Weights GGUF/safetensors, quantized]
  Eng --> KV[KV-cache + activations]
  Eng --> HW[Hardware: CPU+RAM / GPU+VRAM / unified memory]
  HW --> Law[tok/s ≈ bandwidth ÷ model bytes]

8. vLLM serving stack

  many clients ──▶ [ OpenAI-compatible API server (:8000) ]
                            │
                            ▼
                  [ Scheduler: CONTINUOUS BATCHING ]
                   adds/removes requests each step
                            │
                            ▼
        [ PagedAttention KV-cache manager ]  (blocks, no fragmentation)
        [ Prefix cache ]  (shared system prompt / RAG context reused)
                            │
                            ▼
              [ Model executor on GPU(s) ]
              (tensor-parallel across GPUs if model > 1 GPU)
                            │
                            ▼
              streamed tokens (SSE) ──▶ clients
   metrics: TTFT, TPOT, p95/p99, throughput, GPU util

flowchart TD
  C[Clients] --> API[OpenAI-compatible server :8000]
  API --> SCH[Scheduler: continuous batching]
  SCH --> PA[PagedAttention KV manager]
  SCH --> PC[Prefix cache]
  PA --> EX[Model executor on GPUs]
  PC --> EX
  EX -->|tensor-parallel if needed| EX
  EX --> STR[Stream tokens SSE] --> C

9. OpenRouter-style gateway (aggregator)

   client (one API key, OpenAI-compatible)
        │
        ▼
   ┌──────────────────────────────────────────────┐
   │           AGGREGATOR GATEWAY                   │
   │  auth → model registry (model × provider)      │
   │  routing (price/latency/availability)          │
   │  fallback across providers                     │
   │  usage metering + billing                      │
   │  streaming passthrough                         │
   └───────┬───────────┬───────────┬────────────────┘
           ▼           ▼           ▼
       OpenAI      Anthropic    Together/Fireworks...
   (one model may be served by many providers → pick best)

flowchart TD
  CL[Client: one key, OpenAI-compatible] --> GW[Aggregator gateway]
  GW --> AUTH[Auth] --> REG[Model registry: model×provider]
  REG --> RT[Routing: price/latency/availability]
  RT --> FB[Fallback across providers]
  FB --> P1[OpenAI]
  FB --> P2[Anthropic]
  FB --> P3[Together/Fireworks]
  GW --> MET[Usage metering + billing]

10. LiteLLM-style proxy (self-hosted)

   your services
        │  (OpenAI format)
        ▼
   ┌──────────────────────────────────────────────┐
   │     SELF-HOSTED PROXY (you run it)            │
   │  virtual keys (per team/tenant) + budgets      │
   │  provider adapters (real keys server-side)     │
   │  routing + fallback + retries                  │
   │  caching · rate limits · guardrails/PEP        │
   │  logging / spend tracking / audit              │
   └───────┬───────────┬───────────┬────────────────┘
           ▼           ▼           ▼
        OpenAI      Azure       self-hosted vLLM

flowchart TD
  S[Your services: OpenAI format] --> PX[Self-hosted proxy]
  PX --> VK[Virtual keys + budgets per team]
  PX --> AD[Provider adapters: real keys server-side]
  AD --> R[Routing + fallback + retries]
  PX --> G[Cache · rate limit · guardrails/PEP]
  PX --> L[Logging · spend · audit]
  R --> O[OpenAI]
  R --> AZ[Azure]
  R --> V[Self-hosted vLLM]

Next: Diagrams 03 — RAG, Agents & Coding · Concepts: Phase 7.01, Phase 8.01–8.02