LiteLLM-Style Proxy (The Self-Hosted Unifying Proxy)

Phase 8 · Document 02 · LLM Gateways Prev: 01 — OpenRouter-Style Architecture · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Where the aggregator (01) is a hosted third party, the self-hosted unifying proxy — LiteLLM being the canonical one — is the gateway you run on your own infrastructure. It's the most common real-world choice for teams that need a single OpenAI-compatible endpoint across 100+ providers and want to keep traffic, keys, and data inside their environment, define their own budgets/keys/policy, and integrate with their own observability. It's also the fastest path to "build a gateway" — you get adapters, fallbacks, budgets, and an admin layer out of the box, and only build the differentiated parts. Knowing LiteLLM's config model, virtual keys, and deployment is core gateway literacy and the foundation of the capstone gateway project (Phase 15).

2. Core Concept

Plain-English primer: a config-driven gateway you own

LiteLLM has two faces:

• a Python SDK (litellm.completion(...)) that calls 100+ providers behind one function with the OpenAI shape, and
• a Proxy Server — a deployable OpenAI-compatible HTTP gateway configured by a config.yaml.

The proxy is the gateway: you list your models (each mapping a friendly model_name to a provider+key+endpoint), set fallbacks/budgets/keys/policy, and run it. Apps point their OpenAI SDK at it and get every configured backend — including your self-hosted vLLM/Ollama (Phase 6.04, Phase 7.01) — through one endpoint, without your data leaving your infra.

The config (the heart of it)

model_list:
  - model_name: smart                       # the alias apps call
    litellm_params:
      model: anthropic/claude-...            # provider/model the proxy calls
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: smart                        # SAME alias, second deployment → load-balanced + fallback
    litellm_params:
      model: openai/gpt-...
      api_key: os.environ/OPENAI_API_KEY
  - model_name: local
    litellm_params:
      model: openai/llama3.2:3b              # your self-hosted, OpenAI-compatible backend
      api_base: http://vllm:8000/v1
      api_key: none

router_settings:
  routing_strategy: latency-based-routing    # or simple-shuffle, usage-based, least-busy
  enable_pre_call_checks: true               # skip deployments that can't fit context, etc.

litellm_settings:
  fallbacks: [{"smart": ["local"]}]          # cross-model fallback [Phase 7.07]
  num_retries: 2
  request_timeout: 30
  cache: {type: redis}                        # optional response/prompt-cache layer

general_settings:
  master_key: sk-master                       # admin key
  database_url: postgresql://...              # for keys, budgets, usage

Two ideas to internalize:

• model_name is an alias, not a model. Multiple model_list entries with the same model_name become a load-balanced + fallback pool — the router picks among them by strategy. This is how one alias spans providers/deployments.
• It's a real gateway, not just a translator. Beyond adapters, the proxy provides routing strategies (05), fallbacks/retries/cooldowns (circuit-breaker analog, Phase 7.07), virtual keys + budgets (06), caching, and hooks for logging/observability (Phase 7.08).

Virtual keys, budgets, and teams

A production differentiator: the proxy issues virtual API keys (per user/team/project) that map to budgets, rate limits, allowed models, and metadata — backed by Postgres. So you hand each team a key, cap their spend, restrict their models, and get per-key usage — the multi-tenant control plane, self-hosted (06, 09).

pip install 'litellm[proxy]'
litellm --config config.yaml --port 4000
# create a virtual key with a budget (admin API)
curl -X POST localhost:4000/key/generate -H "Authorization: Bearer sk-master" \
  -d '{"models":["smart","local"],"max_budget":50,"metadata":{"team":"research"}}'

Deployment

Production: containerized (Docker/Compose/K8s), Postgres for keys/budgets/usage, Redis for caching/rate limits, behind your LB with HA (it's a SPOF). Wire its logging to your observability stack (Prometheus/OTel/Langfuse). This is the shape of the capstone gateway (Phase 15).

Aggregator vs self-hosted proxy

Many teams use both: an aggregator for breadth/dev, a self-hosted proxy for governed production — and LiteLLM can even route to an aggregator as one of its backends.

3. Mental Model

   apps → ONE OpenAI-compatible endpoint (LiteLLM proxy, YOUR infra)
            config.yaml: model_list (alias → provider+key+endpoint) + router + fallbacks + keys
            │  same model_name across entries = LOAD-BALANCED + FALLBACK POOL
            ▼
       routing strategy → adapter [03] → backend (OpenAI/Anthropic/Google/your vLLM)
            + virtual keys → budgets/rate-limits/allowed-models (Postgres) [06]
            + caching (Redis) + logging hooks (Prometheus/OTel/Langfuse) [7.08]
   YOU own: data path · provider keys · policy · HA   ⟂   you configure each provider yourself

Mnemonic: LiteLLM = config-driven, self-hosted gateway; one model_name alias = a load-balanced/fallback pool; virtual keys = per-team budgets; you keep data + keys + policy.

4. Hitchhiker's Guide

What to look for first: the config.yaml model_list (aliases → backends), fallbacks, and virtual keys + budgets. Those give you the unified API, reliability, and multi-tenant cost control immediately.

What to ignore at first: custom callbacks, exotic routing strategies, and bespoke policy hooks — defaults plus fallbacks/budgets cover most needs. Add custom logic where you're differentiated.

What misleads beginners:

• Thinking model_name is the provider model. It's an alias; same alias across entries = a routing pool — a feature, not a mistake.
• Forgetting HA. The self-hosted proxy is your SPOF for all LLM traffic — run it redundantly with health checks (00).
• Skipping the DB. Without Postgres you lose virtual keys/budgets/usage persistence.
• Ignoring its overhead. It's a network hop + processing — keep it lean and stream through (07).
• Assuming it equals the aggregator. You must configure each provider and hold the keys — that's the point (control), but it's work.

How experts reason: they run LiteLLM (or similar) as the self-hosted control plane when data/keys must stay in-house; define aliases + fallbacks + per-team virtual keys + budgets in config; wire it to their observability; make it HA; and reserve the aggregator for breadth/dev. They treat config as code (version it, review it).

What matters in production: gateway HA + p95 overhead, virtual-key/budget correctness, fallback success, cache hit rate (Phase 7.05/7.09), and that provider keys/data never leak.

How to debug/verify: check the proxy's logs/metrics; confirm the alias resolved to the expected deployment; verify a virtual key's budget enforces; test fallback by breaking a backend.

Questions to ask: Does it support my providers + features (tools/structured/streaming)? virtual keys + budgets + RBAC? observability integration? HA story? overhead at p95?

What silently gets expensive/unreliable: an un-redundant proxy SPOF, missing budgets (runaway spend), broken usage persistence (no chargeback), and latency creep from the hop.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• LiteLLM is the canonical self-hosted unifying proxy: one OpenAI-compatible endpoint across 100+ providers, on your infra.
• It keeps data and provider keys in your environment — the key difference from a hosted aggregator (01).
• model_name is an alias; multiple entries with the same alias form a load-balanced + fallback pool.
• It's a full gateway: routing strategies, fallbacks/retries/cooldowns, virtual keys + budgets, caching, logging hooks.
• Virtual keys map per-team budgets/rate-limits/allowed-models (backed by Postgres) — multi-tenant control.
• Deploy containerized + HA (it's a SPOF) with Postgres (keys/usage) and Redis (cache/limits).
• It can route to an aggregator as one backend — they compose.
• Config-as-code: version and review config.yaml.

9. Observations from Real Systems

• LiteLLM is the most common self-hosted gateway in startups and enterprises wanting unified access without sending data to a third party.
• Virtual keys + budgets are widely used to give each team a capped key with allowed models — chargeback and governance in one (06).
• It fronts self-hosted vLLM alongside cloud APIs, so apps treat local and cloud identically (Phase 7.01).
• The capstone gateway (Phase 15) mirrors this shape: config-driven, multi-provider, metered, policy-aware.
• Teams compose both shapes: LiteLLM self-hosted for prod + governance, OpenRouter for breadth/experimentation (01).

10. Common Misconceptions

11. Engineering Decision Framework

CHOOSE the self-hosted proxy (LiteLLM) WHEN: data/keys must stay in-house, or you need custom
control, multi-tenant budgets, and your own observability.
 1. CONFIG: model_list (aliases → providers/keys/your vLLM); same alias = LB+fallback pool.
 2. RELIABILITY: fallbacks + num_retries + cooldowns; pre_call_checks on.                  [7.07]
 3. MULTI-TENANT: virtual keys → budgets/rate-limits/allowed-models (Postgres).            [06]
 4. PERF/COST: caching (Redis); stream-through; keep overhead low.                         [7.05,07]
 5. OBSERVABILITY: wire logging to Prometheus/OTel/Langfuse.                                [7.08]
 6. HA: run redundantly behind your LB (it's a SPOF).                                       [00]
 7. COMPOSE: optionally add an aggregator as a backend for breadth.                         [01]

12. Hands-On Lab

Goal

Deploy LiteLLM as a self-hosted gateway with a load-balanced/fallback pool, virtual keys + budgets, and observability — proving the self-hosted control plane.

Prerequisites

• API key(s) + a local model (Phase 6.04); Docker or pip install 'litellm[proxy]'; (optional) Postgres + Redis.

Setup

Use the config.yaml from §2 (alias smart mapped to two providers = a pool; plus local).

Steps

1. Run + unified call: start the proxy; call smart and local via the OpenAI SDK at :4000 — same client, multiple backends, data on your infra.
2. Pool behavior: with two smart deployments, send many requests and observe load balancing across them (check logs); kill one and confirm fallback continues (Phase 7.07).
3. Virtual keys + budgets: with Postgres configured, generate a virtual key with max_budget and models:["smart"]; use it; confirm it's restricted to smart and rejected when over budget (06).
4. Caching: enable Redis cache; send a repeated request and observe a cache hit (lower latency/cost) (Phase 7.05).
5. Observability: enable a logging callback (Prometheus/Langfuse); confirm per-request usage/cost/latency is emitted (Phase 7.08).

Expected output

A self-hosted OpenAI-compatible gateway showing: a load-balanced/fallback pool, a budget-restricted virtual key, a cache hit, and emitted observability — all without data leaving your infra.

Debugging tips

• Virtual keys do nothing → database_url missing (Postgres required).
• Pool not balancing → entries don't share the exact same model_name.

Extension task

Add your self-hosted vLLM (Phase 7.01) as a backend and route a data-policy-"sensitive" alias only to it (05, 09).

Production extension

Containerize with Postgres + Redis behind your LB, run two replicas for HA, and wire alerts on the gateway's own p95/error rate (Phase 7.08).

What to measure

Gateway overhead (p95), pool balancing, fallback success, budget enforcement, cache hit rate, usage attribution.

Deliverables

• A running self-hosted gateway with a pool + fallback.
• A virtual key + budget demo (model restriction + over-budget rejection).
• A cache hit + observability emission, and an HA note.

13. Verification Questions

Basic

1. What is the difference between the LiteLLM SDK and the Proxy Server?
2. What does it mean when two model_list entries share a model_name?
3. Why does a self-hosted proxy keep data in your environment?

Applied 4. Write a config.yaml for a smart pool (two providers) + a local backend with a cross-model fallback. 5. How do virtual keys implement per-team budgets and model restrictions?

Debugging 6. Virtual keys/budgets aren't enforced. What's missing? 7. All LLM traffic dropped though providers are up. What about the gateway?

System design 8. Design a self-hosted, multi-tenant gateway: pools, fallbacks, virtual keys/budgets, caching, observability, HA.

Startup / product 9. When do you self-host LiteLLM vs use an aggregator vs build your own gateway — and how do they compose?

14. Takeaways

1. LiteLLM is the self-hosted unifying proxy — one OpenAI-compatible endpoint across 100+ providers, on your infra.
2. It keeps data + provider keys in-house and is config-driven (config.yaml as code).
3. Same model_name across entries = a load-balanced + fallback pool; it's a full gateway (routing, keys, budgets, caching, logging).
4. Virtual keys + budgets deliver multi-tenant control (Postgres-backed).
5. Run it HA (it's a SPOF); it composes with aggregators and fronts your self-hosted vLLM.

15. Artifact Checklist

• A running self-hosted gateway with a load-balanced/fallback pool.
• A virtual key + budget (model restriction + over-budget rejection).
• A cache hit + observability emission.
• A config.yaml committed as code.
• An HA + overhead note for production.

Up: Phase 8 Index · Next: 03 — Provider Adapters