Routing and Fallbacks

Phase 7 · Document 07 · Production Serving Prev: 06 — Streaming · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A single model behind a single provider is a single point of failure and a fixed point on the cost-quality-latency frontier (Phase 5.09). Routing (send each request to the best model/provider for it) and fallbacks (automatically reroute when one fails) are how production systems get reliability (survive provider outages and rate limits) and economics (cheap model for easy requests, premium for hard ones) at the same time. This is the core logic of every LLM gateway (Phase 8) — OpenRouter, LiteLLM, and every serious in-house platform. Without it, one provider blip is a customer-facing outage and your bill is whatever the most expensive model costs for every request.

2. Core Concept

Plain-English primer: pick the right backend, and have a Plan B

Two related ideas:

• Routing = choosing, per request, which model + which provider/endpoint handles it — based on the request's needs and your constraints.
• Fallback = recovering, when the chosen backend fails (error, rate limit, timeout, or low quality), by automatically trying the next option in a chain.

Both exist because the same model is often available from multiple places (first-party API, aggregators, your self-hosted vLLM — Phase 5.10), and different requests have different needs (easy vs hard, cheap vs quality, public vs sensitive).

Routing strategies

You route on request attributes and policy:

The highest-leverage one is difficulty-based cost routing: most traffic is easy, so routing easy→cheap and hard→premium yields a blended cost/quality far better than any single model (Phase 5.09). A classifier or rules decide difficulty.

Fallback chains and the failure types

A fallback chain is an ordered list tried until one succeeds:

Primary:  anthropic/claude-sonnet           (first-party)
  → on RATE LIMIT (429)   : openai/gpt-...            (different provider, similar quality)
  → on UNAVAILABLE (5xx)  : openrouter/anthropic/...  (same model, different host)
  → on ALL FAIL / TIMEOUT : local/qwen-...            (self-hosted safety net)

Different failures want different responses:

• Transient (timeout, 5xx, network): retry with backoff (and jitter) a couple times before failing over.
• Rate limit (429): fail over to another provider/key immediately (retrying the same one just waits).
• Hard error (400 bad request, context too long): don't retry/failover blindly — it'll fail everywhere; fix the request.

Circuit breakers, health checks, load balancing

To avoid hammering a sick provider:

• Circuit breaker: after N consecutive failures, open the circuit (stop sending to that provider for a cooldown), routing straight to fallbacks; half-open probes to test recovery, then close when healthy. This prevents cascading latency from retrying a dead provider.
• Health checks: periodic probes mark providers up/down so routing skips known-bad ones (Phase 8).
• Load balancing: spread traffic across healthy instances/keys (round-robin, least-loaded, weighted) — and respect per-key rate limits.

Idempotency and streaming caveats

• Mid-stream failover is hard: once you've streamed tokens to the client (06), you can't cleanly switch models and "rewind." Fail over before first token where possible; after that, surface an error.
• Avoid double-charging/double-acting: retries must be safe (the model call itself is read-only, but tool side effects in agents are not — guard those, Phase 10).

3. Mental Model

   request → ROUTER ──pick model+provider by: cost · latency · quality · capability · policy · health
                  │           (difficulty-based cost routing = biggest lever, Phase 5.09)
                  ▼
            PRIMARY backend ── success → stream back [06]
                  │ fail?
                  ├ transient (timeout/5xx) → RETRY w/ backoff (a couple times)
                  ├ rate limit (429)        → FAIL OVER to next in chain (don't retry same)
                  └ hard error (400)        → STOP (will fail everywhere; fix request)
            FALLBACK CHAIN: A → B → C → local safety net
            CIRCUIT BREAKER: N fails → open (skip provider) → half-open probe → close
   reliability (survive outages) + economics (cheap-where-possible) in one layer = the GATEWAY [Phase 8]

Mnemonic: route by need + policy; retry transient, fail-over on rate-limit, stop on bad-request; trip a breaker on a sick provider; fail over before first token.

4. Hitchhiker's Guide

What to look for first: a fallback chain (you need one before launch) and a difficulty/cost routing rule (your biggest economics lever). Then a circuit breaker so failures don't cascade.

What to ignore at first: elaborate ML-based routers. Start with rules (by task type, by required capability, by data policy) + a static fallback chain; add learned routing later if data justifies it.

What misleads beginners:

• Retrying everything. Retrying a 429 or a 400 wastes time/quota — match the response to the failure type.
• Fallback to a worse model silently. If you fail over to a weaker model, quality drops without anyone noticing — surface it and eval the fallback (Phase 5.06).
• Mid-stream failover. Can't rewind streamed tokens — fail over before first token.
• Ignoring provider variance. The "same model" on a fallback host may be quantized/capped differently (Phase 5.10).
• No circuit breaker. Retrying a dead provider on every request tanks p95 for everyone.

How experts reason: they design a typed failure policy (retry transient w/ backoff+jitter, fail-over on 429, stop on 4xx), a fallback chain across providers/hosts (not just keys), difficulty-based cost routing for economics, circuit breakers + health checks for stability, and observability on route decisions, fallback rates, and per-route quality (08). They eval the fallback models too, since you will serve from them.

What matters in production: fallback success rate, how often you're on the primary vs fallback (a creeping fallback rate signals a degrading primary), per-route quality and cost, circuit-breaker state, and that data-policy routing is enforced (sensitive data never leaves the allowed boundary).

How to debug/verify: chaos-test by killing the primary and confirming reroute < a token or two; inject 429s and confirm immediate failover; check breaker opens after N failures; verify sensitive requests never hit a disallowed provider.

Questions to ask vendors/gateways: supported routing keys? fallback + retry + circuit-breaker config? per-provider health checks? region/data-policy routing? does it expose route/fallback metrics?

What silently gets expensive/unreliable: silent quality drops on fallback, retry storms against a dead provider (no breaker), creeping fallback rate (unnoticed primary degradation), and policy leaks (sensitive data routed to a public API).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Routing chooses the backend per request; fallback recovers when it fails — together they give reliability + economics.
• Difficulty-based cost routing is the biggest economics lever (most traffic is easy) (Phase 5.09).
• Match the response to the failure type: retry transient (backoff+jitter), fail over on 429, stop on 4xx.
• Circuit breakers prevent cascades — stop sending to a provider after N failures; half-open to probe recovery.
• Fail over before the first token — you can't cleanly rewind a stream (06).
• Fallback models must be evaluated — you will serve from them, and a weaker fallback silently drops quality.
• The fallback "same model" may differ (quant/cap by host) — verify fidelity (Phase 5.10).
• A creeping fallback rate signals a degrading primary — monitor it (08).

9. Observations from Real Systems

• OpenRouter routes across many upstream providers for a model ID and supports model fallbacks + provider preferences — production routing as a service (Phase 8).
• LiteLLM offers retries, fallbacks, cooldowns (a circuit-breaker analog), and load balancing across deployments/keys (what-happens §3.5).
• Coding tools / agent platforms route by sub-task: fast model for autocomplete, strong model for planning, code model for edits (Phase 11).
• Most LLM outages are survived, not prevented, by a tested fallback chain + circuit breaker — the discipline that turns a provider incident into a non-event (10).
• Enterprise gateways enforce data-policy/region routing so regulated data never reaches a disallowed provider (Phase 14).

10. Common Misconceptions

11. Engineering Decision Framework

DESIGN routing + fallback:
 1. ROUTE (per request):
      capability needed (tools/JSON/long-ctx/multimodal)? → filter to capable models   [5.06,5.07]
      data sensitive / region-bound?                       → policy route (local/region) [Phase 14]
      else cost route by DIFFICULTY: easy→cheap, hard→premium                            [5.09]
      tie-break by latency/health/load.
 2. FALLBACK CHAIN: primary → similar-quality other provider → same model other host → local net.
 3. FAILURE POLICY: transient→retry(backoff+jitter, cap); 429→failover; 4xx→stop.
 4. STABILITY: circuit breaker (N fails→open→half-open probe) + health checks + load balance.
 5. STREAMING: fail over BEFORE first token; guard tool side effects on retry.            [06,Phase 10]
 6. OBSERVE: route mix, fallback rate, per-route quality+cost, breaker state.              [08]
 7. EVAL the fallback models — you will serve from them.

12. Hands-On Lab

Goal

Build a small router + fallback layer with a circuit breaker, and prove it survives a provider outage and routes by difficulty.

Prerequisites

• Two endpoints (e.g., a managed API + a local vLLM/Ollama Phase 6.04), or use LiteLLM. pip install litellm openai.

Setup

# Option A: LiteLLM router with fallbacks
from litellm import Router
router = Router(model_list=[
    {"model_name": "smart", "litellm_params": {"model": "anthropic/claude-...", "api_key": "..."}},
    {"model_name": "smart", "litellm_params": {"model": "openai/gpt-...", "api_key": "..."}},   # fallback
    {"model_name": "cheap", "litellm_params": {"model": "openai/gpt-...-mini", "api_key": "..."}},
], fallbacks=[{"smart": ["cheap"]}], num_retries=2, timeout=30)

Steps

1. Fallback on failure: point the primary at a deliberately-broken key/URL; send requests and confirm they reroute to the fallback and still succeed. Log which backend served each.
2. Failure-type policy: simulate a 429 (rate limit) vs a 500 (transient) vs a 400 (bad request); confirm your layer fails over on 429, retries on 500, and stops on 400.
3. Difficulty routing: classify each request easy/hard (length or a tiny classifier) and route easy→cheap, hard→smart. Over a mixed workload, compute the blended cost vs always-smart; show savings (Phase 5.09).
4. Circuit breaker: make the primary fail repeatedly; confirm after N failures the breaker opens (requests skip it for a cooldown) and later half-opens to probe recovery.
5. Quality check the fallback: run your golden set on the fallback model; record any quality delta so a failover isn't a silent regression (Phase 1.07).

Expected output

Logs showing per-request backend choice; a failure-type policy table (retry/failover/stop); blended-cost savings from difficulty routing; circuit-breaker state transitions; and a fallback quality delta.

Debugging tips

• It retries a 429 forever → you're not distinguishing failure types.
• Fallback never triggers → exception not caught / chain misconfigured.

Extension task

Add data-policy routing: tag some requests "sensitive" and force them to the local model only; assert they never hit the cloud provider.

Production extension

Move this into the Phase 8 gateway with health checks, per-key load balancing, and route/fallback metrics on a dashboard (08).

What to measure

Backend mix; fallback success rate; blended cost vs single-model; breaker transitions; fallback quality delta.

Deliverables

• A router + fallback + circuit breaker implementation.
• A failure-type policy table (retry/failover/stop).
• A blended-cost result from difficulty routing + a fallback quality note.

13. Verification Questions

Basic

1. What's the difference between routing and fallback?
2. Which failure types should you retry, fail over, or stop on?
3. What does a circuit breaker do and why?

Applied 4. Design a fallback chain for a chatbot that must survive a provider outage; justify the order. 5. Why is difficulty-based routing the biggest cost lever, and how do you implement it?

Debugging 6. p95 latency spikes whenever a provider has a partial outage. What's missing? 7. After a failover, users complain quality dropped. What did you skip?

System design 8. Design routing + fallback + circuit breaker + data-policy routing for a multi-tenant gateway.

Startup / product 9. How do routing and fallbacks simultaneously improve gross margin and reliability — and what's the risk if the fallback isn't evaluated?

14. Takeaways

1. Routing picks the backend per request; fallbacks recover on failure — reliability + economics in one layer.
2. Difficulty-based cost routing (easy→cheap, hard→premium) is the biggest economics lever.
3. Match response to failure type: retry transient, fail over on 429, stop on 4xx.
4. Circuit breakers + health checks prevent cascades; fail over before first token.
5. Evaluate fallback models and watch the fallback rate + per-route quality/cost.

15. Artifact Checklist

• A router + fallback chain + circuit breaker implementation.
• A failure-type policy table (retry/failover/stop).
• A blended-cost result from difficulty routing.
• A fallback quality delta (eval'd, not silent).
• (Optional) data-policy routing enforcement test.

Up: Phase 7 Index · Next: 08 — Observability