Routing Engine

Phase 8 · Document 05 · LLM Gateways Prev: 04 — Model Registry · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The routing engine is the brain of the gateway — the component that, for every request, decides which model and which provider to use. It's where the gateway delivers its two headline values at once: economics (cheap model for easy work, premium for hard) and reliability (fail over when a backend dies). Phase 7.07 covered routing/fallback as serving concepts; this doc is the gateway component that implements them: how aliases like auto/coding resolve, how candidates are filtered then selected, how data-policy routing enforces compliance, and how it composes with the registry, metering, and policy engine. A weak router overpays, breaks on outages, or leaks sensitive data to the wrong provider; a good one is invisible and saves real money.

2. Core Concept

Plain-English primer: filter, then select

Routing is a two-stage pipeline over the registry's candidates:

1. Filter (hard constraints): drop every candidate that can't serve this request — wrong capability (no tools/JSON/vision/long-context), over budget, violates data policy, or is inactive/unhealthy. These are non-negotiable; a candidate either qualifies or it doesn't (Phase 5.00).
2. Select (soft preference): among the survivors, pick the best by a strategy — cheapest, fastest, highest quality, or a weighted score — plus load/health tie-breaks.

async def route(req, rules, registry, ctx):
    candidates = registry.candidates_for(req.model)          # alias → models [04]
    eligible = [c for c in candidates if
        meets_capabilities(c, req)        # tools/JSON/vision/context  [04]
        and meets_data_policy(c, ctx)     # residency/sensitivity      [09]
        and under_budget(c, ctx.user)     # spend guard                [06]
        and is_healthy(c)]                # circuit breaker/health      [7.07]
    if not eligible:
        raise NoEligibleProvider()        # fail closed — don't route to an unfit/non-compliant model
    return select(eligible, rules.strategy, req)             # cheapest/fastest/quality/score

The order is the lesson: filter (correctness/compliance) before select (optimization). Selecting first and hoping it qualifies is how sensitive data leaks or a tool call hits a non-tool model.

Aliases: routing by intent

Apps shouldn't hardcode model IDs; they call aliases that express intent — auto/coding, auto/fast, auto/quality, auto/cheap. The registry's routes table maps each alias → ordered candidates + constraints, and the engine resolves it at request time. This decouples apps from models: swap the model behind auto/coding centrally and every app benefits, no code change (00).

Selection strategies

The biggest economic lever remains difficulty-based routing: classify easy vs hard and route easy→cheap, hard→premium for a blended cost far below always-premium (Phase 7.07, Phase 5.09).

Data-policy routing (a hard filter, not a preference)

A request tagged sensitive (PII, regulated, region-bound) must route only to allowed backends (e.g., self-hosted/local, or an in-region provider) — enforced as a filter so it can never be overridden by a cheaper/faster option. This is the compliance backbone and ties directly to the policy engine and Phase 14. Fail closed: if no compliant candidate exists, reject — don't route to a non-compliant one.

Routing + fallback are one loop

Selection picks the first backend; fallback (Phase 7.07) handles failure by moving to the next eligible candidate, with the typed-failure policy (retry transient, fail over on 429, stop on 4xx) and a circuit breaker to skip sick providers. So routing isn't a single decision — it's select → try → on failure, re-select from the remaining eligible set, before the first streamed token (07).

3. Mental Model

   request (alias e.g. auto/coding) → REGISTRY candidates [04]
        │
        ├─ FILTER (hard, fail-closed): capabilities · DATA POLICY [09] · budget [06] · health [7.07]
        │      └ no eligible → REJECT (never route to an unfit/non-compliant model)
        ▼
        SELECT (soft): cheapest | fastest | quality | weighted | latency/least-busy
                       biggest lever = DIFFICULTY routing (easy→cheap, hard→premium) [5.09]
        ▼
        TRY backend → on failure: typed policy (retry/failover/stop) + circuit breaker
                      → re-SELECT from remaining eligible (fallback)  [7.07]
   aliases decouple apps from models: change the model behind auto/coding centrally

Mnemonic: filter (correctness/compliance, fail-closed) → select (optimize) → try → fallback. Apps call intent aliases; data-policy is a filter, never a preference.

4. Hitchhiker's Guide

What to look for first: the filter-before-select order and the data-policy filter. Those guarantee correctness and compliance; everything else is optimization.

What to ignore at first: ML-based routers and elaborate scoring. Start with capability filters + a cheapest/quality strategy + difficulty rules + a fallback chain.

What misleads beginners:

• Selecting before filtering. Optimizing first risks picking an ineligible/non-compliant model — filter is the gate.
• Data policy as a soft preference. It must be a hard filter that fails closed, or sensitive data leaks under cost pressure (09).
• Hardcoding model IDs in apps. Use aliases so models can change centrally (00).
• Forgetting fallback is part of routing. A single pick with no re-selection on failure = an outage (Phase 7.07).
• Routing on stale facts. Capabilities/prices come from the registry — stale registry, wrong routes.

How experts reason: they implement routing as filter (capability + policy + budget + health) → select (strategy) → try → fallback, expose intent aliases, make data-policy a fail-closed filter, use difficulty routing for economics, and drive it all from the registry. They log the route decision (chosen model/provider, why) for debugging and observability (Phase 7.08).

What matters in production: route-decision correctness (especially policy), blended cost from difficulty routing, fallback success, p95 added by routing logic, and observability of why each request went where (Phase 7.08).

How to debug/verify: log/trace the candidate set, the filter that eliminated each, and the selected backend; assert sensitive requests only ever hit allowed providers; A/B difficulty routing's blended cost vs single-model.

Questions to ask: are filters applied before selection? is data policy fail-closed? what strategies are supported? how does fallback re-select? is the decision logged?

What silently gets expensive/unreliable: select-before-filter (compliance leaks), no difficulty routing (overpay), soft data policy (data leaks), unlogged decisions (undebuggable), and routing on a stale registry.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Routing is filter (hard) → select (soft) — drop ineligible candidates before optimizing.
• Data-policy routing is a fail-closed hard filter, never a preference — reject if no compliant candidate (09, Phase 14).
• Apps call intent aliases (auto/coding); the registry routes table resolves them — models change centrally.
• Difficulty-based routing is the biggest economics lever (easy→cheap, hard→premium) (Phase 5.09).
• Routing and fallback are one loop: select → try → re-select on failure with a typed-failure policy + circuit breaker (Phase 7.07).
• It runs on registry facts — stale capabilities/prices ⇒ wrong routes (04).
• It coordinates with budget (filter) and metering (06).
• Log the route decision (chosen backend + why) for debugging/observability (Phase 7.08).

9. Observations from Real Systems

• OpenRouter's auto routing + provider preferences and LiteLLM's routing strategies (latency/usage/least-busy) are production implementations of this engine (01, 02).
• Coding platforms route by sub-task — fast model for autocomplete, strong for planning, code model for edits — the alias pattern in action (Phase 11).
• Enterprise gateways enforce data-policy routing so regulated data only reaches approved/in-region backends (09, Phase 14).
• Difficulty routing is widely used to cut spend 30–70% on mixed workloads while holding quality (Phase 5.09).
• Routing incidents trace to soft policy filters (leaks), stale registry facts (bad routes), or no fallback re-selection (outages).

10. Common Misconceptions

11. Engineering Decision Framework

ROUTE a request:
 1. RESOLVE alias → candidate models (registry routes).                          [04]
 2. FILTER (hard, fail-closed):
       capabilities (tools/JSON/vision/context) · DATA POLICY (residency/sensitivity) [09]
       · under budget [06] · healthy (circuit breaker) [7.07]
       none eligible → REJECT.
 3. SELECT (soft) by strategy:
       cost-sensitive → cheapest | real-time → fastest | accuracy → quality | balanced → weighted
       mixed traffic → DIFFICULTY routing (easy→cheap, hard→premium) [5.09]
 4. TRY backend → on failure apply typed policy (retry/failover/stop) + re-select eligible. [7.07]
 5. LOG the decision (chosen model/provider + why) for observability.            [7.08]

12. Hands-On Lab

Goal

Implement a filter→select→fallback routing engine over your registry, with aliases, difficulty routing, and a fail-closed data-policy filter.

Prerequisites

• The registry from 04 (or a small in-memory list of candidates with caps/prices); pip install openai for live calls (optional).

Steps

1. Candidates + aliases: define auto/coding, auto/cheap, auto/quality → ordered candidates from the registry routes table.
2. Hard filters: implement capability, budget, health, and data-policy filters; assert that a sensitive=true request returns only allowed (e.g., local) candidates, and rejects if none — fail closed.
3. Selection strategies: implement cheapest, fastest (use mock/measured p50), and quality; route the same request under each and show different picks.
4. Difficulty routing: classify a mixed workload easy/hard (length or a tiny classifier); route easy→cheap, hard→premium; compute blended cost vs always-premium (Phase 5.09).
5. Fallback loop: make the selected backend fail (429/5xx/4xx); confirm typed handling (retry/failover/stop) and re-selection from remaining eligible (Phase 7.07).
6. Decision log: for each request, log candidate set, eliminations (with reason), and the chosen backend.

Expected output

A router that: resolves aliases, filters correctly (incl. fail-closed data policy), selects by strategy, applies difficulty routing with measured savings, and re-selects on failure — all with a readable decision log.

Debugging tips

• Sensitive request reached a cloud model → data policy was a soft preference, not a filter, or applied after selection.
• Difficulty routing didn't save → easy fraction small or the cheap model fails the quality bar (Phase 5.09).

Extension task

Add a weighted score strategy (Σ wᵢ·normalizedᵢ over cost/quality/latency) and compare its picks to single-axis strategies (Phase 5.00).

Production extension

Wire it into your gateway with registry-driven candidates, circuit breakers, and route-decision metrics on a dashboard; A/B difficulty routing on live traffic (Phase 7.08).

What to measure

Filter correctness (esp. data policy), per-strategy picks, blended cost from difficulty routing, fallback success, route-decision observability.

Deliverables

• A filter→select→fallback routing engine with aliases.
• A fail-closed data-policy demonstration.
• A difficulty-routing blended-cost result + a decision log.

13. Verification Questions

Basic

1. What are the two stages of routing, and which comes first?
2. Why must data-policy routing be a fail-closed filter?
3. What is a routing alias and why use one?

Applied 4. Implement the candidate filter for a tool-calling, in-region, under-budget request. 5. Why is difficulty routing the biggest cost lever, and how do you implement it?

Debugging 6. A sensitive request reached a disallowed provider. What went wrong in the engine? 7. Difficulty routing increased cost-per-resolved-task. Why might that happen?

System design 8. Design a routing engine with aliases, filters (incl. data policy), strategies, and fallback for a multi-tenant gateway.

Startup / product 9. How does the routing engine simultaneously drive gross margin and compliance, and what's the risk of getting the order wrong?

14. Takeaways

1. Routing = filter (hard, fail-closed) → select (soft) → try → fallback re-select.
2. Data-policy routing is a hard filter, never a preference — reject if no compliant candidate.
3. Apps call intent aliases; the registry resolves them so models change centrally.
4. Difficulty routing is the biggest economics lever; pick a strategy per use case.
5. Routing and fallback are one loop, run on registry facts, with the decision logged for observability.

15. Artifact Checklist

• A filter→select→fallback routing engine with aliases.
• A fail-closed data-policy filter demonstration.
• Multiple selection strategies + a difficulty-routing blended-cost result.
• A fallback re-selection with typed-failure handling.
• A route-decision log/metric for observability.

Up: Phase 8 Index · Next: 06 — Usage Metering