Provider Adapters

Phase 8 · Document 03 · LLM Gateways Prev: 02 — LiteLLM-Style Proxy · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A provider adapter is the component that lets one gateway speak to many providers: it translates between the gateway's unified (OpenAI-compatible) format and each provider's specific API — request shape, response shape, streaming events, error codes, and capability quirks. Adapters are why you can swap anthropic/claude-... for openai/gpt-... by changing a string (00). Get them right and the gateway is genuinely provider-agnostic; get them wrong and you ship subtle bugs — dropped system prompts, mangled tool calls, wrong token counts, broken streams. This is the most mechanical and the most error-prone part of a gateway, so it rewards a clean interface and a conformance test suite.

2. Core Concept

Plain-English primer: a translator per provider

Providers agree on the idea of chat completions but disagree on the details. The OpenAI shape puts the system prompt as a message with role:"system"; Anthropic takes a top-level system field and only user/assistant messages. Token counts are prompt_tokens/completion_tokens (OpenAI) vs input_tokens/output_tokens (Anthropic). Streaming is OpenAI's choices[].delta vs Anthropic's typed content_block_delta events (what-happens §10). Tool-calling schemas, stop reasons, and error codes differ too.

An adapter hides all of that behind a uniform interface. The gateway works in one internal format; each adapter maps internal → provider on the way in and provider → internal on the way out.

class LLMProvider(ABC):
    name: str
    @abstractmethod
    async def chat(self, req: ChatRequest) -> ChatResponse: ...
    @abstractmethod
    async def stream_chat(self, req: ChatRequest) -> AsyncIterator[Chunk]: ...
    @abstractmethod
    def estimate_cost(self, usage, model) -> float: ...      # uses the registry [04]
    @abstractmethod
    async def health_check(self) -> bool: ...

The four things an adapter normalizes

1. Request mapping. Internal request → provider request. The classic gotcha: pulling the system message out for Anthropic.

# internal (OpenAI-shaped) → Anthropic
def to_anthropic(req):
    system = next((m.content for m in req.messages if m.role == "system"), None)
    msgs   = [{"role": m.role, "content": m.content} for m in req.messages if m.role != "system"]
    return {"model": req.model, "max_tokens": req.max_tokens, "system": system, "messages": msgs}

2. Response mapping. Provider response → internal, including usage (the token-name differences) and stop/finish reasons (stop_reason:"end_turn" ↔ finish_reason:"stop", max_tokens ↔ length).

def from_anthropic(resp):
    return ChatResponse(
        id=resp.id, model=resp.model,
        content="".join(b.text for b in resp.content if b.type == "text"),
        input_tokens=resp.usage.input_tokens,           # → prompt_tokens
        output_tokens=resp.usage.output_tokens,          # → completion_tokens
        finish_reason={"end_turn":"stop","max_tokens":"length"}.get(resp.stop_reason, resp.stop_reason))

3. Streaming normalization. Each provider's event stream → one internal chunk stream (and capture trailing usage). This is subtle enough to get its own doc (07).

4. Capability + error normalization. Map tool-calling formats, structured-output modes, and error codes to a common taxonomy (rate-limit / transient / bad-request) so routing/fallback can act on typed failures (Phase 7.07). Capability facts (does this model support tools?) live in the registry; the adapter handles the wire format.

The leaky-abstraction reality

Adapters aim for a clean uniform surface, but providers leak: some support features others don't (extended thinking, parallel tool calls, certain structured-output modes), and forcing everything into a lowest-common-denominator shape can hide useful capabilities or, worse, silently drop fields. Mature gateways therefore (a) pass through provider-specific params when asked (extra_body), (b) record capability facts in the registry so routing only sends supported requests (04/05), and (c) ship a conformance test suite that runs the same scenarios against every adapter to catch regressions.

Don't reinvent: LiteLLM is a library of adapters

A key practical point: LiteLLM/OpenRouter already implement 100+ adapters. Building your own from scratch is rarely worth it — use their adapter layer and only write a custom adapter for a niche/internal provider they don't cover. Understanding adapters still matters because you debug through them and occasionally extend them.

3. Mental Model

   GATEWAY works in ONE internal (OpenAI-shaped) format
        │  per provider, an ADAPTER maps:
        ├ REQUEST   internal → provider  (e.g., pull out `system` for Anthropic)
        ├ RESPONSE  provider → internal  (usage names, stop/finish reasons)
        ├ STREAM    provider events → internal chunks (+ trailing usage)   [07]
        └ CAPS/ERRORS  tools/structured/codes → common taxonomy (rate-limit/transient/4xx) [7.07]
   leaky abstraction: pass through provider-specific params; record caps in REGISTRY [04];
                      run a CONFORMANCE SUITE across all adapters
   DON'T reinvent: LiteLLM/OpenRouter ship 100+ adapters — extend, don't rebuild

Mnemonic: one internal format, a translator per provider for request/response/stream/caps+errors; providers leak, so pass-through + registry + conformance tests.

4. Hitchhiker's Guide

What to look for first: the interface (chat/stream/cost/health) and the request+response+usage mapping for each provider — especially the system-message and token-name gotchas. Then error normalization so fallback works.

What to ignore at first: building adapters from scratch — use LiteLLM's. Focus on understanding the mapping and writing conformance tests.

What misleads beginners:

• The system-prompt trap. Forgetting to lift system out for Anthropic-style APIs → the instruction is dropped or mis-placed → degraded behavior.
• Token-name mismatch. input_tokens vs prompt_tokens → broken usage/cost metering (06).
• Lowest-common-denominator loss. Flattening everything can silently drop tools/thinking/structured-output features — pass provider-specific params through.
• Unmapped errors. If a provider's 429 isn't normalized, routing/fallback can't react correctly (Phase 7.07).
• No conformance tests. Adapters silently drift as provider APIs change.

How experts reason: they define a narrow internal format, implement (or reuse) adapters that fully map request/response/usage/stream/errors, pass through provider-specific params for non-universal features, keep capability facts in the registry, and guard everything with a conformance suite run in CI. They prefer reusing LiteLLM's adapter layer over hand-rolling.

What matters in production: correct usage/cost mapping (billing), correct stop-reason handling (truncation detection), normalized errors (fallback), capability accuracy (don't send tools to a non-tool model), and conformance tests catching API drift.

How to debug/verify: diff the adapter's output against the provider's raw response; assert usage/finish-reason/tool-call fields survive the round trip; run the conformance suite; check that extra_body params reach the provider.

Questions to ask: does the gateway/library cover my providers + features? how are errors/usage normalized? can I pass provider-specific params? is there a conformance suite?

What silently gets expensive/unreliable: dropped system prompts, miscounted tokens (billing drift), lost capabilities, unmapped errors (failed fallback), and adapter drift after provider API changes.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• An adapter normalizes four things: request, response (incl. usage), streaming, and capabilities/errors.
• The system-message mapping (OpenAI message vs Anthropic top-level field) is the classic adapter bug.
• Token-name differences (input/output vs prompt/completion) must be mapped or billing breaks (06).
• Stop/finish reasons differ (end_turn/max_tokens ↔ stop/length) — map them to detect truncation.
• Errors must be normalized to a common taxonomy so routing/fallback can react (Phase 7.07).
• The abstraction leaks — pass provider-specific params through; keep capability facts in the registry.
• LiteLLM/OpenRouter ship 100+ adapters — reuse them; hand-roll only niche/internal providers.
• A conformance suite in CI catches adapter drift as provider APIs evolve.

9. Observations from Real Systems

• LiteLLM is fundamentally a giant adapter library — its value is the 100+ normalized providers behind one shape (02).
• OpenRouter normalizes provider differences so its single endpoint works across vendors (01).
• The Anthropic system-field and event-stream deltas are the most common mapping surprises engineers hit.
• Capability leaks (extended thinking, parallel tool calls, structured-output modes) are handled via param pass-through + registry flags (04, Phase 2.09).
• Provider API changes silently break adapters — teams that run conformance tests catch it before customers do.

10. Common Misconceptions

11. Engineering Decision Framework

ADAPTERS:
 1. REUSE first: LiteLLM/OpenRouter cover 100+ providers — use their adapter layer.        [01,02]
 2. Define a NARROW internal format (OpenAI-compatible).
 3. For any custom/niche provider, implement the interface and map:
      request (mind `system`), response+usage (token names), stream [07], stop reasons, errors [7.07].
 4. LEAKS: pass provider-specific params (extra_body); record capability facts in registry. [04]
 5. CONFORMANCE: run the same scenario suite (chat, tools, JSON, streaming, errors) across all adapters in CI.
 6. ROUTE only to capable providers (registry-driven), so adapters never get unsupported requests. [05]

12. Hands-On Lab

Goal

Implement (or inspect) adapters for two providers behind one internal interface, and write a conformance suite that proves request/response/usage/error parity.

Prerequisites

• Keys for two providers (e.g., OpenAI + Anthropic), or use LiteLLM to compare; pip install openai anthropic litellm.

Steps

1. Define the interface: the LLMProvider ABC from §2 (chat/stream/cost/health) with internal ChatRequest/ChatResponse.
2. OpenAI adapter: map internal → OpenAI and back; confirm content, usage (prompt/completion_tokens), and finish_reason.
3. Anthropic adapter: map internal → Anthropic (lift the system message to the top-level field) and back (input/output_tokens → prompt/completion, end_turn→stop). This is the core lesson.
4. Conformance suite: write tests that run the same scenarios — a plain chat, a max_tokens-truncated request (assert finish_reason normalization), a tool-call (assert tool fields survive), and a forced rate-limit/error (assert your normalized error type) — against both adapters and assert identical internal shapes.
5. Compare to LiteLLM: call the same two providers via litellm.completion(...) and confirm it produces the same normalized fields — motivating reuse.

Expected output

Two working adapters behind one interface plus a passing conformance suite that catches the system-prompt and token-name traps, demonstrating provider-agnostic behavior.

Debugging tips

• Anthropic ignores your instructions → you left system as a message instead of the top-level field.
• Cost is wrong → token-name mapping; assert input_tokens→prompt_tokens.

Extension task

Add a third provider (Gemini) and extend the conformance suite; add capability flags to the registry so a tool request never routes to a non-tool model (04/05).

Production extension

Wire the adapters into your gateway with error normalization feeding routing/fallback, and run the conformance suite in CI to catch provider API drift.

What to measure

Field parity across adapters (content/usage/finish_reason/tool calls); normalized error types; conformance pass rate.

Deliverables

• Two adapters behind one interface (with the system/token mappings).
• A conformance suite covering chat/truncation/tools/errors.
• A note: which providers you'd reuse LiteLLM for vs hand-roll.

13. Verification Questions

Basic

1. What four things does a provider adapter normalize?
2. What's the classic system-message mapping bug?
3. Why must token names be mapped?

Applied 4. Map an Anthropic response (usage + stop_reason) to the OpenAI-compatible internal format. 5. Why pass provider-specific params through instead of flattening to a common shape?

Debugging 6. A model ignores the system prompt only on one provider. Diagnosis. 7. Fallback doesn't trigger on a provider's rate limit. What's unmapped?

System design 8. Design an adapter layer + conformance suite that stays correct as provider APIs evolve.

Startup / product 9. When is it worth hand-rolling adapters vs reusing LiteLLM's, and what's the maintenance cost either way?

14. Takeaways

1. Adapters translate one internal (OpenAI-compatible) format ↔ each provider, covering request, response+usage, streaming, and capabilities/errors.
2. Watch the system-message and token-name traps — they silently break behavior and billing.
3. Normalize errors so routing/fallback works; pass provider params through so features aren't lost.
4. Keep capability facts in the registry; route only to capable providers.
5. Reuse LiteLLM's 100+ adapters; guard everything with a conformance suite in CI.

15. Artifact Checklist

• An adapter interface + ≥2 implemented (or inspected) adapters.
• Correct request/response/usage/stop-reason mapping (incl. system + token names).
• Error normalization feeding fallback.
• A conformance suite (chat/truncation/tools/errors) in CI.
• A reuse-vs-build decision per provider.

Up: Phase 8 Index · Next: 04 — Model Registry