Streaming Proxy

Phase 8 · Document 07 · LLM Gateways Prev: 06 — Usage Metering · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The streaming proxy is where the gateway earns its keep on the data plane: it must forward tokens from any provider to the client as they're generated, in one unified (OpenAI-compatible) stream shape, while correctly handling client disconnects (to stop paying for abandoned generations, 06), mid-stream errors, and usage capture — across providers whose stream formats differ (OpenAI deltas vs Anthropic typed events). Phase 7.06 taught streaming for a single endpoint; the gateway adds the hard multi-provider twist: normalize heterogeneous upstream streams into one client-facing stream without buffering and without losing usage. Get it wrong and you ship janky UIs, leak cost on abandoned streams, and lose billing data — the three classic streaming-proxy failures.

2. Core Concept

Plain-English primer: one client stream, many upstream shapes

Streaming itself (SSE, tokens-as-born, perceived-latency win) is covered in Phase 7.06. The gateway's added job: the upstream could be OpenAI (choices[].delta), Anthropic (typed message_start/content_block_delta/message_delta/message_stop), Google, or your vLLM — each a different event stream. The proxy must translate each upstream's events into one consistent client-facing stream (usually OpenAI-shaped deltas) on the fly, token by token. This is the adapter's streaming half, applied in real time.

UPSTREAM (varies)                          GATEWAY normalizes →   CLIENT (one shape)
OpenAI:    data: {choices[].delta.content}          ─┐
Anthropic: event: content_block_delta {text_delta}  ─┼─→ data: {choices[].delta.content}  (OpenAI SSE)
vLLM:      data: {choices[].delta.content}          ─┘   … data: {usage} … data: [DONE]

The five hard parts (now multi-provider)

The Phase 7.06 checklist still applies, with gateway-specific sharpening:

1. Forward, don't buffer — flush each normalized chunk downstream immediately. A buffering hop (gzip, framework wrapper, Nginx proxy_buffering on, a serverless platform) silently collapses streaming. The #1 bug.
2. Normalize per upstream — translate each provider's events to the client shape as they arrive (incremental, stateful parsing of typed-event streams like Anthropic's).
3. Cancel on client disconnect → abort upstream — if the client closes, propagate cancellation to the upstream request, or you keep generating (and paying, 06) and holding the backend's KV/batch slot (Phase 7.03).
4. Mid-stream errors as in-stream events — you already sent 200, so an upstream failure after first token must surface as an error event (and, importantly, you generally cannot fail over mid-stream — fail over before first token, 05/Phase 7.07).
5. Capture usage from the final event — parse each provider's terminal event (OpenAI final chunk with include_usage; Anthropic message_delta/message_stop) to record tokens/cost for metering. Different per provider — the adapter must know each.

A normalizing streaming proxy (shape)

async def proxy_stream(req, provider, client_send, is_disconnected, on_done):
    set_sse_headers()                         # text/event-stream, no-cache, keep-alive; buffering OFF
    usage = None
    try:
        async for ev in provider.stream(req):           # provider yields RAW upstream events
            if await is_disconnected():
                await provider.cancel()                  # ③ abort upstream → stop cost/slot
                break
            chunk, maybe_usage = provider.to_openai_chunk(ev)   # ② normalize per provider [03]
            if maybe_usage: usage = maybe_usage          # ⑤ capture terminal usage
            if chunk: await client_send(chunk)           # ① forward + FLUSH immediately
        await client_send("data: [DONE]\n\n")
    except UpstreamError as e:
        await client_send(sse_error(e))                  # ④ in-stream error (200 already sent)
    finally:
        on_done(usage)                                   # record usage even on disconnect/error [06]

Why mid-stream failover is (mostly) impossible

Once the first token is streamed, the client has partial output; you can't cleanly switch to another model and "rewind" without restarting the response (which the user already saw begin). So the gateway's reliability contract is: fail over before the first token (05), and after that, a failure is an in-stream error the client must handle. This is why TTFT and pre-first-token routing/health checks matter so much.

Operational concerns

• Timeouts + keep-alive: generations run 30s+; the gateway's read timeout and your LB/CDN idle limits must allow long-lived SSE — send heartbeats/comments if needed (Phase 7.06).
• Backpressure: if the client reads slowly, don't grow unbounded buffers.
• Concurrency: each stream is a long-lived connection holding a goroutine/task + upstream connection — size the gateway for concurrent streams, not just RPS.

3. Mental Model

   many UPSTREAM stream shapes (OpenAI delta · Anthropic typed events · vLLM …)
        │  GATEWAY normalizes per-provider [03], in real time, into ONE client SSE stream
        ▼
   ① forward+FLUSH (no buffering)  ② normalize each upstream  ③ client disconnect → CANCEL upstream
   ④ mid-stream error → in-stream EVENT (no mid-stream failover; fail over BEFORE first token [05/7.07])
   ⑤ capture usage from each provider's TERMINAL event → metering [06]
   ops: long timeouts + keep-alive (LB idle limits) · backpressure · size for concurrent STREAMS

Mnemonic: one client stream from many upstream shapes — normalize+flush, cancel-on-disconnect, in-stream errors (no mid-stream failover), capture per-provider terminal usage.

4. Hitchhiker's Guide

What to look for first: that streams flush end-to-end (no buffering), that disconnect cancels upstream, and that usage is captured per provider's terminal event. Those are the three failure points.

What to ignore at first: WebSockets/gRPC — SSE is the standard; and exotic backpressure tuning until you have working forward+cancel+usage.

What misleads beginners:

• Assuming all upstreams stream alike. Anthropic's typed events need stateful incremental parsing; OpenAI's deltas don't. The proxy must normalize per provider (03).
• A buffering hop. Gzip/framework/Nginx/serverless buffering collapses streaming — the #1 bug (Phase 7.06).
• Trying to fail over mid-stream. You can't rewind streamed tokens — fail over before first token (05).
• Losing usage on streams. Each provider's terminal event differs; miss it and metering is blind (06).
• Not canceling on disconnect. Abandoned generations cost money + hold backend capacity (Phase 7.03).

How experts reason: they implement streaming as a per-provider normalizer that forwards+flushes immediately, propagates client cancellation to the upstream, surfaces mid-stream errors as events, and captures the terminal usage for each provider — and they keep the reliability contract (failover before first token). They size the gateway for concurrent long-lived streams and set generous timeouts.

What matters in production: no buffering anywhere, correct per-provider normalization (incl. usage), disconnect→cancel (cost/capacity), in-stream error handling, long-lived-connection limits (LB/CDN), and concurrent-stream capacity.

How to debug/verify: curl -N through the gateway for each provider and confirm tokens trickle in the unified shape; kill the client mid-stream and confirm the upstream stops; verify usage recorded for streamed requests on each provider; check LB idle timeout doesn't cut long streams.

Questions to ask: does it normalize every provider's stream (incl. usage)? forward without buffering? cancel upstream on disconnect? what's the long-lived-connection/idle-timeout config? concurrent-stream capacity?

What silently gets expensive/unreliable: a buffering hop (broken streaming, unnoticed in tests), no disconnect-cancel (cost leak), lost streaming usage (billing drift), and LB idle-timeouts cutting long streams.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The gateway must normalize each provider's stream (OpenAI deltas vs Anthropic typed events) into one client-facing SSE shape (03).
• Forward + flush immediately — a buffering hop silently disables streaming (the #1 bug, Phase 7.06).
• Client disconnect must cancel the upstream — else cost + backend capacity leak (06, Phase 7.03).
• Mid-stream errors are in-stream events (200 already sent); you cannot fail over mid-stream — fail over before first token (05).
• Capture usage from each provider's terminal event for metering (06) — the event differs per provider.
• Long-lived SSE needs long timeouts + keep-alive; LB/CDN idle limits must allow 30s+ streams.
• Size the gateway for concurrent streams (long-lived connections), not just RPS.
• LiteLLM normalizes streaming across providers — reuse it rather than hand-rolling every provider.

9. Observations from Real Systems

• OpenRouter and LiteLLM are, at core, normalizing streaming proxies — turning many providers' streams into one OpenAI-shaped stream (01, 02, what-happens §3.5).
• The "streaming broke" incident is almost always a buffering layer (gzip/serverless/Nginx) at some hop (Phase 7.06).
• Cost-leak incidents trace to no disconnect-cancellation: abandoned generations keep running upstream (06).
• Billing-drift incidents trace to uncaptured streaming usage — different terminal events per provider (06).
• Coding agents stream tool_use/edits through the gateway; the harness parses streamed tool args before executing (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD/OPERATE the gateway streaming path:
 1. REUSE LiteLLM's normalized streaming where possible; else implement per-provider normalizers. [02,03]
 2. DATA PLANE: forward + FLUSH each normalized chunk; disable buffering at EVERY hop
    (app, Nginx proxy_buffering off, CDN/serverless).                                  [7.06]
 3. CANCEL: client disconnect → abort upstream (stop cost + free backend slot).        [06,7.03]
 4. ERRORS: mid-stream → in-stream error event; NO mid-stream failover (fail over before T1). [05,7.07]
 5. USAGE: capture each provider's TERMINAL event → metering.                          [06]
 6. INFRA: long read timeouts + keep-alive/heartbeats; raise LB/CDN idle limits; size for
    concurrent long-lived streams.

12. Hands-On Lab

Goal

Build a normalizing streaming proxy that forwards two providers' streams in one OpenAI shape, cancels upstream on disconnect, and captures per-provider usage.

Prerequisites

• Two streaming backends with different shapes (e.g., an OpenAI-compatible one + Anthropic), or use LiteLLM; fastapi/httpx.

Steps

1. Unified client shape: define the OpenAI-style chunk you'll emit to clients.
2. OpenAI-shaped upstream: proxy a vLLM/OpenAI stream — forward deltas, curl -N to confirm tokens trickle (no buffering).
3. Anthropic upstream → normalize: consume Anthropic's typed events (content_block_delta etc.) and translate each to the OpenAI delta shape in real time; confirm the client sees identical-shaped output for both providers (03).
4. Disconnect → cancel: detect client disconnect and abort the upstream httpx stream; verify the upstream stops (vLLM num_requests_running drops / provider usage stops) (06).
5. Terminal usage: capture usage from each provider's terminal event (OpenAI final chunk with include_usage; Anthropic message_delta/message_stop) and record a usage event (06).
6. Mid-stream error: inject an upstream error after the first chunk; emit an in-stream error event and close; confirm the client handles truncation (and that you did not try to fail over mid-stream).

Expected output

A proxy that streams both providers in one shape (no buffering), cancels upstream on disconnect, records usage for both, and surfaces mid-stream errors as events.

Debugging tips

• Anthropic output looks wrong/garbled to the client → you didn't translate typed events to the OpenAI delta shape.
• Tokens arrive all-at-once → a buffering hop; curl -N and disable buffering.

Extension task

Add heartbeats + a generous read timeout so a 60s+ generation survives an LB idle limit; load-test concurrent streams to find the gateway's connection ceiling.

Production extension

Reuse LiteLLM's normalized streaming behind your gateway and wire terminal-usage capture into metering + cost dashboards (Phase 7.08).

What to measure

TTFT through the gateway per provider; tokens trickle (not buffered); upstream stops on disconnect; usage captured per provider; truncation handled on error.

Deliverables

• A normalizing streaming proxy (two differently-shaped upstreams → one client shape).
• A disconnect→cancel proof + per-provider usage capture.
• A mid-stream error handled as an in-stream event.

13. Verification Questions

Basic

1. What extra job does a gateway streaming proxy have vs a single-endpoint one?
2. Why can't you fail over mid-stream?
3. Where is usage in a streamed response, and why is it provider-specific?

Applied 4. Describe normalizing Anthropic's typed events into OpenAI deltas in real time. 5. Why must a client disconnect cancel the upstream?

Debugging 6. Streaming works for OpenAI but the Anthropic path looks garbled to clients. Cause. 7. Streamed requests record no usage. Cause and fix.

System design 8. Design the gateway streaming path: normalization, forward/flush, cancel, errors, usage, timeouts, concurrency.

Startup / product 9. How do disconnect-cancellation and accurate streaming usage protect your unit economics?

14. Takeaways

1. The gateway normalizes many upstream stream shapes into one client SSE stream in real time (03).
2. Forward+flush (no buffering), cancel upstream on disconnect, in-stream errors, capture per-provider terminal usage — the five hard parts, multi-provider.
3. No mid-stream failover — fail over before first token (05).
4. Long-lived SSE needs long timeouts + keep-alive; size for concurrent streams.
5. Reuse LiteLLM's normalized streaming; capture usage into metering.

15. Artifact Checklist

• A normalizing streaming proxy (≥2 differently-shaped upstreams → one client shape).
• A forward/flush (no buffering) verification (curl -N).
• A disconnect → cancel upstream proof.
• Per-provider terminal usage capture into metering.
• A mid-stream error handled as an in-stream event + a timeouts/heartbeat note.

Up: Phase 8 Index · Next: 08 — Admin Dashboard