Streaming

Phase 7 · Document 06 · Production Serving Prev: 05 — Prefix and Prompt Caching · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Streaming is why ChatGPT feels fast even when a full answer takes 20 seconds: tokens appear as they're generated instead of after a long silence. For any interactive LLM product it is non-negotiable UX — and it's deceptively tricky to do correctly in a production proxy/gateway, where a long-lived connection must forward chunks without buffering, handle client disconnects (to stop paying for abandoned generations), survive mid-stream errors, and still capture usage for billing. Getting streaming right is a hallmark of a serious serving layer; getting it wrong shows up as janky UIs, runaway costs from abandoned requests, and missing usage data.

2. Core Concept

Plain-English primer: tokens as they're born

Recall decode produces one token at a time (Phase 2.05, what-happens §1.D). Non-streaming ("unary") waits until all tokens are generated, then returns the whole response — the user stares at a spinner for the full generation time. Streaming sends each token (or small chunk) to the client the moment it's produced, so the user starts reading after the first token.

Key insight: streaming changes perceived latency, not real latency. The total time is the same; what improves is time-to-first-token (TTFT) as the felt latency — the user sees motion almost immediately instead of after the full TTFT + TPOT×N (Phase 1.05). For a 500-token answer at 50 tok/s, non-streaming feels like 10s of silence; streaming feels like ~0.3s to first word.

The transport: Server-Sent Events (SSE)

LLM APIs stream over SSE — a simple HTTP mechanism where the server keeps the connection open and pushes data: lines as events arrive. (It's one-way server→client over plain HTTP, simpler than WebSockets, which is why every major API uses it.) The OpenAI-compatible shape:

data: {"choices":[{"delta":{"role":"assistant"}}]}
data: {"choices":[{"delta":{"content":"Hello"}}]}
data: {"choices":[{"delta":{"content":" world"}}]}
data: {"choices":[{"finish_reason":"stop"}], "usage":{...}}
data: [DONE]

Anthropic uses typed events (message_start → content_block_delta … → message_delta with stop_reason+usage → message_stop, what-happens §10). Either way the client accumulates deltas into the final message.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(model="m", messages=[...], stream=True,
                                        stream_options={"include_usage": True})
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:                      # final chunk carries usage (with include_usage)
        print("\n", chunk.usage)

The five hard parts of a streaming proxy

When a gateway sits between client and model (Phase 8), streaming gets subtle. A correct streaming proxy must:

1. Forward, don't buffer. Flush each chunk downstream as it arrives. Accidentally buffering (a framework default, a response wrapper, gzip middleware) collapses streaming back into one big response — the #1 streaming bug.
2. Handle client disconnect → cancel upstream. If the user closes the tab, the proxy must detect the dropped connection and abort the upstream generation — otherwise you keep generating (and paying for) tokens nobody reads, holding KV/batch slots (03).
3. Handle mid-stream errors. The upstream can fail after streaming started (provider hiccup, timeout). You've already sent a 200 + partial content, so you can't send an HTTP error — you must emit an error event in the stream and close cleanly, and the client must handle a truncated answer.
4. Capture usage at the end. Token counts (and cache_read, etc.) arrive in the final chunk/event — the proxy must parse and record it for billing even while forwarding (09, 08). With OpenAI you must pass stream_options={"include_usage": true}.
5. Set sane timeouts + keep-alive. Generations can run 30s+; idle/read timeouts must allow that, and proxies/load balancers must not kill long-lived SSE connections. Send heartbeats if needed.

Streaming + tools

In agents, the model may stream tool_use blocks; the harness can't act until the tool call is complete, so it parses streamed tool arguments, runs the tool, and continues the loop (what-happens §1.G). Structured-output/JSON streaming similarly yields partial JSON that's only valid once complete (Phase 5.07).

3. Mental Model

   NON-STREAM:  [....... generate all N tokens .......] → whole response   (user waits TTFT+TPOT·N)
   STREAM (SSE):  t1→ t2→ t3→ … each token pushed as born → user reads at TTFT  (perceived ↓↓)
                  data: {"delta":{"content":"…"}} … data: {usage} … data: [DONE]

   STREAMING PROXY must:  ① forward, don't buffer   ② disconnect → cancel upstream (stop paying)
                          ③ mid-stream error → error EVENT (200 already sent)
                          ④ grab usage from the FINAL chunk   ⑤ long timeouts + keep SSE alive
   streaming changes PERCEIVED latency, not total time

Mnemonic: push tokens as they're born (SSE); the proxy's job is forward-don't-buffer, cancel-on-disconnect, error-as-event, capture-usage, don't-time-out.

4. Hitchhiker's Guide

What to look for first: is streaming actually flushing end-to-end (no buffering layer), and does a client disconnect cancel the upstream? Those two are the most common production failures.

What to ignore at first: WebSockets/gRPC streaming — SSE is the standard for LLM APIs; don't over-engineer the transport.

What misleads beginners:

• A buffering layer silently kills streaming. Gzip middleware, response wrappers, some serverless platforms, or Nginx proxy_buffering on collapse the stream into one chunk. Test that tokens actually trickle.
• Forgetting to cancel on disconnect. Abandoned generations keep consuming tokens (cost) and KV/batch slots (capacity) (03, 09).
• Missing usage. If you don't request/parse the final-chunk usage, your billing/observability is blind (08).
• Treating mid-stream errors as HTTP errors. You already sent 200 — errors must be in-stream events.

How experts reason: they treat the streaming path as a long-lived, cancelable pipe: forward+flush immediately, propagate cancellation to upstream on client close, emit structured in-stream errors, and always capture trailing usage. They set generous timeouts and disable buffering at every hop (app, proxy, CDN/LB).

What matters in production: TTFT (the metric streaming optimizes), inter-token latency smoothness, disconnect→cancel correctness (cost/capacity), mid-stream error handling, and complete usage capture. Also: backpressure — if the client reads slowly, don't let buffers grow unbounded.

How to debug/verify: curl -N (no buffering) the endpoint and watch tokens trickle; kill the client mid-stream and confirm upstream generation stops (check engine num_requests_running drops, cost stops); verify the final usage event is recorded.

Questions to ask providers/frameworks: SSE supported? does the platform buffer responses? is include_usage/trailing usage available? are long-lived connections allowed (timeouts, LB idle limits)? does disconnect propagate cancellation?

What silently gets expensive/unreliable: buffering that disables streaming (bad UX, no one notices in tests), abandoned-generation cost from no cancellation, missing usage (billing drift), and LB/proxy idle-timeouts cutting long streams.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Streaming improves perceived latency (TTFT), not total time — same tokens, delivered as born.
• LLM APIs stream over SSE (data: events); clients accumulate deltas into the final message.
• A streaming proxy must forward+flush, not buffer — buffering (gzip, wrappers, proxy_buffering on) silently disables streaming.
• Client disconnect must cancel the upstream — else you pay for and hold capacity for tokens nobody reads (03, 09).
• Mid-stream errors are in-stream events, not HTTP errors (200 already sent).
• Usage arrives in the final chunk — capture it for billing (stream_options.include_usage on OpenAI) (08).
• Long generations need long timeouts + keep-alive — LBs/proxies must not cut SSE connections.
• Tool calls and JSON stream incrementally and are only actionable/valid once complete (what-happens §1.G, Phase 5.07).

9. Observations from Real Systems

• Every interactive LLM product streams — chat UIs, coding assistants (token-by-token edits), voice (stream→TTS). Non-streaming is for batch/automation only.
• OpenRouter / LiteLLM are fundamentally streaming proxies — normalizing different providers' SSE/event shapes into one and forwarding chunks (what-happens §3.5, Phase 8).
• The classic "streaming broke" incident is a buffering layer (a new gzip middleware, a serverless platform, an Nginx default) collapsing the stream — fixed by disabling buffering at that hop.
• Cost-leak incidents trace to no disconnect-cancellation: users abandon long generations and the server keeps going (09).
• Coding agents stream tool_use and apply edits incrementally; the harness parses streamed tool args before executing (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD/OPERATE the streaming path:
 1. Interactive UX? → stream (SSE). Batch/automation? → non-streaming is fine.
 2. PROXY correctness checklist:
      ① forward + FLUSH each chunk (disable buffering: app, Nginx proxy_buffering off, CDN)
      ② client disconnect → CANCEL upstream generation (stop cost + free KV/batch slot)  [03,09]
      ③ mid-stream error → emit error EVENT, close cleanly (client handles truncation)
      ④ parse + record USAGE from the final chunk (include_usage on OpenAI)               [08]
      ⑤ timeouts ≥ max generation time; keep SSE alive (LB idle limits, heartbeats)
 3. Tools/JSON: accumulate streamed deltas; act/validate only when complete.              [Phase 5.07]
 4. MEASURE TTFT + inter-token smoothness; verify cancel-on-disconnect actually stops upstream.

12. Hands-On Lab

Goal

Build a minimal streaming proxy that forwards SSE, cancels upstream on client disconnect, and records usage — then prove each property.

Prerequisites

• A streaming endpoint (vLLM 01 or a managed API); Python with fastapi/httpx (or your stack).

Setup

pip install fastapi uvicorn httpx openai

Steps

1. Baseline client: call the endpoint with stream=True and print tokens as they arrive; confirm they trickle (use curl -N too).
2. Proxy that forwards: write a tiny FastAPI route that opens an upstream stream and yields each chunk downstream (use a StreamingResponse); verify with curl -N that tokens still trickle through the proxy (no buffering).
3. Break it on purpose: wrap the response in something that buffers (or enable gzip) and observe the stream collapse into one chunk — then fix it. This cements the #1 bug.
4. Cancel on disconnect: in the proxy, detect client disconnect (e.g., await request.is_disconnected() / generator GeneratorExit) and abort the upstream httpx stream. Test by killing the client mid-stream and confirming the upstream/engine stops generating (watch vLLM num_requests_running drop, or provider usage).
5. Capture usage: request trailing usage (stream_options={"include_usage": true}), parse the final chunk, and log input/output tokens. Confirm it's recorded even though you streamed.
6. Mid-stream error: simulate an upstream error after the first chunk; emit an in-stream error event and close; confirm the client handles a truncated answer.

Expected output

A working streaming proxy demonstrating: tokens trickle (no buffering), client disconnect stops upstream, usage is captured, and mid-stream errors surface as events.

Debugging tips

• Tokens arrive all-at-once → a buffering layer (framework, gzip, proxy_buffering); flush/disable it.
• Upstream keeps running after client close → you didn't propagate cancellation to the httpx stream.

Extension task

Add heartbeats/keep-alive comments and a generous timeout so a slow 60s generation survives an LB idle limit.

Production extension

Add per-request usage → billing events and TTFT metrics to the proxy (08, 09); normalize Anthropic events into OpenAI-shaped deltas (Phase 8).

What to measure

TTFT through the proxy; that tokens trickle (not buffered); upstream stops on disconnect; usage captured; truncation handled on error.

Deliverables

• A streaming proxy with forward+flush, cancel-on-disconnect, usage capture, in-stream errors.
• A buffering before/after demonstration.
• A note proving disconnect cancels upstream (cost/capacity saved).

13. Verification Questions

Basic

1. Does streaming reduce total latency? What does it actually improve?
2. What transport do LLM APIs use to stream, and what's in each event?
3. Where does token usage appear in a streamed response?

Applied 4. Your streamed tokens arrive all at once at the end. Name three layers that could be buffering. 5. Why must a mid-stream error be an in-stream event rather than an HTTP error?

Debugging 6. Costs are higher than token counts suggest; many users abandon long answers. Cause and fix. 7. Streams get cut at ~60 seconds. What's misconfigured?

System design 8. Design a streaming gateway route: forwarding, cancellation, error handling, usage capture, timeouts.

Startup / product 9. How does cancel-on-disconnect protect your unit economics, and what does TTFT do for conversion/UX?

14. Takeaways

1. Streaming improves perceived latency (TTFT), delivering tokens as they're generated over SSE.
2. A correct streaming proxy: forward+flush (no buffering), cancel upstream on disconnect, in-stream errors, capture trailing usage, long timeouts.
3. Buffering is the #1 streaming bug; no-cancel-on-disconnect is the #1 cost leak.
4. Usage arrives in the final chunk — capture it for billing/observability.
5. Tools and JSON stream incrementally — accumulate, then act/validate when complete.

15. Artifact Checklist

• A streaming proxy (forward+flush, cancel-on-disconnect, usage capture, in-stream errors).
• A buffering before/after demo.
• Proof that disconnect cancels upstream.
• TTFT measured through the proxy.
• Usage/billing events wired toward 08/09.

Up: Phase 7 Index · Next: 07 — Routing and Fallbacks