TGI, SGLang, and TensorRT-LLM

Phase 7 · Document 02 · Production Serving Prev: 01 — vLLM · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

vLLM (01) is the default, but it isn't the only serious engine — and the right one for your workload can mean a real difference in throughput, latency, structured-output speed, or operational fit. TGI (Hugging Face's engine), SGLang (fast for structured/agentic and prefix-heavy workloads), and TensorRT-LLM (NVIDIA's max-throughput compiler-based engine) each win in specific situations. A seasoned serving engineer can name what each is good at, knows they're largely interchangeable behind an OpenAI-compatible API, and chooses deliberately instead of cargo-culting vLLM everywhere. This doc gives you that comparative literacy without months of trial and error.

2. Core Concept

Plain-English primer: same job, different trade-offs

All four engines do the same job — load weights, manage the KV cache, batch many requests, stream tokens — and they all implement the core ideas you already know: continuous batching (03), paged/efficient KV (04), and usually prefix caching (05). They differ in how they're built, what they optimize, and how much setup they cost. Crucially, most expose an OpenAI-compatible API, so swapping engines rarely touches your client code (01).

The four engines

vLLM (recap, 01) — the well-rounded default. Easiest to run, broad model support, strong throughput, big community. Your baseline.

TGI — Text Generation Inference (Hugging Face). A production-hardened Rust+Python server with deep HF-ecosystem integration, easy Docker deployment, and solid observability. Continuous batching + paged KV + (newer) prefix caching. Tends to be chosen by teams already standardized on Hugging Face tooling/Inference Endpoints and who want a batteries-included, well-supported server.

SGLang. A high-performance engine notable for RadixAttention — prefix caching organized as a radix tree that maximizes KV reuse across many requests that share branching prefixes (e.g., few-shot templates, agent trees, multi-turn). It also has a fast constrained-decoding path, making it strong for structured output / JSON / function-calling workloads (Phase 5.07) and agent serving. Often top-tier throughput on prefix-heavy and structured workloads.

TensorRT-LLM (NVIDIA). A compiler-based engine: it builds an optimized engine plan for a specific model + GPU using TensorRT (fused kernels, FP8, in-flight batching). It delivers the highest throughput/lowest latency on NVIDIA hardware — at the cost of a heavier build/setup step and tighter coupling to specific GPUs/model configs. Often served via NVIDIA Triton Inference Server. Chosen by teams squeezing maximum performance from homogeneous NVIDIA fleets.

The comparison

All four: continuous batching, efficient/paged KV, OpenAI-compatible serving (TensorRT-LLM via Triton or its OpenAI frontend). The choice is operational fit × workload shape × performance ceiling, not capability gaps for most models.

3. Mental Model

   SAME JOB (load · KV · batch · stream)        DIFFERENT TRADE-OFFS
   ┌───────────────────────────────────────────────────────────────────────┐
   vLLM        → default all-rounder; easiest; broadest models               │
   TGI         → HF ecosystem; Docker batteries-included; hardened           │
   SGLang      → RadixAttention (branching prefix reuse) + fast JSON/agents  │
   TensorRT-LLM→ compiled, FP8, top NVIDIA perf; heavy setup (via Triton)    │
   └───────────────────────────────────────────────────────────────────────┘
   all OpenAI-compatible → swapping engines rarely changes client code [01]
   choose by:  workload shape  ×  operational fit  ×  performance ceiling

Mnemonic: vLLM default · TGI = HF-native · SGLang = structured/prefix-heavy · TensorRT-LLM = max NVIDIA perf (heavy build).

4. Hitchhiker's Guide

What to look for first: does vLLM already meet your throughput/latency/feature needs? If yes, stop — it's the lowest-friction choice. Only differentiate when a specific need (structured-output speed, absolute perf, HF integration) is unmet.

What to ignore at first: micro-benchmark leaderboards between engines — they shift release to release and rarely change the decision versus operational fit.

What misleads beginners:

• "TensorRT-LLM is fastest, so use it." Its build/maintenance cost is real; the win only matters at scale where it amortizes.
• "They're totally different." They share the same core ideas; switching is usually a config/deploy change, not a rewrite, thanks to OpenAI compatibility.
• "SGLang is only for structured output." It's a strong general engine; structured/prefix-heavy is just where it shines most.
• "Engine choice fixes a model-quality problem." It doesn't — engines change speed/cost, not the model's answers (modulo quant/precision, Phase 5.10).

How experts reason: default to vLLM; pick SGLang for heavy JSON/function-calling or agent trees with shared prefixes; pick TGI when standardized on HF tooling/endpoints; invest in TensorRT-LLM only when at sufficient NVIDIA scale to amortize the build and you've proven vLLM is the bottleneck. They keep clients OpenAI-shaped so the engine stays swappable.

What matters in production: the same as 00 — p95/p99, KV ceiling, queue depth — plus the engine's build/upgrade story (TensorRT-LLM rebuilds per model/GPU), model coverage (does it support your architecture day-1?), and quantization support.

How to debug/verify: A/B the same model + workload on two engines at equal precision; compare throughput, p95 TTFT/TPOT, and ops effort. Confirm structured-output correctness on SGLang's constrained path.

Questions to ask: Does it support my model architecture and quant? OpenAI-compatible? What's the upgrade/build cost? Prefix caching / constrained decoding quality? GPU coverage?

What silently gets expensive/unreliable: TensorRT-LLM engine rebuilds blocking model updates; choosing a niche engine that lags on new-model support; over-optimizing perf before you've saturated vLLM.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• All four engines share the core: continuous batching + efficient/paged KV + (usually) prefix caching, behind an OpenAI-compatible API.
• vLLM is the low-friction default; differentiate only for a specific unmet need.
• SGLang's RadixAttention maximizes KV reuse across branching shared prefixes (few-shot, agent trees) and it has fast constrained decoding for structured output.
• TGI is the Hugging-Face-native, Docker-friendly, hardened option.
• TensorRT-LLM delivers top NVIDIA throughput via a compiled engine (FP8, fused kernels) but costs a per-model/GPU build and is often served through Triton.
• Switching engines rarely touches client code because of OpenAI compatibility — keep clients engine-agnostic.
• Engine choice changes speed/cost, not answers (except via quant/precision, Phase 5.10).
• New-model support lag differs by engine — check day-1 coverage for the architecture you need.

9. Observations from Real Systems

• vLLM and TGI power a large share of self-hosted and inference-as-a-service backends; many providers expose one of them under an OpenAI-compatible surface.
• SGLang is increasingly chosen for agent and structured-output serving where shared prefixes and JSON correctness dominate (Phase 10).
• TensorRT-LLM + Triton appears in large NVIDIA production fleets chasing the last 20–40% of throughput/latency.
• Gateways (Phase 8) treat all of these as interchangeable OpenAI-compatible upstreams, which is exactly why teams can switch engines without client changes.
• Benchmark wars between engines flip across releases — teams that pin to operational fit age better than those chasing the leaderboard.

10. Common Misconceptions

11. Engineering Decision Framework

CHOOSE A SERVING ENGINE:
 1. Default to vLLM. Does it meet throughput/latency/feature needs?  YES → ship.  [01]
 2. Unmet need?
      heavy JSON / function-calling / agent trees / branching prefixes → SGLang
      standardized on Hugging Face tooling / want Docker batteries     → TGI
      need max NVIDIA throughput AND at scale to amortize a build      → TensorRT-LLM (+Triton)
 3. Constraints: model architecture supported? quant supported? OpenAI-compatible?
 4. Validate: A/B vs vLLM at equal precision — throughput, p95 latency, ops effort.
 5. Keep clients OpenAI-shaped so the engine stays swappable.                       [Phase 8]

12. Hands-On Lab

Goal

Run the same model + workload on vLLM and one alternative, and quantify the difference in throughput, p95 latency, and operational effort.

Prerequisites

• A CUDA GPU; Docker; the model in a supported format. pip install openai httpx.

Setup

# Baseline (from doc 01):
vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 4096 --port 8000

# Alternative — TGI (Docker):
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id Qwen/Qwen2.5-1.5B-Instruct --max-total-tokens 4096
# (or SGLang: python -m sglang.launch_server --model-path ... --port 8081)

Steps

1. Equalize: same model, same precision, same max-model-len/total-tokens, same prompts.
2. Concurrency sweep (1/8/32) against each endpoint; record aggregate tok/s and p50/p95 TTFT/TPOT.
3. Structured workload: if testing SGLang, run a batch of JSON-schema-constrained requests on SGLang vs vLLM; compare throughput and JSON validity (Phase 5.07).
4. Prefix-heavy workload: send many requests sharing a long few-shot prefix; compare TTFT (SGLang RadixAttention vs vLLM prefix caching, 05).
5. Ops log: note setup time, model-load time, and any rough edges for each.

Expected output

A comparison table: engine → tok/s, p95 TTFT/TPOT, JSON validity (if applicable), prefix-heavy TTFT, and a qualitative ops-effort score.

Debugging tips

• TGI/SGLang won't load your model → check architecture support and the correct launcher flags.
• Unfair comparison → ensure identical precision, context cap, and prompt set.

Extension task

If you have the patience, build a TensorRT-LLM engine for the model and serve via Triton; compare its throughput to vLLM and record the build time as the "cost."

Production extension

Put both engines behind a gateway and route a fraction of traffic to each (shadow/A-B) while comparing live p95 and cost (07, 08).

What to measure

tok/s, p95 TTFT/TPOT per engine; JSON validity; prefix-heavy TTFT; setup/build time.

Deliverables

• An engine comparison table on your workload.
• A recommendation with the deciding factor (perf vs ops vs feature).
• Notes on model/quant support and OpenAI compatibility per engine.

13. Verification Questions

Basic

1. Name the four engines and one differentiator each.
2. What do all four share at the core?
3. Why does OpenAI compatibility make engines swappable?

Applied 4. Pick an engine for: (a) heavy function-calling agent service, (b) HF-standardized team, (c) max-throughput NVIDIA fleet. Justify. 5. What is RadixAttention and which workloads benefit most?

Debugging 6. TensorRT-LLM gives the best throughput but blocks your weekly model updates. What's the trade-off you're hitting? 7. An engine swap "didn't change quality." Why is that expected?

System design 8. Design an engine choice + migration path that keeps clients unchanged as you scale.

Startup / product 9. When is investing in TensorRT-LLM justified for your unit economics, and when is it premature?

14. Takeaways

1. vLLM is the default; TGI, SGLang, and TensorRT-LLM win in specific situations.
2. They share the core (continuous batching, paged KV, prefix caching) and are OpenAI-compatible — usually swappable by config.
3. SGLang → structured output / prefix-heavy / agents (RadixAttention); TGI → HF-native; TensorRT-LLM → max NVIDIA perf at a build cost.
4. Choose by workload shape × operational fit × performance ceiling, not benchmark leaderboards.
5. Keep clients OpenAI-shaped so the engine remains a swappable detail.

15. Artifact Checklist

• An engine comparison table (tok/s, p95, JSON validity, prefix TTFT) on your workload.
• A recommendation with the deciding factor.
• Model/quant support notes per engine.
• (Optional) a TensorRT-LLM build-time measurement as the "cost."
• Confirmation that clients stayed OpenAI-compatible across engines.

Up: Phase 7 Index · Next: 03 — Continuous Batching