Brother Talk — Inference Serving Internals

Off the record. The stuff I'd tell you over a beer, not in a design review.

This is the phase that turns "I use LLMs" into "I run LLMs." Everyone can call an API. The people who get paid the senior money are the ones who can keep that API up, fast, and cheap under real traffic — and that's PagedAttention, continuous batching, and speculative decoding. You don't have to write the CUDA kernel. You have to understand the three ideas well enough to tune the engine and diagnose it at 2 a.m. That's a smaller ask than it sounds and it's worth a lot.

The KV-cache will OOM you in production, and you'll be the hero who saw it. I promise: at some point a service that "fits in memory, we checked" will fall over at peak, and a room full of smart people will be confused. You'll say "what's the concurrency × context? That's the KV blocks, not the weights" — and you'll fix it by capping max_num_seqs and raising gpu_memory_utilization in ten minutes. PagedAttention is why there are blocks to cap. This is the most reliable "looks like magic, is actually arithmetic" moment in the whole job.

Continuous batching is the cheapest 3× you'll ever ship. No new hardware, no model change — just scheduling at the iteration level instead of the request level. The first time you watch p99 latency drop and throughput jump because you stopped letting one long answer hold the batch hostage, you'll get why this is the default in every serious engine. If a system is doing static batching in 2026, that's free money on the floor.

Speculative decoding is the magic trick that's actually math. People hear "the small model guesses and the big model checks" and assume quality drops. It does not — the acceptance rule is engineered so the output is exactly the target's distribution. Learn the one-line proof (min(q,p) + max(0,p−q) = p). Being able to say "it's provably distribution-preserving, the draft only changes speed" in an interview is a genuine oh, this person actually knows it moment. And in practice: the acceptance rate is everything — a great draft on predictable text flies, a mediocre draft on creative text is a net loss. Match the drafter to the workload (n-grams for code/RAG quotes are nearly free).

Don't reach for the fancy thing first. There's a pull toward "let's add speculative decoding" or "let's compile with TensorRT-LLM" before you've turned on prefix caching or sized the KV pool right. The boring wins come first: fit more sequences (paging, gpu_memory_utilization), keep the batch full (continuous batching, which is already on in vLLM), share the fat system prompt (prefix caching). Only then chase speculation and compiled kernels. Most "we need a bigger GPU" tickets are actually "we mis-tuned three knobs."

You serve a contract, not a number. Junior instinct is "make it fast." Senior instinct is "fast on which metric, and what's the budget?" TTFT, TPOT, throughput, and $/1M pull against each other. Walking into a review with "here's the load test hitting all three SLOs at this batch config and this cost" ends the conversation in your favor. Walking in with "it feels faster now" starts an hour of vibes. Load-test first, every time.

What's worth caring about: the three mechanisms cold, the KV-block capacity math, the distribution-preservation proof, and which knob moves which SLO. What's not worth losing sleep over: memorizing every vLLM flag, the exact CUDA of the PagedAttention kernel, or whether the speedup is 2.7× or 3.1×. Estimates that are right to within a factor and a named deciding constraint win design reviews; chasing the last 5% before you've tuned the obvious knobs is how juniors burn a week.

The career framing. Anyone can stand up a demo endpoint. Almost nobody can take it to production scale at a defensible cost — and that's exactly the gap companies pay seniors to close. This phase is where Phase 00's "a model is a physical system" becomes a system you actually operate. Get the three mechanisms and the capacity math into your bones and you become the person they ask before they provision the cluster. Now go make pytest green.