Interview Prep 04 — Model-Serving Engineer

Deep specialist in inference engines: throughput, latency, KV-cache, batching, quantization, speculative decoding. Phases: 6, 7.

1. Core concepts (deepest of the serving roles)

• Prefill (compute-bound, TTFT) vs decode (memory-bandwidth-bound, TPOT) (diagrams/01).
• KV-cache: what it stores, memory scaling (2×layers×heads×head_dim×seq×batch×bytes).
• PagedAttention (paging KV → no fragmentation), continuous batching, prefix caching.
• Speculative decoding / MTP (draft→verify→accept).
• Quantization for serving (AWQ/GPTQ/FP8) (cheatsheet 06).
• Throughput vs latency tradeoff (batch size); tensor parallelism.

2. System design questions

1. Maximize throughput for a shared model under SLA p95. (Batch tuning, KV budget, prefix caching, quant.)
2. Minimize TTFT for an interactive product. (Prompt size, prefix cache, speculative decoding, smaller prefill model.)
3. Serve many fine-tunes cheaply → multi-LoRA on one base (Phase 13.07).

vLLM stack diagram.

3. Debugging questions

• Throughput good, latency bad → batch too large; tune for latency.
• OOM at long context → KV-cache; lower max-model-len / paged / quantized KV.
• Speculative decoding gives no speedup → low draft acceptance rate.
• p99 worse than p95 by a lot → tail from batching/preemption; investigate scheduler.

4. Coding exercises

• Implement a toy KV-cache and show the O(n²)→O(n) win.
• A continuous-batching simulator (requests join/leave each step) measuring goodput.
• Benchmark a model across batch sizes; plot throughput vs p95 (labs/lab-07).

5. Architecture exercises

• Draw prefill vs decode, KV-cache, PagedAttention, speculative decoding from memory.
• Explain how PagedAttention enables continuous batching.

6. Model selection questions

• For a fixed GPU, which model+quant maximizes quality within the p95 budget?
• When does FP8 vs AWQ vs GPTQ matter? (hardware, quality, speed).

7. Production incident questions

• Latency regression after a config change → batch/KV settings; roll back; bisect.
• A long-context request degrades everyone → per-request context caps; isolation.
• GPU memory creep → KV not freed; sequence leak; check scheduler.

8. Portfolio project

A serving deep-dive: benchmark TTFT/TPOT/p95/throughput vs concurrency and batch size on vLLM; quant comparison (FP16 vs AWQ/GPTQ); a KV-cache memory model; (stretch) measure speculative decoding speedup. README with curves + the bandwidth-law analysis.

9. 30-minute drill

• 5 min: prefill vs decode; why decode is bandwidth-bound.
• 10 min: explain PagedAttention + continuous batching (whiteboard).
• 10 min: tune a server for throughput vs latency; KV-cache math.
• 5 min: speculative decoding mechanics + when it fails.

10. 2-hour take-home

Benchmark an inference engine: produce throughput/latency curves vs concurrency and batch size, a KV-cache memory estimate for a given context, and a quant comparison. Explain results using prefill/decode and the bandwidth law.

11. Senior-level expectations

• Junior: runs vLLM, reads metrics.
• Mid: tunes batching/quant; benchmarks correctly under load.
• Senior: reasons from KV-cache/bandwidth first principles; optimizes the throughput/latency frontier; deploys speculative decoding/multi-LoRA where they pay off.
• Principal: sets serving architecture and SLAs; chooses engines/hardware; drives cost/perf across the fleet.

Next: 05 — RAG Engineer · Index: interview-prep/