Exercises 05 — Production Serving (10)

Concepts: Phase 7, Phase 8. Cheatsheets: 09, 13, 17.

Scenario: A vLLM server is fast for one user, slow at 50 concurrent.
Task: Diagnose + tune.
Constraints: Same hardware.
Expected answer: KV-cache pressure limits concurrent sequences. Lower --max-model-len, raise --max-num-seqs carefully, enable prefix caching, quantize, or add GPUs (tensor-parallel). Measure p95 under load, not single-request (cheatsheet 09).
Rubric: Senior = KV-cache/batching reasoning + load testing.
Common mistakes: Trusting single-request latency.
Extension: Plot throughput vs concurrency to find the knee.

Scenario: Users say "slow to start" but tokens stream fine once started.
Task: Which metric, which fix?
Constraints: Chat.
Expected answer: TTFT (prefill) is high. Reduce prompt size (token budget), prefix-cache shared context, smaller/faster model for prefill, or speculative decoding. Stream so perceived latency = TTFT.
Rubric: Senior = separates TTFT (prefill) from TPOT (decode).
Common mistakes: Optimizing decode when prefill is the problem.
Extension: How does a long RAG context hurt TTFT?

Scenario: Interviewer: "Why does vLLM get high GPU utilization?"
Task: Explain.
Constraints: 60s.
Expected answer: PagedAttention (no KV fragmentation) + continuous batching (requests join/leave the batch each step) keep the GPU busy across many sequences, vs static batching that idles waiting for the slowest request (Phase 7.03–7.04).
Rubric: Senior = paged KV + continuous batching together.
Common mistakes: Confusing with static batching.
Extension: What's the latency cost of larger batches?

Scenario: Primary provider has an outage; product goes down.
Task: Design resilience.
Constraints: Multi-provider.
Expected answer: Gateway with fallback across providers/models, retries with backoff, circuit breakers, timeouts, and health checks. Same model often on multiple providers — fail over (Phase 7.07).
Rubric: Senior = fallback + circuit breaker + timeout + retry.
Common mistakes: Single provider, no timeout.
Extension: How do you avoid retry storms?

Scenario: Every request resends a 2k-token system prompt + shared RAG context.
Task: Cut cost/latency.
Constraints: High volume.
Expected answer: Prefix/prompt caching — the shared prefix is computed/charged once and reused, cutting input cost and TTFT (Phase 7.05). Order stable prefixes first.
Rubric: Senior = identifies caching + structures prompts for cache hits.
Common mistakes: Re-sending identical context uncached.
Extension: Measure cache hit rate and its cost impact.

Scenario: You stream tokens but must block unsafe output.
Task: Resolve the tension.
Constraints: Safety required.
Expected answer: Buffer/scan in windows, run output guardrails before/while emitting, or stream with a moderation pass that can halt/redact; for high-risk, validate before showing. Trade some perceived latency for safety (Phase 14.05).
Rubric: Senior = balances streaming UX with fail-closed output checks.
Common mistakes: Streaming raw output with no output guardrail.
Extension: Design windowed moderation for streaming.

Scenario: You must front 4 providers with one internal API.
Task: List the components.
Constraints: Multi-tenant.
Expected answer: Provider adapters, model registry (model×provider), routing, fallback/retry, streaming proxy, usage metering, caching, guardrails/PEP, auth + key custody, observability/audit (cheatsheet 13).
Rubric: Senior = names the full component set + key custody.
Common mistakes: Forgetting metering/key custody/observability.
Extension: Build a minimal one (labs/lab-08-llm-gateway).

Scenario: A multi-step agent fails intermittently in prod; logs are unstructured.
Task: What to add?
Constraints: Debuggable.
Expected answer: Correlation IDs across retrieval→model→tool calls; traces (OTel); p50/p95/p99, cost, error dashboards; structured logs; PII-safe audit (Phase 7.08, Phase 10.08).
Rubric: Senior = correlation IDs + traces + percentiles.
Common mistakes: Average-only metrics; no request tracing.
Extension: Reconstruct one failed run end-to-end from logs.

Scenario: Spend doubled overnight; cause unknown.
Task: Investigate + prevent.
Constraints: Production.
Expected answer: Break down cost by tenant/feature/model (cost observability); find the driver (loop/retry storm, a power user, a routing change, longer outputs); add per-tenant budgets/caps + alerts; fix the driver (Phase 7.09, diagrams/16).
Rubric: Senior = attributes cost + caps + alerts.
Common mistakes: No per-tenant attribution; no caps.
Extension: Add anomaly alerting on spend.

Scenario: Deploy an LLM app + vLLM on Kubernetes.
Task: Sketch the architecture.
Constraints: Autoscaling, GPU.
Expected answer: Ingress → API gateway (auth/rate/route/guardrails) → stateless app pods (HPA) + vLLM pods on GPU node pool (HPA/KEDA on queue depth) → vector DB + Redis + Postgres (RLS) → Prometheus/Grafana/OTel + Vault; canary/rollback; weights on PVC/object store (diagrams/17).
Rubric: Senior = GPU node pool + autoscaling signal + observability + canary.
Common mistakes: Autoscaling GPU pods on CPU metrics; no canary.
Extension: What scales GPU pods correctly (queue depth/GPU util, not CPU)?

Next: 06 — RAG & Agents · Index: exercises/

LLM Inside-Out Mastery