Interview Prep 03 — AI Infrastructure Engineer

Owns the compute/deployment layer: GPUs, autoscaling, Kubernetes, cost/capacity, reliability of the serving fleet. Phases: 6, 7.

1. Core concepts

• GPU memory: weights (params×bytes) + KV-cache + activations; quantization (cheatsheet 06).
• The bandwidth law: tok/s ≈ bandwidth ÷ model bytes; decode is memory-bound.
• Serving internals: PagedAttention, continuous batching, prefix caching (Phase 7).
• Tensor/pipeline parallelism; multi-GPU; node pools.
• Autoscaling on the right signal (queue depth/GPU util, not CPU).
• Self-host break-even vs APIs; cost/capacity planning.

2. System design questions

1. Deploy vLLM on Kubernetes with autoscaling + observability + canary (diagrams/17).
2. Capacity-plan a serving fleet for N QPS at a p95 target. (KV-cache budget, batch size, GPU count, headroom.)
3. Design a hybrid (API + self-hosted) serving strategy by volume/cost.

system-design template.

3. Debugging questions

• GPU OOM under load → KV-cache pressure; lower max-model-len, fewer concurrent seqs, quantize.
• Low GPU utilization → static batching / no continuous batching; KV fragmentation.
• p99 latency spikes → batch pressure, cold starts, autoscaling lag.
• Autoscaler thrashing → scaling on wrong metric; tune to queue depth.

4. Coding exercises

• Write a KV-cache + VRAM memory calculator (labs/lab-06).
• A load-test harness measuring TTFT/TPOT/p95 under concurrency (labs/lab-12).
• A bandwidth-law tok/s estimator.

5. Architecture exercises

• Draw the vLLM serving stack and the K8s deployment.
• Design GPU node pools, taints/tolerations, and weight storage (PVC/object store).

6. Model selection questions

• Given a GPU budget, which model/quant fits at the needed context+batch? (memory math).
• When does tensor-parallel across GPUs make sense vs a smaller model?

7. Production incident questions

• A GPU node dies → drain/reschedule; replica headroom; no single point (template 04).
• Cost overrun from over-provisioned GPUs → right-size; autoscale to zero off-peak; spot.
• Throughput collapse at peak → capacity headroom; backpressure; queueing.

8. Portfolio project

A serving benchmark + deploy: run vLLM (or llama.cpp) for a model; measure TTFT/TPOT/p95 vs concurrency; compare quants; a memory calculator; a K8s manifest (or compose) with autoscaling notes. README with the perf/cost curves.

9. 30-minute drill

• 5 min: GPU memory breakdown + bandwidth law.
• 10 min: design K8s vLLM deployment with autoscaling (whiteboard).
• 10 min: debug OOM-under-load + low GPU util.
• 5 min: self-host break-even reasoning.

10. 2-hour take-home

Stand up a local serving engine; produce a benchmark report: tok/s and p95 vs concurrency, memory usage, quant comparison, and a capacity estimate for a target QPS. Include the memory math and the autoscaling signal you'd use.

11. Senior-level expectations

• Junior: can run a serving engine.
• Mid: deploys + benchmarks + basic autoscaling.
• Senior: capacity-plans from first principles (KV-cache, bandwidth), tunes batching/parallelism, picks the right autoscaling signal, and optimizes cost/perf with data.
• Principal: designs the fleet architecture and cost model org-wide; build-vs-buy (self-host break-even) strategy; reliability/DR for GPU infra.

Next: 04 — Model-Serving Engineer · Index: interview-prep/