Interview Prep 02 — System Design Walkthroughs

The highest-weight round for JD2. The prompt is always some form of "size and design inference for [workload]." The interviewer watches how you reason, not whether you hit a magic number. This file gives a repeatable framework and five worked walkthroughs spanning the role's surface.

Table of Contents

The framework (use it every time)

Say this structure out loud — it signals seniority before you've computed anything.

1. CLARIFY    → turn the vague ask into a spec: model, I/O lengths, throughput,
                concurrency, p95 TTFT/TPOT SLO, accuracy floor, budget, power, hardware on hand.
2. CHARACTERIZE → prefill- or decode-heavy? compute/memory/capacity-bound? (roofline + I/O mix)
3. MEMORY     → weights + KV(batch,context) + activations → does it fit? → parallelism (TP/PP).
4. THROUGHPUT → per-instance goodput at the SLO → #instances → total accelerators + headroom.
5. OPTIMIZE   → quantize first, then batch/serve (vLLM/TRT-LLM), then shard, then spec-decode/prefix-cache.
6. TRADE-OFFS → present 2–3 points on the Pareto curve with $/token; recommend one, justify.
7. RISKS      → accuracy validation, burst/autoscaling, failure modes, monitoring.

Always quantize before you shard and define the SLO before you size. Always end with a recommendation + the curve, not one number.

Walkthrough 1 — Serve a 70B chat assistant at scale

Prompt: "Design inference for a Llama-3-70B customer-support chatbot. ~1k-token prompts, ~400-token answers, 200 req/s peak, p95 TTFT < 1s, p95 TPOT < 40 ms, on H100-80GB nodes (NVLink intra-node, IB inter-node)."

1. Clarify → got the numbers. Ask: accuracy floor (named eval)? budget/$ target? is 70B fixed or can we rightsize? (Flag: a tuned 8B+RAG might clear the floor — K08 §5 — but assume 70B is required here.)

2. Characterize → 1k in / 400 out ≈ balanced, leaning decode-heavy (400 sequential decode steps dominate wall-clock). Decode → memory-bandwidth-bound. TPOT 40 ms is the binding SLO.

3. Memory → 70B FP16 = 140 GB > 80 GB → must shard. TP=2 over NVLink holds weights (70 GB/GPU) with KV room; or INT8 (70 GB) to fit tighter / leave KV headroom. KV (K01 §3): GQA-8, 80 layers, ~320 KB/token → 1.4k context ≈ 450 MB/seq. At TP=2 (160 GB − 140 weights = ~20 GB KV) → ~40 concurrent seqs/instance; INT8 weights or INT8 KV roughly doubles that.

4. Throughput → benchmark the TP=2 instance: find batch where p95 TPOT = 40 ms → that's per-instance goodput (say ~15 req/s). #instances = ⌈200/15⌉ = 14, ×1.4 headroom ≈ 20 instances × 2 GPUs = 40 H100s, ~5 nodes. (Illustrative — the real number comes from Lab 03 benchmarking.)

5. Optimize → INT8/FP8 weights (2× decode + capacity), INT8 KV (2× concurrency), prefix caching for the shared system prompt (big TTFT win), continuous batching + PagedAttention (vLLM/TRT-LLM). Each can cut the instance count materially.

6. Trade-offs → (a) FP16 TP=2, most GPUs, safest quality; (b) INT8 TP=2 + INT8 KV — recommended, ~half the GPUs, −0.4 pts on their eval; (c) FP8 if Hopper FP8 path is mature in their stack. Present $/token for each.

7. Risks → validate INT8 on the hardest tickets; 200 is peak — confirm peak-vs-average and warm-buffer autoscaling (K08 §4); monitor p99 TPOT, KV preemption, goodput.

Walkthrough 2 — Port & optimize a customer model onto a new accelerator

Prompt: "A customer has a custom PyTorch vision-language model running on A100s. Port it to our accelerator (Qualcomm-AI100-class, INT8-first, ONNX→vendor compiler) and hit parity throughput at lower power."

1. Clarify → exact model/ops, dynamic shapes (variable image res? variable text len?), accuracy floor + eval, current A100 latency/throughput baseline, power budget.

2. Conversion plan (K03) → export to ONNX (dynamo_export, correct opset, dynamic_axes), inspect in Netron, validate against ONNX Runtime as oracle. Then feed the vendor compiler (qaic/AIC).

3. Anticipate the three breakages (K03 §6) → (a) unsupported ops (custom attention / vision op) → rewrite with primitives or fallback; (b) dynamic shapes (variable resolution) → optimization profiles / bucketing; (c) numerical divergence → bisect FP32→FP16→INT8 to localize.

4. Quantize (K02) → INT8-first accelerator → W8A8 needs activation handling → SmoothQuant + calibration on the customer's real images/text via AIMET. Sensitivity-analyze; keep vision encoder's first/last layers and the LM head higher precision if needed.

5. Validate → layer-by-layer vs the A100 FP16 reference (cosine, max error), then task eval (the customer's VL benchmark), then throughput/power on the accelerator.

6. Deliverable → port report: ops fixed, quant scheme, accuracy delta on their eval, throughput parity, and the power win (the actual pitch). Plus a runbook + a GitHub issue for any genuinely unsupported op (a JD duty).

7. Trade-offs → if INT8 misses the accuracy floor: mixed precision (some layers INT16/FP16), or QAT if a training pipeline exists, or accept slightly lower throughput for FP16-critical layers. Present the accuracy/throughput/power frontier.

Walkthrough 3 — Real-time 100-camera video analytics

Prompt: "Design a real-time system: 100 cameras, 1080p H.265 @ 30fps, detect + track people and vehicles, recognize license plates, alert on rules. Minimize hardware."

1. Clarify → real-time latency budget (33 ms/frame? or buffered analytics?), accuracy needs, peak object density (drives recognizer load), on-prem or cloud, power/space.

2. The decode-first insight (K07 §3) → 100×1080p30 HEVC. NVDEC capacity, not detector FLOPs, likely caps streams/GPU (~30–35 HEVC 1080p30). → ~3–4 GPUs just for decode — surface this immediately; it's the binding constraint juniors miss.

3. Pipeline (K07 §2,5) → DeepStream (GStreamer): NVDEC → nvstreammux (batch across cameras) → TensorRT detector (INT8) → nvtracker (ByteTrack, per-stream) → plate recognizer on vehicle crops only → rules → alerts. Zero-copy GPU memory throughout (K07 §8) — no CPU round-trips.

4. Batching → gather one frame per camera → detector batch 30/GPU → demux for per-stream tracking. Confirm decode+preprocess+infer+track sums within the 33 ms budget.

5. Sizing → min(decode, inference, memory) per GPU (K07 §9) → decode-bound at ~30/GPU → 4 GPUs for 100 cameras, detector has headroom, plate recognizer is the variable cost (size for peak vehicle density).

6. Trade-offs → fewer GPUs with a multi-NVDEC SKU; INT8 detector (almost always fine for detection); FFmpeg only for the archival/transcode side-job, GStreamer/DeepStream for the live path.

7. Risks → camera reconnects/stream drops, peak-density recognizer spikes, alert latency SLO, storage for evidence clips.

Walkthrough 4 — Cut a customer's inference bill in half

Prompt: "A customer spends $X/month serving an LLM on 16 GPUs and wants it halved without hurting quality. Where do you start?"

1. Measure before cutting (K06) → profile: are they even using the GPUs? nsys (idle vs busy), MFU/MBU, utilization. Most over-spend is under-utilization — 30% utilization means half the bill is idle silicon.

2. The cheap wins, in order:

  • Continuous batching + PagedAttention if they're on naive serving → often 5–20× throughput → fewer GPUs (K04 §3–4). Frequently the single biggest win.
  • Quantize to INT8/FP8 (K02) → 2× throughput + capacity, validate accuracy on their eval.
  • Prefix caching if they have big shared prompts (RAG/agents) (K04 §5).
  • Rightsize the model (K08 §5) → does a tuned smaller model clear the floor? Often the biggest lever.
  • Spec decode if low-concurrency latency-bound (K01 §9).
  • Better hardware fit → if decode-bound, a bandwidth/efficiency-optimized chip may win on $/token and perf/watt.

3. Quantify → each lever's throughput gain → recomputed #GPUs → new $/token and monthly bill. Stack them: batching ×8, INT8 ×2, rightsizing — easily past 2×.

4. Validate & present → honest benchmarks, accuracy held on their hardest task, the before/after $/token, and the residual risk. Recommend the smallest safe set of changes that hits the target.

This is the most commercially representative JD2 scenario — it tests whether you optimize with a profiler and a spreadsheet, not vibes.

Walkthrough 5 — Air-gapped on-prem GenAI appliance

Prompt: "A regulated/sovereign customer needs an LLM assistant fully on-prem, no internet, on a fixed hardware budget. Design it." (Bridges to the original Digital Twin folder.)

1. Clarify → hardware on hand (GPUs? CPU-only? edge?), security/compliance constraints, accuracy floor, who operates it after handover.

2. Constraints reshape the stack (K08 §9, original jd.md) → freeze weights + deps, ship the engine, no runtime pip/API. Toolchain: GGUF/llama.cpp/Ollama (CPU/edge) or self-hosted vLLM/Triton (GPU), all offline. Quantize aggressively (INT4 GGUF k-quants) to fit the fixed budget.

3. Design → self-hosted vLLM/Ollama + local vector DB for RAG + tool-calling over internal data (reuse the original folder's labs) + guardrails — all air-gapped, --network none.

4. Size to the fixed budget → work backwards: given N GPUs, what model + quantization + context fits and clears the floor? This is sizing in reverse (K08 §2).

5. Handover → the runbook is the deliverable — they operate it without you (K08 §9). Getting-started, deployment, troubleshooting, validation report, training.

6. Trade-offs → on-prem capex + ops burden vs cloud, fixed capacity vs elasticity, INT4 quality vs fit. Present honestly.

This shows range: you handle the cutting-edge accelerator world and the constrained sovereign-deployment world — exactly the breadth the combined folder demonstrates.

What interviewers score

  • Did you clarify before computing? Jumping to a number without the SLO is an instant junior tell.
  • Did you characterize the workload (prefill/decode, bottleneck class) before choosing optimizations?
  • Did you quantize before sharding and reason about memory vs throughput correctly?
  • Did you give a trade-off curve and a recommendation, not one answer?
  • Did you state assumptions so your sizing is defensible when inputs change?
  • Did you surface the non-obvious binding constraint (decode capacity, KV memory, PCIe-vs-NVLink, power cap)?
  • Did you connect it to $/token and the customer's goal, not just tokens/s?

Nail those seven and you pass, even if your arithmetic is rough. The framework is the signal.