Interview Prep — Senior AI Engineer (LLMs, VLMs, Agents)

A running reference for the questions a strong AI/research panel actually asks — and the level of answer that separates "uses the API" from "owns the model stack." Pair each section with the phase that builds the mechanism, so your answer is "I implemented this," not "I read about it."

How senior AI interviews differ

A junior candidate explains what a model/framework does. A senior explains how it works under the hood, when it breaks, what it costs in FLOPs/bytes/latency, and what they'd trade. The loop in a strong panel is usually: (1) a from-scratch coding round (implement attention / a sampler / LoRA / the DPO loss), (2) an ML-depth round (derive a gradient, reason about a training failure), (3) a serving/systems round (size a deployment, debug latency), (4) an agent/system-design round, and (5) research taste / "what did you build and why." Every answer should end with a tradeoff and, where possible, a number.

1. Foundations & scaling (P00)

"How many FLOPs to train a 7B model on 1T tokens? How much GPU memory to serve it?" — 6ND ≈ 4.2e22 to train; weights 14 GB fp16 (7 GB int8) plus the KV-cache, which grows with batch×context — show that the KV-cache, not the weights, is what OOMs you.
"Is decoding compute- or memory-bound? Why does that decide your whole serving strategy?" — memory-bandwidth bound; you re-read all weights per token, so batching (amortize the read) and quantization (fewer bytes) are the levers, not more FLOPs.
"Chinchilla — what did it change?" — compute-optimal is ~20 tokens/param; most early large models were under-trained, so a smaller model on more data beats a bigger under-trained one at equal compute and is cheaper to serve.
"Train a bigger model or fine-tune a smaller one? Defend with numbers." — inference cost is forever (2N/token × every request); quantize/distill/PEFT a small model unless the quality gap is measured and the volume justifies the perpetual bill.

2. Transformer internals (P02)

"Implement attention on a whiteboard. Why √d_k? Why subtract the max in softmax?" — scale stops dot products from saturating softmax; max-subtraction is overflow safety (mathematically identical). Be ready to write the causal mask.
"MHA vs MQA vs GQA — what's the tradeoff?" — fewer K/V heads shrink the KV-cache (the serving memory wall) at a small quality cost; GQA is the modern compromise.
"Why RoPE over learned absolute positions?" — relative, parameter-free, extrapolates to longer context; explain the rotation-of-(q,k) intuition.
"Where do the parameters live in a transformer block?" — ~2/3 in the FFN, ~1/3 in attention projections; matters for what to shard (TP) and what to quantize.

3. Training & distributed (P03, P10)

"Derive the gradient of softmax+cross-entropy." — it collapses to (p − y); if you can't show this you haven't earned the training round.
"Your loss spikes to NaN at step 4000. Debug it." — LR too high / no warmup, missing grad clip, fp16 overflow (need loss scaling/bf16), bad data batch; bisect by reverting the LR schedule and checking grad norms.
"A 70B model won't fit for training. Walk me through the parallelism." — DP for throughput, TP to split the matmuls within a node (high comms ⇒ NVLink), PP across nodes, ZeRO/FSDP to shard optimizer+grads+params; name where the bubble and the all-reduce live.
"Explain ring all-reduce and why it's bandwidth-optimal." — 2(N−1) steps each moving 1/N of the tensor; per-GPU traffic is ~2× the tensor regardless of N.

4. Multimodal / VLMs (P04)

"How does an LLM 'see' an image?" — a vision encoder (ViT) produces patch features; a projector maps them into the token-embedding space; they're prepended as "tokens" the LLM attends over (LLaVA), or fused via cross-attention (Flamingo).
"Explain CLIP's training objective." — contrastive InfoNCE: a batch's similarity matrix, diagonal = matched pairs = the target; temperature sharpens it. Enables zero-shot + retrieval.
"Projector tuning vs full fine-tune for a new VLM task?" — freeze encoders, train the projector (+ LoRA on the LLM) first; cheap, preserves pretraining, usually enough.

5. Fine-tuning & compression (P05, P06)

"How does LoRA work and why is inference free after merging?" — ΔW = (α/r)BA, train r·(d+k) params; merge into W so the served model is identical-shape, zero added latency.
"QLoRA — how do you fine-tune 65B on one GPU?" — NF4 4-bit frozen base + double quant + paged optimizer states + LoRA adapters in bf16; only the adapters get gradients.
"int8 vs int4 — when does quality fall off, and how do you protect it?" — int8 ~lossless with per-channel weights; int4 needs outlier handling (AWQ/GPTQ); activations are harder than weights. Always measure on your eval, not perplexity alone.
"Distillation — what's the loss and why temperature?" — KL of softened teacher/student logits (×T²); T exposes the teacher's "dark knowledge" (relative wrong-class probs).

6. Alignment (P07)

"RLHF in three stages." — SFT → reward model (Bradley-Terry on preferences) → PPO with a KL penalty to the SFT policy.
"DPO vs PPO — why did the field move to DPO?" — DPO has a closed-form loss on preference pairs (no reward model, no rollout, no RL instability) using the frozen reference policy and temperature β; simpler, cheaper, more stable — at some loss of online exploration.
"What's reward hacking and how do you prevent it?" — the policy exploits reward-model flaws; the KL/reference term and held-out human eval are the guards.

7. Decoding (P08)

"top-k vs top-p vs min-p — pick one for a JSON-extraction service and defend it." — temperature=0 + constrained decoding (mask to schema-valid tokens), not sampling at all; sampling is for open-ended generation.
"How do you guarantee valid JSON / a valid function call?" — a grammar/FSM that masks the logits to only valid next tokens each step; prompting alone is best-effort.

8. Serving (P09, P17)

"Explain PagedAttention and continuous batching like I'm a new hire." — KV-cache in fixed blocks like OS pages (no fragmentation, prefix sharing) + sequences joining/leaving the batch every step (no head-of-line blocking) ⇒ much higher GPU occupancy.
"Speculative decoding — does it change the output?" — no; the verification step makes the accepted tokens distributionally identical to the target model; you just get them 2–3× faster.
"Your p99 TTFT is fine but TPOT is bad. Diagnose." — decode is memory-bound; check batch occupancy, KV-cache pressure (evictions), quantize the KV/weights, or shard with TP.
"DeepSpeed vs vLLM vs TensorRT-LLM vs ONNX Runtime — when each?" — vLLM for throughput serving (PagedAttention), TensorRT-LLM for lowest latency on NVIDIA (compiled kernels), DeepSpeed for training+inference at scale, ONNX Runtime for portability/edge.

9. RAG & retrieval (P11)

"RAG quality is bad. Is it retrieval or generation?" — measure separately: recall@k for the retriever, faithfulness for the generator. Most bugs are retrieval (chunking, embedding mismatch, missing reranker).
"HNSW vs IVF vs flat — pick for 100M vectors at p99 < 50ms." — HNSW for low-latency high-recall in memory; IVF/IVF-PQ when memory-bound; flat only for small/exact. Name the recall–latency–memory triangle.

10. Agents (P12–P14)

"Design an agent that books travel. What can go wrong and how do you contain it?" — typed tools with validation, a step cap, errors-as-observations, human approval for irreversible actions, and a verifier/critic; idempotent tool calls so a retry doesn't double-book.
"Single agent vs multi-agent — when is multi-agent actually worth it?" — when roles are genuinely distinct (planner/executor/critic) or tasks parallelize; otherwise it adds latency, cost, and coordination bugs for little gain. Have a number.
"Where does neurosymbolic help?" — when answers must be verifiable (math, code, config, constraints): LLM proposes, a deterministic checker accepts/rejects; you get LLM flexibility with symbolic correctness.

11. Evaluation & safety (P16)

"How do you know a model change is actually better?" — a fixed eval set, the right metric per task, a confidence interval, and a regression gate in CI — never a vibe check on one prompt.
"LLM-as-judge — what are its biases and how do you control them?" — position, verbosity, self-preference; randomize order, use a rubric, calibrate against human labels.

The system-design loop (use in every round)

Clarify the workload: task, modality (text/image/both), QPS, prompt/response token sizes, latency SLO (TTFT/TPOT), quality bar, safety/compliance, budget.
Pick the model strategy: API vs open-weights; size; fine-tune (PEFT) vs prompt vs RAG; quantization target. Justify with the FLOPs/memory/cost math.
Design the serving path: tokenizer → model (sharding?) → sampler/constraints → cache → batching → autoscaling; and the offline path (training/fine-tune/eval pipeline).
Add retrieval/agents only if needed, and design their failure containment.
Quantify: tokens/sec, $/1M tokens, KV-cache memory, recall@k, eval score with CI.
Fail it: hallucination, prompt injection, tool misuse, cost blowup, drift, the rollback.

See ../system-design/README.md for worked reference designs.

Senior AI Engineer — LLMs, VLMs & Agentic Systems