LLM Engineer Interview Prep — All Roles

How to Use This Guide

Each section covers a role archetype, the questions you should expect, and the answers that signal senior-level thinking.

The golden rule: answers must show you understand production tradeoffs, not just definitions.

Role 1: LLM Application Engineer

Who hires: Product companies adding AI features. Usually mid-market SaaS.

What they test: Prompt engineering, RAG basics, API usage, integration skills.

Core Questions

Q: Walk me through how you'd build a customer support chatbot.

Strong answer includes:

  1. Start with a system prompt that defines persona, scope, and guardrails
  2. Add a knowledge base (FAQ documents) via RAG
  3. Add an escalation path ("I'll connect you with a human agent")
  4. Instrument with logging and user satisfaction tracking
  5. Build an eval dataset from real support tickets
  6. Iterate based on evaluation results

Weak answer: "I'd use ChatGPT and write a good prompt."

Q: How do you handle a model that keeps going off-topic?

Strong answer:

  • First, tighten the system prompt with explicit topic restrictions
  • Add few-shot examples of in-scope responses
  • Consider adding a classification step at the start: "Is this question in scope?" before routing to the main model
  • If prompt engineering fails, add an output filter that rejects off-topic responses
  • Measure before and after with an eval set

Q: How would you reduce hallucinations in a RAG system?

Strong answer:

  1. Ground the model: "Answer ONLY based on the provided context. If not in context, say so."
  2. Improve retrieval: better chunking, hybrid search, reranking
  3. Reduce temperature to 0
  4. Add citation requirements (forces the model to link claims to sources)
  5. Add a faithfulness checker after generation
  6. Include negative examples in the system prompt

Role 2: ML Engineer (LLM Inference / Serving)

Who hires: AI-native companies, model serving teams, cloud AI services.

What they test: vLLM, KV cache, batching, quantization, hardware sizing.

Core Questions

Q: Explain how PagedAttention works and why it matters.

Strong answer:

  • Traditional serving: KV cache is allocated contiguously per request. Fragmentation wastes 20-40% of GPU memory.
  • PagedAttention divides KV cache into fixed-size pages (like OS virtual memory).
  • Pages are allocated on demand, not all upfront.
  • This allows more requests to share GPU memory → higher throughput.
  • vLLM uses PagedAttention to achieve ~24x higher throughput than naive serving.

Q: A client wants to serve a 70B parameter model with 8 users simultaneously. Walk me through your hardware and serving setup.

Strong answer:

  • 70B FP16 = ~140GB VRAM needed
  • Two H100-80GB (160GB total) with tensor parallelism = comfortable fit
  • Or: 70B Q4 quantized ≈ 35GB → fits single A100-80GB with room for KV cache
  • Use vLLM with --tensor-parallel-size 2 for the two-GPU setup
  • Set --max-model-len based on expected context length
  • Enable prefix caching if requests share common prefixes (e.g., same system prompt)
  • Monitor: VRAM utilization, batch queue depth, P50/P99 TTFT and TPS

Q: What is continuous batching and why does it improve throughput?

Strong answer:

  • Traditional batching: wait for all requests in a batch to finish before adding new ones. GPU idles waiting for slow requests.
  • Continuous batching (iteration-level scheduling): after each token generation step, add new requests to the batch immediately, drop completed requests.
  • Consequence: GPU utilization goes from ~30-60% to ~85-95%.
  • vLLM, TGI, SGLang all use continuous batching.

Role 3: AI Platform / Infrastructure Engineer

Who hires: Large companies building internal LLM platforms, AI gateways.

What they test: System design, multi-tenant serving, cost management, observability.

Core Questions

Q: Design an internal LLM gateway for a 2000-engineer company.

Strong answer covers:

  • Authentication: API key management, user/team attribution
  • Routing: Route by model name, fall back on failure
  • Cost controls: Per-team token budgets, alerts, hard limits
  • Caching: Semantic caching for repeated queries
  • Observability: Log all requests/responses, latency, token usage, errors
  • Compliance: PII scrubbing before logging, retention policies
  • Provider abstraction: Same API regardless of whether using OpenAI, Anthropic, or internal models

Implementation: LiteLLM proxy as the core, PostgreSQL for usage/budgets, Prometheus + Grafana for metrics.

Q: How would you implement cost controls for 50 different engineering teams?

Strong answer:

  • Allocate monthly token budgets per team (stored in DB)
  • Track usage in real-time per team/model
  • Soft limit: warning at 80% → notify team lead
  • Hard limit: block requests when 100% reached
  • Per-model pricing table (different models cost differently)
  • Monthly rollover vs. carry-over policy
  • Admin dashboard for quota management
  • Anomaly detection: alert if team's usage spikes 10x overnight

Role 4: AI Research Engineer

Who hires: Labs, advanced product teams, model-focused startups.

What they test: Architecture knowledge, training, benchmarks, recent papers.

Core Questions

Q: Explain the difference between GQA and MHA and why GQA matters for inference.

Strong answer:

  • MHA (Multi-Head Attention): every head has its own full Q, K, V matrices
  • GQA (Grouped-Query Attention): multiple Q heads share a single K/V pair per group
  • MQA (Multi-Query Attention): all Q heads share a single K/V pair
  • Why it matters: KV cache size = num_kv_heads × layers × seq_len × head_dim × 2 bytes
  • GQA/MQA dramatically reduces KV cache memory → more requests can share GPU → higher throughput
  • Models using GQA: Llama-3, Mistral, Gemma

Q: What is speculative decoding and when does it help?

Strong answer:

  • Decoding bottleneck: large models are memory-bandwidth bound (not compute-bound) during decode
  • Speculative decoding: use a small "draft" model to generate K tokens cheaply, then verify all K tokens with the large model in a single parallel forward pass
  • If verified: accept all K tokens (major speedup)
  • If rejected: fall back to large model from rejection point
  • Works well when: draft model shares vocabulary with target model, inputs are predictable (common phrases, repetition)
  • Typical speedup: 2-3x on generation-heavy workloads
  • Products using it: vLLM (speculative decoding support), Medusa (multi-head draft)

Role 5: AI Safety / Alignment Engineer

What they test: Red-teaming, RLHF, Constitutional AI, evaluation methods.

Core Questions

Q: How would you design a safety evaluation for a coding assistant?

Strong answer:

  1. Capability eval: Does it write working code? Test on 100 coding problems with automated test running.
  2. Safety eval — intended harm: Can it be prompted to write malware? Exploits? Rate limiters?
  3. Safety eval — unintended harm: Does it suggest insecure patterns (SQL injection, eval() misuse)?
  4. Instruction following: Does it follow system prompt restrictions reliably?
  5. Red team: 20 adversarial scenarios trying to elicit unsafe code
  6. Regression dataset: Locked set of scenarios, must pass all before any model update ships

Role 6: Senior LLM Engineer (Staff+)

What they test: System design, cross-cutting expertise, how you'd lead technical direction.

Core Questions

Q: Your RAG system has 72% user satisfaction. How do you systematically improve it?

Strong answer (decompose the problem):

  1. Instrument the pipeline: log retrieval queries, retrieved chunks, generated answers, and user feedback per session
  2. Identify failure mode distribution: Is it retrieval? Generation? Data freshness?
  3. Retrieval failures: Chunks aren't there → improve ingestion. Chunks exist but not retrieved → improve embedding/search. Retrieved but ranked low → add reranker.
  4. Generation failures: Good chunks, bad answers → improve generation prompt. Model ignores context → add explicit citation instructions.
  5. Data failures: Documents are outdated → add freshness filtering. Documents have wrong granularity → rechunk.
  6. Build a labeled dataset of 100 hard cases. Measure pipeline changes against this dataset.
  7. Set up an A/B test framework to validate changes before full rollout.

Q: How would you architect a system where 100k users each have their own "memory" of past interactions?

Strong answer:

  • Per-user memory store: vector DB partitioned by user_id with metadata (timestamp, conversation_id)
  • Memory ingestion: after each conversation, run a summarization job to extract key facts → store as embeddings
  • Memory retrieval: at conversation start, embed the user's new query → retrieve top-5 relevant memories from their partition
  • Memory lifecycle: set a TTL or limit per user (e.g., 500 memories max → summarize oldest)
  • Privacy: memory store is user-specific, no cross-user contamination possible
  • Scale: with 100k users × 500 memories = 50M vectors → Qdrant or Pinecone at scale
  • Cost: embedding 50M vectors ≈ $5,000 one-time + $10/day for new memories

General Interview Principles

1. Always mention tradeoffs. "X is better than Y for latency, but Y is better for cost at scale."

2. Quantify whenever possible. "That saves about $0.03 per request, which at 10M requests/day is $300k/year."

3. Show you think about failure modes. "The main failure mode is X, so I'd add Y as a safeguard."

4. Demonstrate you've done this in production. If you have experience, anchor on real examples. If not, say "I haven't deployed this at scale but here's how I'd approach it..."

5. Don't pretend to know things you don't. "I don't know the exact implementation of X, but based on what I know about Y, I'd expect..."

System Design Checklist (LLM Edition)

For any LLM system design question, cover:

□ Data flow: input → model → output, with each transformation
□ Model selection: which model, why, what are the tradeoffs
□ Context management: what goes in the prompt, context window limits
□ Serving infrastructure: single model vs. multiple, hosting
□ Latency targets: TTFT, TPS, P50/P99 requirements
□ Cost model: tokens/request × requests/day × $/token
□ Caching strategy: exact match, semantic, prefix
□ Fallback logic: what happens when the primary model fails
□ Evaluation: how do you know it's working
□ Observability: what do you log, what do you alert on
□ Safety: input validation, output validation, rate limiting
□ Privacy: PII handling, data retention