Interview Prep 01 — LLM Application Engineer

Builds LLM-powered features and products on top of model APIs. The most common LLM role. Focus: turning models into reliable, evaluated, cost-aware product features. Phases: 1, 5, 9, 10, 12, 14.

1. Core concepts (must explain cold)

Tokens/context/temperature; prefill vs decode; streaming + TTFT vs total latency (cheatsheet 01).
RAG vs fine-tuning vs prompting (knowledge vs behavior; the ladder).
Tool/function calling + the trust boundary (model proposes, app executes).
Structured output: constrained ≠ correct; validate semantically.
Evaluation: golden set, metric per task, LLM-judge caveats.
Cost: cost per resolved task; caching; routing.
Prompt injection basics; never put secrets in prompts.

2. System design questions

Design a customer-support assistant over a company's docs (RAG + escalation). (Cover: ingestion/chunking, retrieval+rerank, grounding+citations, eval split, tenant isolation, cost, fallback.)
Design a "summarize + extract structured fields" feature for 50k docs/day. (Batch vs realtime, schema validation, cost per doc, retries.)
Add an AI feature to an existing SaaS without blowing up latency/cost. (Routing, caching, async, streaming.)

Use the system-design template; proactively address cost/latency/reliability/safety.

3. Debugging questions

RAG returns "I don't know" though the doc exists → retrieval vs generation isolation (exercise RA1).
JSON validates but values are wrong → constrained ≠ correct; add semantic validation.
Feature works in dev, flaky in prod → nondeterminism (temp), no retries/timeouts, rate limits.
Cost doubled overnight → output length/loops; attribute + cap.

4. Coding exercises

Implement a chat call with streaming + tool calling + structured output (OpenAI SDK) (labs/lab-01).
Build a minimal RAG Q&A with citations (labs/lab-03).
Add a JSON-schema validation + repair loop (labs/lab-10).

5. Architecture exercises

Draw the RAG architecture and the agent loop from memory; mark the trust boundary.
Design retry/fallback/timeout wrapping around provider calls.

6. Model selection questions

Choose a model for a latency-sensitive chat feature on a budget → framework + routing (cheatsheet 04).
When do you use a reasoning model vs a normal one? (hard reasoning only; cost/latency).
Fill a selection memo for one feature.

7. Production incident questions

A jailbreak makes the assistant go off-policy → output guardrails (fail closed), red-team, trust boundary.
A provider outage takes the feature down → fallback + circuit breaker (template 04).
Users see another user's data in answers → tenant isolation (retrieval filter) (Phase 14.04).

A grounded RAG assistant over a real corpus with: hybrid search + rerank + citations, a golden-set eval (retrieval recall@k + generation faithfulness), streaming UI, cost-per-query instrumentation, and a prompt-injection test. README with the numbers.

9. 30-minute drill

5 min: define tokens/context/temperature/RAG-vs-FT/trust-boundary.
10 min: design a doc-Q&A feature (whiteboard the RAG diagram).
10 min: debug "RAG says I-don't-know" + "JSON wrong values."
5 min: how you'd eval + control cost.

10. 2-hour take-home

Build a RAG Q&A over provided docs: ingestion + retrieval + grounded generation with citations; a 10-question eval (retrieval + faithfulness); a short writeup of failure cases, cost per query, and how you'd productionize (isolation, caching, fallback).

11. Senior-level expectations

Junior: wires up the SDK; gets a demo working.
Mid: ships a RAG/agent feature with basic eval and error handling.
Senior: owns the eval harness + cost/latency budget; isolates retrieval vs generation failures; designs for injection/isolation; routes models for margin; every claim has a number.
Principal: sets the org's eval/cost/safety standards; designs the platform features other teams reuse; anticipates second-order effects (drift, vendor risk).

Next: 02 — LLM Platform Engineer · Index: interview-prep/

LLM Inside-Out Mastery