Exercises 01 — Vocabulary (15)

Format per exercise: Scenario · Task · Constraints · Expected answer · Rubric · Common mistakes · Extension. Concepts: Phase 1, Phase 2. Cheatsheet: 01.

V1 — Tokens vs words

  • Scenario: A PM says "our prompt is 500 words, that's 500 tokens."
  • Task: Correct them and give the rule of thumb.
  • Constraints: English text.
  • Expected answer: Tokens are sub-word units; ~1 token ≈ 0.75 words (~4 chars), so 500 words ≈ ~650–700 tokens. Pricing and context limits are in tokens, not words.
  • Rubric: Mid = states ratio. Senior = notes it varies by tokenizer/language (code and non-English tokenize worse) and ties it to cost/context.
  • Common mistakes: Assuming 1:1; ignoring that output tokens are billed separately (often higher).
  • Extension: Estimate tokens for a 10-page PDF and the input cost at a given $/Mtok.

V2 — Context window

  • Scenario: A 128k-context model errors on a 130k-token document + question.
  • Task: Explain why and two fixes.
  • Constraints: Single request.
  • Expected answer: Context = input + output tokens, capped at 128k; 130k input exceeds it (before leaving room for output). Fixes: chunk + RAG (Phase 9); summarize/compress; use a larger-context model; reserve output budget.
  • Rubric: Senior = remembers max-output is often smaller than context and budgets for it.
  • Common mistakes: Thinking context is input-only; forgetting output budget.
  • Extension: When is bigger context worse than RAG? (cost, lost-in-the-middle, latency).

V3 — Temperature

  • Scenario: A classification endpoint gives different labels for the same input across calls.
  • Task: Diagnose and fix.
  • Constraints: Need deterministic labels.
  • Expected answer: Temperature > 0 causes sampling variation. Set temperature=0 (and a seed if supported). Note: even temp 0 isn't perfectly deterministic across providers/batching.
  • Rubric: Senior = mentions seed + residual nondeterminism + same-provider for tests.
  • Common mistakes: Blaming the model; setting temp high "for variety" on a deterministic task.
  • Extension: Which tasks want temperature > 0, and why?

V4 — Prefill vs decode

  • Scenario: Interviewer: "Why is the first token slow but the rest stream fast — and what makes 'the rest' slow on big models?"
  • Task: Explain both phases.
  • Constraints: 60 seconds.
  • Expected answer: Prefill processes the whole prompt in parallel (compute-bound) → sets TTFT. Decode generates one token at a time (sequential, memory-bandwidth-bound) → sets TPOT. Big models decode slowly because each token reloads weights from VRAM: tok/s ≈ bandwidth ÷ model bytes.
  • Rubric: Senior = names the bandwidth law and the compute-vs-memory distinction.
  • Common mistakes: Saying decode is compute-bound; conflating TTFT and throughput.
  • Extension: How do KV-cache and speculative decoding change each phase?

V5 — KV-cache

  • Scenario: VRAM usage grows as a chat conversation gets longer, eventually OOMs.
  • Task: Explain the cause and the memory scaling.
  • Constraints: Single long conversation.
  • Expected answer: The KV-cache stores keys/values per token to avoid recomputation; it grows with sequence length × layers × heads × batch. Long context = more KV-cache VRAM → eventual OOM. Fixes: cap context, summarize history, paged/quantized KV, smaller model.
  • Rubric: Senior = gives the scaling factors and mitigations.
  • Common mistakes: Blaming weights (weights are fixed); forgetting batch multiplies it.
  • Extension: Estimate KV-cache size for 32k context on a 70B model.

V6 — RAG vs fine-tuning

  • Scenario: "The model doesn't know our internal product docs. Should we fine-tune?"
  • Task: Recommend and justify.
  • Constraints: Docs change weekly.
  • Expected answer: Use RAG — fine-tuning teaches behavior, not facts, and weekly-changing facts would require constant retraining. RAG retrieves current docs at query time. Fine-tune only for behavior/format the model can't be prompted into.
  • Rubric: Senior = cites the ladder (prompt→few-shot→RAG→fine-tune) and maintenance burden.
  • Common mistakes: Defaulting to fine-tuning for knowledge; thinking fine-tuning "adds facts" reliably.
  • Extension: When would you do both RAG and fine-tuning?

V7 — Hallucination

  • Scenario: A legal assistant cites a non-existent case.
  • Task: Define hallucination and give 3 mitigations.
  • Constraints: High-trust domain.
  • Expected answer: Confident but false/unsupported output. Mitigations: RAG with grounding + citations (Phase 9.08); output validation/faithfulness check; "say 'I don't know'" prompting; human approval for high-impact (Phase 10.05).
  • Rubric: Senior = ties to faithfulness eval and trust-as-product.
  • Common mistakes: "Just prompt it to not hallucinate" as the only fix.
  • Extension: How do you measure hallucination rate?

V8 — Embeddings

  • Scenario: A junior asks "what's an embedding and why do we need a vector DB?"
  • Task: Explain.
  • Constraints: From zero.
  • Expected answer: An embedding is a vector capturing text meaning; similar texts → nearby vectors. A vector DB does fast nearest-neighbor (ANN) search over embeddings → semantic retrieval for RAG. Keyword search can't match paraphrases; embeddings can.
  • Rubric: Senior = mentions ANN (HNSW/IVF) and hybrid (dense+BM25).
  • Common mistakes: Confusing embeddings with the generative model.
  • Extension: Why hybrid search instead of pure vector?

V9 — Tool / function calling

  • Scenario: "Does the model run my send_email function?"
  • Task: Explain the protocol and the trust implication.
  • Constraints: Security-sensitive.
  • Expected answer: No — the model proposes a structured tool_call; your app executes it after validation/permission checks. This is the trust boundary that defuses prompt injection (Phase 14.01).
  • Rubric: Senior = states model-proposes/app-executes and why it's a security control.
  • Common mistakes: Believing the model executes code; trusting tool args blindly.
  • Extension: Why is "constrained output" not the same as "correct output"?

V10 — Quantization

  • Scenario: A 70B model won't fit on a 48GB GPU in FP16.
  • Task: Explain quantization and whether it helps.
  • Constraints: One GPU.
  • Expected answer: Store weights in fewer bits (FP16→INT4) → ~140GB→~35GB; fits, and decode gets faster (bandwidth-bound). Small quality loss; Q4_K_M/AWQ are common. (cheatsheet 06)
  • Rubric: Senior = gives memory math + names methods + notes quality tradeoff.
  • Common mistakes: Thinking quantization is free; forgetting KV-cache still needs room.
  • Extension: When does quantization hurt (math/code precision)?

V11 — MoE / active vs total params

  • Scenario: A model card says "total 100B, active 12B."
  • Task: Explain and the implications.
  • Constraints: Selection context.
  • Expected answer: Mixture-of-Experts routes each token to a few experts → active params used per token (12B → cheaper compute) but total (100B) must fit in memory. So: memory like a 100B, compute like a 12B.
  • Rubric: Senior = separates memory (total) from compute/throughput (active).
  • Common mistakes: Treating active=total; underestimating memory.
  • Extension: Why might an MoE be cheaper per token but harder to serve?

V12 — Reasoning models

  • Scenario: A "reasoning" model is slower and pricier than expected.
  • Task: Explain why and when to use one.
  • Constraints: Cost-sensitive.
  • Expected answer: Reasoning models generate hidden reasoning tokens before the answer → more tokens, higher latency/cost. Use only for genuinely hard multi-step reasoning; use a normal model for simple tasks (route).
  • Rubric: Senior = ties to routing and cost-per-resolved-task.
  • Common mistakes: Using a reasoning model for everything.
  • Extension: How does CoT prompting relate to reasoning models?

V13 — Streaming / SSE

  • Scenario: Users complain the app "feels slow" though throughput is fine.
  • Task: Recommend a UX fix.
  • Constraints: Chat UI.
  • Expected answer: Stream tokens via SSE so users see output as it's generated → perceived latency drops to TTFT. Throughput unchanged, but UX much better.
  • Rubric: Senior = distinguishes perceived (TTFT) vs total latency.
  • Common mistakes: Optimizing total latency while ignoring streaming.
  • Extension: What breaks when you stream + need output validation/guardrails?

V14 — p95 vs average latency

  • Scenario: Avg latency is 800ms but users complain it's "sometimes really slow."
  • Task: Explain the right metric.
  • Constraints: SLA discussion.
  • Expected answer: Average hides the tail; measure p95/p99. A good avg with a bad p99 means many users hit slow requests. SLAs and capacity should target tail latency under load.
  • Rubric: Senior = "under load" + percentiles + why tails matter for UX.
  • Common mistakes: Reporting averages; measuring single-request latency.
  • Extension: What causes tail latency in LLM serving? (batching, KV pressure, cold starts).

V15 — Agent reliability compounding

  • Scenario: A 10-step agent "works in the demo" but fails ~40% of the time in prod.
  • Task: Explain mathematically and give two fixes.
  • Constraints: Multi-step task.
  • Expected answer: Per-step reliability compounds: 0.95^10 ≈ 0.60 → ~40% failure. Fixes: fewer steps, validation/retry per step, human approval, better tools, eval per step (Phase 10.09). "Least-agentic that works."
  • Rubric: Senior = does the math and proposes reducing step count + per-step checks.
  • Common mistakes: Blaming the model only; adding more autonomy.
  • Extension: How would you instrument per-step reliability?

Next: 02 — Model Cards · Index: exercises/