Exercises 02 — Model Card Interpretation (10)

Concepts: Phase 3. Cheatsheet: 02.

MC1 — Context vs max output

  • Scenario: Card: "context 200k, max output 8k." Team wants 50k-token reports.
  • Task: Is it possible? What to do?
  • Constraints: Single call.
  • Expected answer: No — max output is 8k regardless of the 200k context. Generate in sections (map-reduce), use a model with larger max output, or chunk the report.
  • Rubric: Senior = distinguishes context from max-output and proposes chunked generation.
  • Common mistakes: Assuming context = output capacity.
  • Extension: Design a long-report pipeline within an 8k output cap.

MC2 — Self-reported benchmarks

  • Scenario: A card touts "92% on MMLU, SOTA."
  • Task: How much should this drive selection?
  • Constraints: Production task differs from MMLU.
  • Expected answer: Use it to shortlist only. Benchmarks are self-reported and may be contaminated; they rarely match your task. Decide on your golden-set eval (Phase 12.01).
  • Rubric: Senior = "benchmarks shortlist, my eval decides" + contamination awareness.
  • Common mistakes: Selecting on leaderboard numbers.
  • Extension: How would you detect benchmark contamination?

MC3 — Knowledge cutoff

  • Scenario: A support bot gives outdated API instructions.
  • Task: Identify the card field and the fix.
  • Constraints: Docs update monthly.
  • Expected answer: The knowledge cutoff predates the changes. Fix with RAG over current docs (not fine-tuning) — facts change too often (Phase 9).
  • Rubric: Senior = cutoff → RAG, not retraining.
  • Common mistakes: Expecting the model to "just know" current facts.
  • Extension: How do you keep the RAG index fresh?

MC4 — License / weights

  • Scenario: Startup wants to self-host an open-weight model commercially.
  • Task: What to check on the card?
  • Constraints: Commercial SaaS.
  • Expected answer: The license — "open weights" ≠ free commercial use. Check restrictions (commercial use, scale caps, competing-model training). Apache/MIT permissive; some are custom/restricted (cheatsheet 05).
  • Rubric: Senior = knows open-weights ≠ open-source ≠ free-commercial.
  • Common mistakes: Assuming "open" means unrestricted.
  • Extension: Compare two real licenses' commercial terms.

MC5 — Model card vs system card

  • Scenario: Security asks for safety/red-team details; you have a model card.
  • Task: What document do you need and why?
  • Constraints: Enterprise review.
  • Expected answer: A system card (frontier labs) covers safety mitigations, red-team results, policy — the deployment system, not just the model. Model cards focus on capability/license.
  • Rubric: Senior = clean model-card vs system-card distinction.
  • Common mistakes: Conflating the two.
  • Extension: What safety info would you require before shipping in healthcare?

MC6 — Capabilities flags

  • Scenario: You need structured JSON output and tool calling.
  • Task: Which card fields gate the choice?
  • Constraints: Agent product.
  • Expected answer: Check tool/function calling and structured output support; without native support you bolt on fragile parsing. Also verify reasoning/vision if needed (cheatsheet 03).
  • Rubric: Senior = treats these as hard constraints in selection.
  • Common mistakes: Assuming all models support tool calling.
  • Extension: How do you handle structured output on a model lacking native support?

MC7 — Active vs total params (MoE)

  • Scenario: Card: "total 200B, active 20B." You must size hardware.
  • Task: Memory vs compute implications.
  • Constraints: Self-host.
  • Expected answer: Memory must hold 200B (total); compute/throughput behaves like 20B (active). Size VRAM for total, expect cheaper per-token compute.
  • Rubric: Senior = separates memory (total) from compute (active).
  • Common mistakes: Sizing memory for active params.
  • Extension: Why are MoE models harder to serve efficiently?

MC8 — Pricing asymmetry

  • Scenario: Card: input $0.15/Mtok, output $0.60/Mtok. Chatty assistant.
  • Task: Which dominates cost and why?
  • Constraints: Long answers.
  • Expected answer: Output dominates (4× price here) for chatty/long-output tasks. Cap max_tokens, prefer concise outputs, model cost on output-heavy workloads (Phase 15.05).
  • Rubric: Senior = identifies output-token dominance + mitigations.
  • Common mistakes: Modeling only input cost.
  • Extension: Re-derive cost-per-resolved-task with retries.

MC9 — Reasoning-token cost

  • Scenario: A reasoning model's bill is 3× the visible output tokens.
  • Task: Explain.
  • Constraints: Card mentions "reasoning."
  • Expected answer: Reasoning models bill hidden reasoning tokens beyond the visible answer → real cost exceeds visible output. Budget for it; route simple tasks to non-reasoning models.
  • Rubric: Senior = anticipates hidden token cost and routes.
  • Common mistakes: Estimating cost from visible output only.
  • Extension: How to cap/limit reasoning effort where supported?

MC10 — Comparing two cards

  • Scenario: Two candidate models; you have both cards.
  • Task: Build a comparison and a decision process.
  • Constraints: One page.
  • Expected answer: Tabulate context, max output, price (in/out), capabilities, cutoff, license, latency class; filter on hard constraints; shortlist; then decide on your eval, not the cards (template).
  • Rubric: Senior = cards shortlist; eval decides; documents in a memo.
  • Common mistakes: Picking the "better card" without testing.
  • Extension: Fill the model-selection memo for a real task.

Next: 03 — Model Selection · Index: exercises/