Exercises 04 — Local Inference & Hardware (10)

Concepts: Phase 6. Cheatsheets: 06, 08–11.

Scenario: 13B model, 16GB GPU, FP16.
Task: Does it fit? What to do?
Constraints: One GPU.
Expected answer: FP16 ≈ 26GB weights → no. Quantize to Q4 (~6.5GB) → fits with KV-cache headroom; or use a smaller model. Remember KV-cache + activations add to weights.
Rubric: Senior = memory math (params×bytes) + KV-cache headroom.
Common mistakes: Counting weights only; ignoring KV-cache.
Extension: Compute max context that fits after weights.

Scenario: 7B Q4 model (~4GB) on hardware with 200 GB/s memory bandwidth.
Task: Estimate tok/s.
Constraints: Decode-bound.
Expected answer: tok/s ≈ bandwidth ÷ model bytes ≈ 200/4 ≈ ~50 tok/s (upper bound; real lower). Decode is memory-bandwidth-bound.
Rubric: Senior = applies the bandwidth law, calls it an upper bound.
Common mistakes: Expecting compute (FLOPs) to set decode speed.
Extension: How does quantization change the estimate?

Scenario: Mac M-series laptop, want a quick local OpenAI-compatible endpoint.
Task: Pick a tool.
Constraints: Easy + fast.
Expected answer: Ollama (or MLX for best Apple-native perf). Ollama gives an OpenAI-compatible server on :11434 with one command (cheatsheet 11).
Rubric: Senior = matches tool to use case (dev vs edge vs GPU prod).
Common mistakes: Reaching for vLLM on a Mac laptop.
Extension: When would you switch to llama.cpp or vLLM?

Scenario: Local coding assistant; quality matters but VRAM is tight.
Task: Pick a GGUF quant.
Constraints: Balance.
Expected answer: Start Q4_K_M (best balance); bump to Q5_K_M/Q6_K if VRAM allows (code benefits from precision); avoid Q2/Q3 (cheatsheet 06).
Rubric: Senior = balances size/quality, notes code sensitivity.
Common mistakes: Over-quantizing precision-sensitive tasks.
Extension: Benchmark Q4 vs Q8 on a code eval.

Scenario: M-series with 64GB unified memory runs a 70B Q4 but slowly.
Task: Explain.
Constraints: Mac.
Expected answer: Unified memory lets the big model fit (RAM shared with GPU), but bandwidth caps decode speed (tok/s ≈ bandwidth ÷ bytes). Fits ≠ fast.
Rubric: Senior = separates "fits" (capacity) from "fast" (bandwidth).
Common mistakes: Expecting big-fits = fast.
Extension: Compare to a discrete GPU with higher bandwidth.

Scenario: Local model OOMs at long context though weights fit.
Task: Diagnose + fix.
Constraints: Long docs.
Expected answer: KV-cache grows with context length → OOM. Reduce context (-c/num_ctx), summarize, use KV quantization, or a smaller model. Weights are fixed; KV is the variable cost.
Rubric: Senior = identifies KV-cache as the culprit + mitigations.
Common mistakes: Blaming weights.
Extension: Estimate KV-cache for your context/model.

Scenario: Local decode is too slow for interactive use.
Task: A technique to speed it up without quality loss.
Constraints: Same outputs.
Expected answer: Speculative decoding / MTP — a small draft model proposes tokens the big model verifies in parallel → lower latency, same distribution (Phase 6.07).
Rubric: Senior = explains draft→verify→accept.
Common mistakes: Thinking it changes outputs.
Extension: When does speculative decoding not help? (low acceptance rate).

Scenario: Considering moving from API to self-hosted GPUs.
Task: When is it worth it?
Constraints: Cost-driven.
Expected answer: Past the break-even volume where self-host total (GPU + ops) < API total. APIs win at low/spiky volume; self-host at high steady volume with utilized GPUs. Include ops/engineering cost (Phase 15.05).
Rubric: Senior = includes ops cost + utilization caveat.
Common mistakes: Comparing GPU $ to API $ only.
Extension: Compute the break-even for a given volume.

Scenario: You fine-tuned a LoRA; want to run it locally.
Task: Outline the path.
Constraints: llama.cpp/Ollama.
Expected answer: Merge LoRA into base → convert to GGUF (convert_hf_to_gguf.py) → quantize (llama-quantize ... Q4_K_M) → run in llama.cpp/Ollama (cheatsheet 10, Phase 13.07).
Rubric: Senior = merge→convert→quantize→serve.
Common mistakes: Forgetting to merge the adapter.
Extension: When keep the adapter separate (multi-LoRA) instead?

Scenario: Healthcare client can't send data to any external API.
Task: Architecture recommendation.
Constraints: Air-gapped.
Expected answer: Self-host an open-weight model on-prem/air-gapped (no external calls) — local inference is the strongest privacy posture (Phase 6, Phase 14.02).
Rubric: Senior = ties local inference to residency/air-gap compliance.
Common mistakes: Suggesting an API with "we promise not to log."
Extension: What ops burden does on-prem add? (Phase 15.08)

Next: 05 — Production Serving · Index: exercises/

LLM Inside-Out Mastery