Local Model Debugging

Phase 6 · Document 08 · Local Inference Prev: 07 — MTP and Speculative Decoding · Up: Phase 6 Index

Table of Contents

  1. Why This Matters
  2. Core Concept
  3. Mental Model
  4. Hitchhiker's Guide
  5. Warmup Readings
  6. Deep Readings and External References
  7. Key Terms
  8. Important Facts
  9. Observations from Real Systems
  10. Common Misconceptions
  11. Engineering Decision Framework
  12. Hands-On Lab
  13. Verification Questions
  14. Takeaways
  15. Artifact Checklist

1. Why This Matters

Everything in Phase 6 comes together when something breaks — and locally, it breaks in a handful of recognizable ways: it won't load, it OOMs (sometimes only later), it's painfully slow, or it produces fluent-looking garbage. The difference between an hour of flailing and a 5-minute fix is having a diagnostic decision tree that maps each symptom to its small set of root causes — almost all of which trace back to the four pillars you've already learned: memory (02), hardware/bandwidth (01), the chat template/quant (03, 06), and the engine's flags (04). This is the doc that makes you fast and calm when a local model misbehaves.

2. Core Concept

Plain-English primer: four symptoms, known causes

Local inference failures cluster into four buckets. Learn the symptom → cause mapping and you've learned 90% of local debugging:

A) Won't load / crashes at startup.

  • Not enough memory for the weights — weights_GB > available (02). Fix: lower quant / smaller model / bigger device.
  • Format/engine mismatch — feeding safetensors to llama.cpp, a GGUF to MLX, or a too-new architecture to an old engine build. Fix: convert to the right format / update the engine.
  • Corrupt or partial download. Fix: re-download; verify checksum/size.

B) Out-of-memory (OOM) — especially later.

  • Loads fine, then OOMs after a long chat or under concurrency: the KV cache grew past budget (02). KV scales with context × concurrency. Fix: cap -c/num_ctx, cap parallelism, quantize the KV cache, or quantize weights harder.
  • No headroom — you filled memory to ~100%; OS/activations spill. Fix: leave ≥20%.

C) Slow (low tokens/sec).

  • Running on CPU when a GPU exists — forgot -ngl (llama.cpp) or wrong/no GPU backend in the build. The #1 cause. Fix: -ngl 99, GPU-enabled build (03).
  • Partial offload — some layers on CPU (-ngl too low / doesn't fit). The slow layers dominate (non-linear) (01). Fix: fit more layers (lower quant) or accept.
  • Memory swapping — exceeded RAM/unified memory → disk paging (catastrophic). Fix: smaller model/quant; watch memory pressure (05).
  • Thermal throttling — sustained load drops clocks (laptops especially). Fix: cooling; expect lower sustained tok/s.
  • You measured prefill, not decode — a huge prompt makes TTFT dominate; that's compute-bound, not "slow decode" (Phase 2.07).

D) Garbage / wrong output (but fluent-ish).

  • Wrong/missing chat template — the model isn't seeing the role format it was trained on, so it rambles, never stops, or ignores the system prompt (what-happens §0). Fix: use the model's correct template (GGUF usually embeds it; custom converts may not).
  • Wrong stop tokens / EOS — it runs on forever or cuts mid-word. Fix: set the right stop/EOS for the model.
  • Over-aggressive quantization — 2–3-bit (or RTN) breaks reasoning/structure (06). Fix: higher bit-width / better method.
  • Tokenizer mismatch — wrong tokenizer for the weights → nonsense. Fix: matched tokenizer (GGUF bundles it; convert carefully).
  • Sampling too hot — temperature way up → incoherence (Phase 1.03). Fix: lower temperature.

The unifying diagnostic question

For any symptom, ask: memory, hardware, format/template, or flags/sampling? Almost every local bug lives in exactly one of those four, and you already know each. The rest of this doc is the decision tree and the commands.

3. Mental Model

   SYMPTOM ────────────────► LIKELY BUCKET ───────────► FIRST FIX
   won't load              MEMORY / FORMAT          quant↓ / convert / re-download
   OOM later               MEMORY (KV grew!)        cap ctx & concurrency, KV-quant
   slow (low tok/s)        HARDWARE / FLAGS         -ngl 99, GPU build, no swap/throttle
   garbage but fluent      FORMAT/TEMPLATE / QUANT  correct template, EOS, higher quant

   THE FOUR PILLARS behind every local bug:
     MEMORY [02]   ·   HARDWARE/BANDWIDTH [01]   ·   TEMPLATE/QUANT [03,06]   ·   FLAGS/SAMPLING [04]
   ALWAYS read the startup log: "offloaded N/M layers", context size, KV bytes.

Mnemonic: classify the symptom → one of four pillars → apply the known fix. Read the startup log first.

4. Hitchhiker's Guide

What to look for first: the engine's startup log — it tells you offloaded layers (N/M), context size, KV allocation, and the chat template it chose. Most root causes are visible there before you change anything.

What to ignore at first: rewriting prompts when the output is garbage — first rule out template/quant; a bad template makes any prompt fail.

What misleads beginners:

  • Blaming the model for garbage that's actually a template/EOS bug.
  • Blaming "the GPU is weak" for slowness that's actually CPU offload (-ngl missing).
  • Thinking an OOM at hour two is random — it's KV growth, fully predictable from 02.
  • Trusting that "it fit" from a short test — concurrency/long context will re-test it.

How experts reason: they classify the symptom into one of four buckets in seconds, confirm with one command (nvidia-smi, ollama ps, the startup log), apply the known fix, and re-measure. They size memory and set caps before deploying so OOM/throttle don't surprise them.

What matters in production: hard context and concurrency caps (so KV can't OOM), pinned model+quant+template (so behavior doesn't drift, Phase 5.10), memory/latency monitoring, and a smoke test that checks output quality (not just 200 OK) to catch template/quant regressions.

How to debug/verify: reproduce with a minimal prompt; toggle one variable at a time (-ngl, quant, template, num_ctx, temperature); compare tok/s to the bandwidth ceiling; diff against a known-good config.

Questions to ask: What does the startup log say about offload/context/template? What's peak memory vs device capacity? Is this prefill or decode slowness? Is the template/quant the published one?

What silently gets expensive/unreliable: uncapped context/concurrency (latent OOM), partial offload (quiet 2–5× slowdown), thermal throttling on laptops, and template drift after a model update.

5. Warmup Readings

TitleWhy to read itWhat to extractDifficultyTime
02 — RAM/VRAM/UnifiedOOM is a memory-math storyKV growthBeginner25 min
01 — Hardware LiteracySlowness is bandwidth/offloadceiling, offloadBeginner25 min
03 — GGUF and llama.cpp-ngl, templates, flagsstartup logBeginner20 min
06 — Quantization GuideGarbage from over-quantquality cliffIntermediate20 min

6. Deep Readings and External References

TitleURLWhy it mattersRead firstLab connection
llama.cpp troubleshooting/discussionshttps://github.com/ggml-org/llama.cpp/discussionsReal symptom threadsoffload, templateAll buckets
Ollama FAQ/troubleshootinghttps://github.com/ollama/ollama/blob/main/docs/faq.mdGPU detection, memorynum_ctx, GPUBucket B/C
nvidia-smi docshttps://docs.nvidia.com/deploy/nvidia-smi/Read VRAM/util/throttlethe columnsBucket B/C
HF chat templateshttps://huggingface.co/docs/transformers/chat_templatingWhy templates matterapply_chat_templateBucket D
Phase 5.10 — Provider Variance(curriculum)Drift from quant/templateper-endpoint behaviorPinning

7. Key Terms

TermSimple meaningTechnical meaningWhy it mattersWhere it appearsHow to use it
OOMOut of memoryAllocation exceeds device memoryCrash / failed loadlogsCap ctx/concurrency [02]
OffloadLayers on GPU-ngl countSpeedstartup logSet to 99 [03]
SwappingSpill to diskRAM exceeded → pagingCatastrophic slowdownActivity MonitorSmaller model [05]
ThrottlingClock dropThermal/power limitLower sustained tok/snvidia-smiCooling
Chat templateRole formattingSpecial-token wrappingGarbage if wrongmodel filesUse correct one [03]
EOS / stopEnd tokenGeneration terminatorRunaway/cutoffconfigSet per model
Tokenizer mismatchWrong vocabTokenizer ≠ weightsNonsenseconvertsMatch exactly
Startup logEngine boot outputOffload/ctx/KV/templateFirst diagnosticterminalRead it first

8. Important Facts

  • Almost every local bug is memory, hardware, template/quant, or flags/sampling — classify first.
  • OOM-later = KV cache growth (context × concurrency), not randomness (02).
  • #1 slowness cause is CPU execution — missing -ngl or a non-GPU build (03).
  • Partial offload is non-linear — a few CPU layers can dominate latency (01).
  • Fluent garbage ≈ wrong chat template / EOS / tokenizer — not a "bad model."
  • Over-aggressive quant (≤3-bit/RTN) breaks reasoning & structure (06).
  • Swapping (RAM exceeded) and thermal throttling silently tank tok/s — watch memory pressure and clocks.
  • The startup log reports offload, context, KV, and template — read it before changing anything.
  • Pin model+quant+template in production to prevent behavioral drift (Phase 5.10).

9. Observations from Real Systems

  • The most common forum issue is "slow on my GPU" → answer is almost always -ngl/GPU-build (03).
  • "It worked, then crashed" reports are usually KV growth under long sessions/concurrency — the memory model predicts them.
  • "The model is dumb in llama.cpp but fine elsewhere" is typically a chat-template mismatch in a custom GGUF convert.
  • Ollama num_ctx truncation (04) shows up as "it ignores the start of my document" — a template/context bug, not a reasoning failure.
  • Mac slowdowns + fan noise are memory pressure (swap) or thermal throttling (05).

10. Common Misconceptions

MisconceptionReality
"Random OOM after a while"Deterministic KV growth — cap ctx/concurrency
"My GPU is too weak" (slow)Usually CPU offload (-ngl missing), not the GPU
"The model is broken" (garbage)Usually wrong template/EOS/tokenizer
"More quant always = a bit worse"Below ~4-bit it can break structure entirely
"It fit in testing, so it's fine"Concurrency/long context re-tests memory
"Slow = bad decode"Could be prefill (TTFT) on a huge prompt

11. Engineering Decision Framework

DIAGNOSE a misbehaving local model:
 0. READ THE STARTUP LOG (offload N/M, context, KV, template).
 1. WON'T LOAD?
      weights_GB > memory → quant↓ / smaller / bigger device           [02,06]
      format/arch mismatch → convert / update engine                   [03]
      corrupt → re-download (check size/hash)
 2. OOM LATER?
      KV grew (long ctx / concurrency) → cap -c/num_ctx, cap parallel,
                                          KV-quant, weight quant↓        [02]
      no headroom → leave ≥20%
 3. SLOW?
      on CPU? → -ngl 99 + GPU build                                     [03]
      partial offload? → fit more layers (quant↓) / accept              [01]
      swapping? → smaller model/quant (watch memory pressure)           [05]
      throttling? → cooling; expect lower sustained
      huge prompt? → it's prefill/TTFT, not decode                      [2.07]
 4. GARBAGE (fluent)?
      wrong template/EOS → use the model's correct template             [03]
      tokenizer mismatch → match tokenizer
      over-quantized → higher bits / better method                      [06]
      temperature too high → lower it                                   [1.03]
 5. Re-measure; change ONE variable at a time; diff vs known-good.

12. Hands-On Lab

Goal

Deliberately reproduce and fix one bug from each of the four buckets, building a personal runbook.

Prerequisites

  • A local engine (03/04) and one small model.

Setup

# Have nvidia-smi (NVIDIA) or Activity Monitor (Mac) ready to watch memory.

Steps

  1. Slowness (bucket C): run llama.cpp with -ngl 0 (force CPU) and then -ngl 99; record tok/s for both. Confirm the GPU build via the startup log ("offloaded N/N").
  2. OOM-later (bucket B): set a large --parallel/num_ctx and a long input; watch memory climb in nvidia-smi/Activity Monitor until it OOMs or nears the limit. Then cap context/parallelism and/or enable KV quant; show it's stable. Tie the numbers to your memory calculator.
  3. Garbage (bucket D): run a base (non-instruct) model with a chat prompt, or disable the chat template, and observe rambling/no-stop output; then apply the correct template and show coherent output. Separately, push temperature to 2.0 and back to 0.2.
  4. Won't load / quant cliff (buckets A+D): load a 2-bit quant of a small model and eval a reasoning/JSON item; show degradation; reload at Q4_K_M and show recovery (06).
  5. Write the runbook: for each, record symptom → command that confirmed it → fix.

Expected output

Four reproduced-and-fixed cases with before/after evidence (tok/s, memory, output samples) and the confirming command for each.

Debugging tips

  • Always change one variable at a time.
  • If unsure which bucket, the startup log + a memory watch usually disambiguates in seconds.

Extension task

Add a garbage detector to your smoke test: assert the output stops (hits EOS) and passes a tiny correctness check, so template/quant regressions fail CI.

Production extension

Encode the caps (context, concurrency) and a quality smoke test into your serving config so the four buckets can't reach production silently (Phase 7).

What to measure

tok/s (CPU vs GPU), memory trajectory to OOM, output coherence before/after template fix, quality at 2-bit vs 4-bit.

Deliverables

  • A four-bucket runbook (symptom → confirm command → fix).
  • Before/after evidence for each reproduced bug.
  • A smoke test asserting output quality (stops + correctness), wired toward CI.

13. Verification Questions

Basic

  1. What are the four buckets every local bug falls into?
  2. Why does a model OOM after running fine for a while?
  3. What's the #1 cause of slow local inference on a GPU machine?

Applied 4. A custom GGUF gives fluent but nonsensical answers. Walk through your diagnosis. 5. tok/s is fine at first then drops on a laptop during a long run. Two causes.

Debugging 6. The startup log says "offloaded 12/33 layers." What does that imply for speed, and the fix? 7. Output never stops generating. Most likely cause and fix.

System design 8. Design production caps + smoke tests that prevent all four buckets from reaching users.

Startup / product 9. A customer reports your self-hosted model "got dumber overnight." How do you investigate (think quant/template drift)?

14. Takeaways

  1. Classify the symptom into one of four pillars: memory, hardware, template/quant, flags/sampling.
  2. OOM-later = KV growth — cap context and concurrency; size beforehand.
  3. Slow on a GPU ≈ missing -ngl/CPU offload; or swap/throttle/prefill.
  4. Fluent garbage ≈ wrong template/EOS/tokenizer or over-quantization — not a bad model.
  5. Read the startup log first, change one variable at a time, and pin model+quant+template in production.

15. Artifact Checklist

  • A four-bucket diagnostic runbook (symptom → confirm → fix).
  • Before/after evidence for one bug per bucket.
  • A smoke test asserting output stops + basic correctness.
  • Production caps (context, concurrency) documented.
  • A note on pinning model+quant+template to prevent drift.

Up: Phase 6 Index · Next: Phase 7 — Production Serving