Phase 6 — Local Inference
How to run open-weight models on hardware you control — sizing memory, choosing formats and engines (GGUF/llama.cpp, Ollama/LM Studio, MLX, vLLM), quantizing without wrecking quality, speeding up decode with MTP/speculation, and debugging when it breaks.
Why this phase matters
Phase 5 taught you to choose a model; Phase 6 teaches you to run one yourself. Local inference is the lever for privacy, cost-at-scale, offline/edge, and control over the exact weights being served — and the ground truth against which you detect provider quantization (Phase 5.10). It's also the prerequisite for production serving (Phase 7): vLLM, PagedAttention, and continuous batching only make sense once you understand weights, KV cache, memory, and bandwidth.
Everything reduces to two questions: does it fit? (memory capacity) and is it fast enough? (memory bandwidth).
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | Local Inference Overview | See the whole stack and the fit/speed decision that governs the phase |
| 01 | Hardware Literacy | Pick hardware via capacity (fit) and bandwidth (tok/s ≈ band ÷ model_GB) |
| 02 | RAM, VRAM, Unified Memory | Size weights + KV cache + overhead; predict OOM before it happens |
| 03 | GGUF and llama.cpp | Read GGUF variants; run llama-server as an OpenAI-compatible endpoint |
| 04 | Ollama and LM Studio | Spin up local models fast with a drop-in OpenAI API; avoid the num_ctx trap |
| 05 | MLX and Apple Silicon | Exploit unified memory; run MLX vs llama.cpp/Metal on a Mac |
| 06 | Quantization Guide | Choose bit-width + method (k-quant/AWQ/GPTQ/QAT) without breaking quality |
| 07 | MTP and Speculative Decoding | Decode the "2× faster" claim; tune draft/verifier and acceptance |
| 08 | Local Model Debugging | Map symptoms (won't load / OOM / slow / garbage) to fixes fast |
How to work through it
Read 00 (the map) and 02 (the memory math) first — they're the backbone. 01 gives the hardware reasoning behind speed. Then pick the engine docs you need: 03 (the universal GGUF/llama.cpp path), 04 (developer convenience), 05 (Mac). 06 (quantization) underpins all of them. 07 (MTP/speculation) is the optimization layer, and 08 (debugging) is the practical capstone — keep it next to you while you run the labs. Every doc opens with a from-zero plain-English primer and ends with a measured, artifact-producing lab.
Phase 6 artifacts
- A hardware profile (capacity + bandwidth) and a ceiling-vs-actual tokens/sec table (00, 01).
- A reusable memory calculator + fit heatmap over context × concurrency (02).
- A working
llama-server+ Ollama/LM Studio + MLX OpenAI-compatible endpoint, each benchmarked (03, 04, 05). - A quantization trade-off report (size/speed/quality, per-category) with a recommended quant (06).
- A speculation speedup report by content type and batch, with acceptance rate (07).
- A four-bucket debugging runbook + a quality smoke test (08).