Cheatsheet 08 — Local Inference (overview + commands)
The local-inference toolbox and when to use each. Full: Phase 6. Engine-specific: Ollama · llama.cpp · vLLM.
The tools — pick by use case
| Tool | Best for | Format | Platform |
|---|---|---|---|
| Ollama | Easiest local dev, quick start | GGUF (managed) | Mac/Linux/Win |
| llama.cpp | Max control, CPU/Mac/edge, custom builds | GGUF | Everywhere |
| LM Studio | GUI, non-CLI users | GGUF | Mac/Win/Linux |
| MLX | Apple Silicon native (unified memory) | MLX/safetensors | Mac (M-series) |
| vLLM | GPU production serving, throughput | safetensors/AWQ/GPTQ | NVIDIA GPU (cheatsheet 09) |
Memory & speed first principles
- Weights memory ≈
params × bytes/param(FP16=2, Q4≈0.5; cheatsheet 06). - Total = weights + KV-cache (grows with context × batch) + activations.
- Speed law:
tok/s ≈ memory bandwidth ÷ model bytes— decode is bandwidth-bound. - Unified memory (Apple Silicon) = RAM shared with GPU; big models fit but bandwidth caps speed.
Quick start (Ollama — fastest path)
# install: https://ollama.com
ollama pull llama3.1:8b # download a model
ollama run llama3.1:8b # interactive chat
ollama list # list local models
ollama ps # running models + memory
# OpenAI-compatible server on :11434
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"hi"}]}'
Picking a quant locally
- Default Q4_K_M; bump to Q5_K_M/Q6_K if you have VRAM and want quality; Q8_0 for near-lossless (cheatsheet 06).
Decision: which engine?
Quick dev / chat / OpenAI-compatible local API → Ollama
Max control / CPU / edge / custom → llama.cpp
GUI → LM Studio
Apple Silicon, best perf → MLX (or Ollama)
GPU production throughput → vLLM [09]
Next: 09 — vLLM Commands · Full: Phase 6