Local Model Terms — Formats, Quantization, and Runtimes

Phase 1 · Document 06 · LLM Vocabulary and Mental Models Prev: 05 — Serving Terms · Next: 07 — Evaluation Terms

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Running models locally — on a laptop, a workstation GPU, or an on-prem box — is how you get privacy, offline operation, zero per-token cost, and full control. But the local ecosystem has its own dense vocabulary: GGUF, safetensors, quantization variants (Q4_K_M, AWQ, GPTQ), runtimes (llama.cpp, Ollama, LM Studio, MLX), and the memory math that decides whether a model runs at all. Misread these and you download a 40 GB file that won't load on your 16 GB GPU, pick a quantization that destroys quality, or choose a runtime that can't use your hardware. This vocabulary is the gateway to Phase 6 — Local Inference and Lab 02.

2. Core Concept

Model file formats

The key practical fact: your runtime dictates your format. vLLM wants safetensors; llama.cpp/Ollama want GGUF. The same model is often re-published in several formats.

Quantization (the central local-inference lever)

Quantization stores weights in fewer bits to shrink memory and increase speed, trading some quality. From 02 — Parameters: memory ≈ params × bytes/param.

Quantization method matters as much as bit-width:

• GGUF k-quants (Q4_K_M, Q5_K_M, …): llama.cpp's mixed-precision scheme. The letter suffix (_S/_M/_L) is size/quality; Q4_K_M is the common default.
• GPTQ / AWQ: post-training quantization methods for GPU (safetensors) inference; AWQ often preserves quality well at 4-bit.
• QAT (Quantization-Aware Training): the model is trained with quantization in mind, so the quantized version loses far less quality than naive post-training quant.
• Dynamic / "unsloth dynamic" quants: mixed strategies that keep sensitive layers higher-precision.

Runtimes

Most expose an OpenAI-compatible API, so the same client code talks to local and cloud models — the backbone of provider-agnostic gateways (Phase 8).

Hardware/memory vocabulary

• RAM (system) vs VRAM (GPU) vs unified memory (Apple Silicon shares one pool for CPU+GPU).
• CUDA (NVIDIA), ROCm (AMD), Metal (Apple) — the GPU compute backends a runtime must support.
• Memory headroom — spare memory beyond weights for KV cache + overhead; running near 100% causes OOM under load (the concurrency trap from 05).

3. Mental Model

FORMAT decides RUNTIME:  GGUF → llama.cpp/Ollama/LM Studio · safetensors → vLLM/transformers · MLX → Apple

QUANTIZATION = JPEG compression for model weights
  fewer bits  → smaller file, faster, less memory, lower quality
  4-bit (Q4_K_M / AWQ) = the usual "good enough" sweet spot
  method (QAT, AWQ, k-quant) matters as much as the bit count

CAN IT RUN?
  needed = weights(at chosen bits) + KV cache(context×concurrency) + overhead + headroom
  needed  ≤  VRAM (NVIDIA) | unified memory (Apple) | RAM (CPU)?
  too big → quantize harder, smaller model, shorter context, or more hardware

4. Hitchhiker's Guide

What to look for first: the file format (matches your runtime?), the quantization variant (4-bit is the usual start), and the memory it needs vs what you have.

What to ignore at first: exotic 2–3 bit quants and hand-tuning quant mixes. Start at Q4_K_M (or AWQ 4-bit on GPU) and only go lower if you must.

What misleads beginners:

• Downloading full-precision safetensors for a laptop (won't fit) instead of a GGUF quant.
• Assuming "4-bit" is one thing — method (k-quant vs GPTQ vs AWQ vs QAT) changes quality a lot.
• Forgetting KV cache + overhead and OOM-ing despite "the weights fit."
• Thinking Ollama and vLLM are interchangeable — different formats, different hardware targets.

How experts reason: runtime → required format → largest model that fits at a quant they trust → verify quality on their task, not a generic benchmark. They always leave memory headroom for KV cache.

What matters in production (local): quality at the chosen quant (measure it), memory headroom under concurrency, tokens/sec on the real hardware, and an OpenAI-compatible endpoint so app code is portable.

Debug/verify: run a few of your prompts at FP16 vs 4-bit and compare; watch memory while increasing context; if output degrades or loops, suspect over-aggressive quantization.

Questions to ask: Is there a QAT or AWQ build? What's the recommended GGUF quant? What context length fits in my memory after KV cache? Does my GPU backend (CUDA/ROCm/Metal) have a supported build?

What silently gets expensive/unreliable: too-aggressive quantization quietly tanking reasoning/coding quality; running with no headroom (OOM under load); CPU-only inference being far slower than expected.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Your runtime dictates the format: GGUF (llama.cpp/Ollama/LM Studio) vs safetensors (vLLM/transformers) vs MLX (Apple).
• 4-bit ≈ 0.5 byte/param, roughly 4× smaller than FP16 — the popular local sweet spot.
• Quantization method matters as much as bit-width: QAT and AWQ preserve quality far better than naive quant at the same bits.
• Apple Silicon uses unified memory — CPU and GPU share one pool, so a Mac can run surprisingly large models.
• Weights fitting is not enough — leave headroom for KV cache + overhead or you OOM under load.
• Most local runtimes expose an OpenAI-compatible API, so app code is portable between local and cloud.
• The same model is published in many formats/quants; pick the one matching your runtime and memory.
• CPU inference works but is slow; GPU/Metal acceleration is usually essential for interactive use.

9. Observations from Real Systems

• Ollama pulls GGUF models by name and serves an OpenAI-compatible endpoint, hiding format/quant details behind ollama run — great for getting started.
• llama.cpp's llama-server exposes --ctx-size, sampling params, and an OpenAI-compatible API; quant choice is in the GGUF filename (...Q4_K_M.gguf).
• Hugging Face hosts community GGUF/GPTQ/AWQ re-quantizations of popular models (e.g. "TheBloke"-style and Unsloth dynamic quants) alongside the original safetensors.
• LM Studio gives a GUI to browse, download, and run GGUF quants with a built-in server.
• MLX delivers strong performance on Apple Silicon by using unified memory natively — often the best Mac path.
• vLLM loads safetensors (often AWQ/GPTQ for 4-bit on GPU); it does not run GGUF, a frequent mismatch.

10. Common Misconceptions

11. Engineering Decision Framework

Pick a runtime by hardware + goal:
  Mac (Apple Silicon)        → MLX (best perf) or Ollama (easiest).
  NVIDIA GPU, production      → vLLM (safetensors, AWQ/GPTQ 4-bit).
  Any machine, portability    → llama.cpp/Ollama (GGUF).
  Want a GUI                  → LM Studio.

Pick a quantization:
  Start at 4-bit (Q4_K_M or AWQ).
  Quality not good enough?    → step up (Q5_K_M, Q6, INT8) if memory allows.
  Out of memory?              → step down cautiously; below 4-bit, MEASURE quality.
  QAT or AWQ build available? → prefer it (better quality per bit).

Will it run? (use the Phase 6 calculator)
  need = params×bytes(quant) + KV(context×concurrency) + overhead + headroom
  need ≤ available memory? ship it; else shrink model/context/quant or add hardware.

12. Hands-On Lab

Goal

Run a small model locally via Ollama or llama.cpp, hit its OpenAI-compatible endpoint, and compare a 4-bit quant to a higher-precision build on a few of your prompts.

Prerequisites

• A machine with ≥8 GB RAM (16 GB+ recommended); internet to pull a model.

Setup (Ollama path)

# Install from https://ollama.com, then:
ollama pull qwen2.5:1.5b          # small, fast, good for laptops
ollama pull qwen2.5:1.5b-instruct-q8_0   # higher-precision variant

Steps

# 1. Run interactively
ollama run qwen2.5:1.5b "Explain quantization in 3 sentences."

# 2. Use the OpenAI-compatible API (same code as cloud!)

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
for model in ["qwen2.5:1.5b", "qwen2.5:1.5b-instruct-q8_0"]:
    r = client.chat.completions.create(
        model=model,
        messages=[{"role":"user","content":"Write a Python function to reverse a linked list."}],
        temperature=0, max_tokens=300)
    print(f"\n=== {model} ===\n{r.choices[0].message.content}")

# 3. Observe memory + speed
ollama ps          # shows loaded model + memory

Expected output

• Both models answer; the higher-precision (q8) build is larger in memory and may produce slightly better code.
• A working local OpenAI-compatible endpoint at localhost:11434/v1.

Debugging tips

• "model not found" → run ollama pull first.
• Very slow → no GPU/Metal acceleration; try a smaller model or fewer tokens.
• OOM → reduce model size, quant, or num_ctx.

Extension task

With llama.cpp directly, download a Q4_K_M and a Q8_0 GGUF of the same model, start llama-server, and compare quality + tokens/sec.

Production extension

Point your Phase 1 cost/usage logging code at the local endpoint; note per-token cost is $0 but you now track latency and quality instead.

What to measure

Memory per model (ollama ps), tokens/sec, and a side-by-side quality judgment on 5 of your own prompts at two quant levels.

Deliverables

• A 4-bit vs 8-bit comparison table (size, speed, quality).
• A note: which quant you'd ship on your hardware and why.

13. Verification Questions

Basic

1. Which runtimes use GGUF, and which use safetensors?
2. What does 4-bit quantization do to a 7B model's memory footprint?
3. Why does quantization method matter, not just bit-width?

Applied 4. You have a 16 GB GPU. What model size/quant is a reasonable starting target, and why leave headroom? 5. Why can a Mac with 64 GB unified memory run a model that a 24 GB NVIDIA GPU cannot?

Debugging 6. vLLM won't load the GGUF you downloaded. What's wrong and how do you fix it? 7. Your 4-bit model loops and produces garbage on reasoning tasks. What do you try?

System design 8. Design a private, offline assistant for a laptop fleet. Pick runtime, format, quant, and context, and justify each.

Startup / product 9. When does self-hosting a quantized open-weight model beat a commercial API on unit economics? Name the break-even factors.

14. Takeaways

1. Runtime dictates format: GGUF (llama.cpp/Ollama) vs safetensors (vLLM) vs MLX (Apple).
2. Quantization is the core local lever — start at 4-bit, and the method (QAT/AWQ/k-quant) matters as much as the bits.
3. Apple Silicon's unified memory lets Macs punch above their weight.
4. Fitting weights isn't enough — leave headroom for KV cache and overhead.
5. Most local runtimes are OpenAI-compatible, so app code is portable.
6. Always verify quality at your chosen quant on your own task.

15. Artifact Checklist

• Working local endpoint (Ollama or llama-server) reachable via OpenAI client.
• Benchmark result: 4-bit vs 8-bit size/speed/quality table.
• Code: local-endpoint client reusing your Phase 1 logging.
• Notes: runtime/format/quant decision for your hardware.
• Memory estimate: model + KV + headroom for your machine.
• Cheat card: quant variants and their quality/size trade-offs.

Next: 07 — Evaluation Terms