GGUF and llama.cpp

Phase 6 · Document 03 · Local Inference Prev: 02 — RAM, VRAM, Unified Memory · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

GGUF is the file format and llama.cpp is the engine that, together, run more local LLMs than anything else on Earth — they're the substrate inside Ollama, LM Studio, and a long tail of desktop apps. If you can read a GGUF filename, pick the right quant variant, build llama.cpp for your hardware, and stand up llama-server with an OpenAI-compatible endpoint, you can run essentially any open-weight model on a laptop, a Raspberry Pi, a Mac, or a GPU box — and wire it into the same code that talks to OpenAI. This is the most portable, most widely supported path in local inference, and the foundation the next two docs (Ollama/LM Studio, MLX) build on.

2. Core Concept

Plain-English primer: what GGUF actually is

A trained model on Hugging Face is usually a folder: safetensors weight shards + a config.json + a tokenizer + chat-template files. To run it you need all of them, wired together correctly. GGUF (GPT-Generated Unified Format) collapses that into one self-contained binary file that holds:

• the weights (usually quantized — compressed to ~4–8 bits/param, 06),
• the architecture metadata (layers, heads, dims — the numbers you plugged into the memory formula),
• the tokenizer (vocab + merges), and
• the chat template (how to format roles into the prompt — what-happens §0).

So "download one .gguf and run it" works because everything the engine needs is inside. Contrast safetensors — the HF/GPU-server format (a safe, mmap-able tensor container with no executable code, unlike old pickled .bin checkpoints) — which vLLM and Transformers load, typically at full/half precision, for GPU serving (Phase 7).

Reading a GGUF filename (the quant variants)

GGUF models ship in many quantization variants; the filename encodes them. Example: Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf.

The letters: K = "k-quant" (smarter block-wise quantization that keeps more bits where they matter); the trailing S/M/L = small/medium/large (how aggressively the less-important tensors are squeezed). Q4_K_M is the community default — near-lossless on most tasks at ~¼ the size. (Mechanics and how k-quants/imatrix work: 06.)

llama.cpp: the engine

llama.cpp is a portable C/C++ implementation of transformer inference. It compiles to fast CPU code (AVX/AVX-512 on x86, NEON on ARM) and offloads to GPUs via CUDA (NVIDIA), Metal (Apple), ROCm/HIP (AMD), or Vulkan. Two binaries you'll use constantly:

• llama-cli — one-shot/interactive generation from the terminal.
• llama-server — an HTTP server exposing an OpenAI-compatible /v1/chat/completions endpoint (plus a native /completion API and a web UI), so any OpenAI SDK can talk to it.

The flags that matter

-m   model.gguf      # the GGUF file
-ngl N               # number of layers to OFFLOAD TO GPU (N=99 → all; 0 → CPU only) ← biggest speed knob
-c   8192            # context size (sets KV cache memory, [02])
-b / -ub             # batch / micro-batch size (prefill throughput)
--parallel K         # K concurrent slots (each needs its own KV → [02])
-fa                  # FlashAttention (faster, less KV memory) where supported
--cache-type-k/-v    # KV cache quantization (e.g. q8_0) to save memory [02]
-ts 3,1              # tensor split across multiple GPUs
-t   N               # CPU threads (for CPU/partial offload)

-ngl is the one beginners forget: without it, llama.cpp runs on CPU even on a GPU machine (the classic "why is it slow" bug).

3. Mental Model

HF model folder (safetensors + config + tokenizer + template)
        │  convert + quantize  (convert_hf_to_gguf.py → llama-quantize)
        ▼
   ┌──────────── ONE .gguf FILE ────────────┐
   │ quantized weights · arch · tokenizer ·  │   ← self-contained: "download & run"
   │ chat template                           │
   └─────────────────────────────────────────┘
        │  load
        ▼
   llama.cpp engine  ──(-ngl → GPU via CUDA/Metal/ROCm)──►  tokens/sec [01]
        │
   ┌────┴─────────────┐
   llama-cli        llama-server  ──►  OpenAI-compatible /v1/chat/completions
   (terminal)       (HTTP + web UI)     ↳ any OpenAI SDK just works

Mnemonic: GGUF = the file (quant variant in the name); llama.cpp = the engine (-ngl = put it on the GPU).

4. Hitchhiker's Guide

What to look for first: the quant variant in the filename (start at Q4_K_M) and whether your build has GPU support. Then set -ngl to offload layers.

What to ignore at first: exotic quants (IQ-series, Q2), Vulkan vs CUDA micro-choices, and hand-tuning batch sizes. Defaults are good.

What misleads beginners:

• Forgetting -ngl → silent CPU inference at 1/20th the speed (08).
• Picking the biggest quant "for quality" and OOMing — Q8 of a 13B may not fit a 24 GB GPU once KV is added (02).
• Wrong chat template → coherent-looking but degraded output; GGUF usually embeds the template, but mismatches happen with custom converts (08).
• Assuming safetensors == GGUF — you must convert (and quantize) to GGUF first.

How experts reason: they choose the highest k-quant that fits with headroom (02), max out -ngl, enable -fa, set -c to the real context they need (not the max), and benchmark tok/s against the bandwidth ceiling. For concurrency they reach for vLLM instead (Phase 7).

What matters in production: llama.cpp/llama-server is excellent for single or low-concurrency serving, edge, and embedding into apps. For many concurrent users, its batching is weaker than vLLM's continuous batching — know the boundary.

How to debug/verify: check the startup log for "offloaded N/M layers to GPU" and the per-token timing llama.cpp prints; verify the endpoint with curl. Garbage output → suspect template/quant.

Questions to ask: Which quant variant? Does the build have my GPU backend? What context and concurrency? Is the chat template embedded/correct?

What silently gets expensive/unreliable: CPU offload (a few layers off-GPU tanks tok/s), too-large -c (wastes KV memory), and high --parallel without sizing KV (02).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• GGUF is self-contained (weights + arch + tokenizer + chat template) — one file runs.
• safetensors ≠ GGUF: convert with convert_hf_to_gguf.py, then quantize with llama-quantize.
• Q4_K_M is the default quant — near-lossless on most tasks at ~¼ size; go higher (Q5/Q6/Q8) if memory allows (06).
• -ngl controls GPU offload — omit it and you run on CPU even with a GPU present.
• llama-server exposes an OpenAI-compatible API — existing SDKs/clients work unchanged.
• -c and --parallel set KV memory — size them with the memory formula.
• -fa (FlashAttention) speeds attention and reduces KV memory where supported.
• llama.cpp is the engine inside Ollama and LM Studio — those are convenience layers over it (04).

9. Observations from Real Systems

• Ollama and LM Studio wrap llama.cpp (GGUF) with model management and a GUI/API (04) — learning llama.cpp explains both.
• Unsloth/TheBloke-style HF pages publish GGUFs in every k-quant with size tables — exactly the variant choice this doc teaches.
• llama-server's OpenAI compatibility is why gateways (LiteLLM/OpenRouter-style) can treat a local box as just another provider (Phase 8).
• Edge/embedded deployments (phones, single-board computers) almost always use llama.cpp GGUF because it has no heavyweight runtime dependency.
• Apple users often compare llama.cpp+Metal vs MLX for the same GGUF/model — see 05.

10. Common Misconceptions

11. Engineering Decision Framework

RUN model M with llama.cpp:
 1. Get GGUF: download a published .gguf OR convert_hf_to_gguf.py + llama-quantize.
 2. Pick variant: highest k-quant that FITS with headroom (default Q4_K_M).   [02,06]
 3. Build/install llama.cpp with your GPU backend (CUDA/Metal/ROCm).           [01]
 4. Run: llama-cli (test) → llama-server (serve, OpenAI-compatible).
 5. Flags: -ngl 99, -c <real context>, -fa, --parallel <sized>, KV-quant if tight.
 6. Verify: curl the endpoint; check "offloaded N/N layers"; measure tok/s vs ceiling.
 7. Concurrency needs? → graduate to vLLM.                                     [Phase 7]

12. Hands-On Lab

Goal

Run a GGUF with llama-cli, then serve it via llama-server and call it with the OpenAI SDK and curl, measuring TTFT and tokens/sec.

Prerequisites

• A machine from 01; ~5 GB free for a 7B Q4_K_M.

Setup

# Build (or `brew install llama.cpp`); CUDA example:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j      # macOS: -DGGML_METAL=ON (default)

# Download a GGUF (example)
huggingface-cli download Qwen/Qwen2.5-Coder-7B-Instruct-GGUF \
  qwen2.5-coder-7b-instruct-q4_k_m.gguf --local-dir ./models

Steps

1. One-shot test:

./build/bin/llama-cli -m ./models/qwen2.5-coder-7b-instruct-q4_k_m.gguf \
  -ngl 99 -c 8192 -fa -p "Write a Python function to merge two sorted lists."

Note the startup log: "offloaded N/N layers to GPU" and the tokens/sec line. 2. Serve it:

./build/bin/llama-server -m ./models/qwen2.5-coder-7b-instruct-q4_k_m.gguf \
  -ngl 99 -c 8192 -fa --port 8080

3. Call with curl:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Explain GGUF in 2 sentences."}]}'

4. Call with the OpenAI SDK (base_url="http://localhost:8080/v1", api_key="local") and time TTFT vs total.
5. Vary -ngl (0 vs 99) and the quant variant (Q4_K_M vs Q8_0); record tok/s and memory.

Expected output

Working completions from both curl and the SDK; a table of (-ngl, quant) → tok/s, TTFT, peak memory.

Debugging tips

• "offloaded 0 layers" → build lacks GPU backend or -ngl missing (08).
• Gibberish → wrong/old GGUF or template mismatch; re-download a known-good quant.

Extension task

Convert an HF safetensors model to GGUF (convert_hf_to_gguf.py) and quantize it (llama-quantize ... Q4_K_M) yourself.

Production extension

Put llama-server behind a gateway as an OpenAI-compatible provider and route to it (Phase 8).

What to measure

tok/s and TTFT vs -ngl and quant; peak memory; offloaded-layer count.

Deliverables

• A working llama-server endpoint + a client snippet.
• A quant × offload benchmark table on your hardware.
• A note on the highest quant that fits with headroom.

13. Verification Questions

Basic

1. What four things does a GGUF file contain?
2. What does -ngl do, and what happens if you omit it?
3. What's the difference between GGUF and safetensors?

Applied 4. Decode Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf: bits, approx size, quality tier, when to choose it. 5. You have a 16 GB GPU and want a 14B model at 8k context — which quant and why (02)?

Debugging 6. llama-cli runs at 2 tok/s on a GPU machine. First thing to check? 7. Output is fluent but wrong/garbled. Two likely format causes?

System design 8. Design a low-concurrency internal API on llama-server; specify quant, flags, and the boundary at which you'd switch to vLLM.

Startup / product 9. Why does llama-server's OpenAI compatibility matter for a product that wants to offer both local and cloud models?

14. Takeaways

1. GGUF = one self-contained, quantized file; the quant variant is in the name (default Q4_K_M).
2. safetensors → GGUF requires conversion; safetensors is the GPU/vLLM format.
3. llama.cpp runs everywhere; -ngl offloads to the GPU and is the #1 speed knob.
4. llama-server is OpenAI-compatible — drop-in for existing clients and gateways.
5. Pick the highest k-quant that fits with headroom; size -c/--parallel via the memory formula.
6. Great for low concurrency; switch to vLLM for many users.

15. Artifact Checklist

• A downloaded/converted GGUF at a chosen quant.
• A running llama-server OpenAI-compatible endpoint.
• curl + SDK client snippets that work against it.
• A quant × -ngl benchmark (tok/s, TTFT, memory).
• A note: highest quant that fits + when you'd move to vLLM.

Up: Phase 6 Index · Next: 04 — Ollama and LM Studio