Cheatsheet 10 — llama.cpp Commands

Max-control local inference (CPU/Mac/edge), GGUF. Full: Phase 6.03.

Build & get a model

# build (Metal on Mac / CUDA on NVIDIA auto-detected by cmake)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release
# get a GGUF (e.g., from Hugging Face) — pick a quant like Q4_K_M [cheatsheet 06]
huggingface-cli download <repo> <file>.gguf --local-dir models/

Run

# interactive chat
./build/bin/llama-cli -m models/model-Q4_K_M.gguf -p "Hello" -cnv
# one-shot
./build/bin/llama-cli -m models/model.gguf -p "Summarize: ..." -n 256

OpenAI-compatible server

./build/bin/llama-server -m models/model-Q4_K_M.gguf \
  -c 8192 \        # context size
  -ngl 99 \        # offload N layers to GPU (99 = all)
  --host 0.0.0.0 --port 8080
# then call http://localhost:8080/v1/chat/completions (OpenAI format)

Key flags

FlagWhat
-mModel path (GGUF)
-cContext size (tokens) — more = more KV-cache RAM
-ngl# layers offloaded to GPU (Metal/CUDA); 99 = all, 0 = CPU-only
-nMax tokens to generate
-tCPU threads
--temp, --top-p, --top-kSampling (cheatsheet 07)
-bBatch size (prompt processing)
--mlockKeep model in RAM (avoid swap)

Convert & quantize your own

python convert_hf_to_gguf.py <hf-model-dir> --outfile model-f16.gguf
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Tips

  • -ngl is the perf lever on Mac/GPU — offload as many layers as VRAM allows.
  • Smaller -c saves KV-cache RAM if you don't need long context.
  • This is also the path to ship a fine-tune locally: merge LoRA → convert → quantize → GGUF (Phase 13.07).

Next: 11 — Ollama Commands · Full: Phase 6.03