Ollama and LM Studio

Phase 6 · Document 04 · Local Inference Prev: 03 — GGUF and llama.cpp · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

llama.cpp is powerful but raw: you find GGUFs, pick quants, manage files, and remember flags. Ollama and LM Studio wrap that engine with model management, sensible defaults, and a drop-in OpenAI-compatible API so a developer goes from zero to a running local model in two commands. For 90% of development, prototyping, and low-concurrency internal tools, this is the right tool — and because the API is OpenAI-shaped, code you write against it also works against the cloud and against a gateway. Knowing exactly what these tools do (and where they stop — concurrency) lets you move fast locally without painting yourself into a corner.

2. Core Concept

Plain-English primer: what these tools add over llama.cpp

Both Ollama and LM Studio are convenience layers over a GGUF inference engine (Ollama uses its own llama.cpp-based engine; LM Studio bundles llama.cpp and an MLX runtime). They add the things you'd otherwise hand-roll:

• A model registry + downloader. ollama pull llama3.2:3b fetches the right GGUF and remembers it; LM Studio has a search/browse GUI. No hunting Hugging Face for the correct quant file.
• Automatic hardware setup. They detect your GPU/Metal and set offload (-ngl) and threads for you — no flags to forget.
• An OpenAI-compatible server. Both expose /v1/chat/completions, so the OpenAI SDK works by just changing base_url.
• Sane defaults + config. Context length, chat template, stop tokens, and keep-alive are pre-wired per model.

Ollama: the developer default

curl -fsSL https://ollama.com/install.sh | sh     # macOS: brew install ollama
ollama pull qwen2.5-coder:7b                       # download (a quant tag, e.g. :7b-q4_K_M)
ollama run qwen2.5-coder:7b "Refactor this loop."  # interactive / one-shot
ollama list                                         # what's installed
ollama ps                                           # what's loaded in memory now

The server listens on http://localhost:11434, OpenAI-compatible at /v1:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key required, ignored
r = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role":"user","content":"Write a binary search in Rust."}])
print(r.choices[0].message.content)

Model tags encode quant: llama3.1:8b (default quant, usually Q4_K_M), llama3.1:8b-instruct-q8_0 (explicit). A Modelfile customizes a model — base + system prompt + parameters — like a Dockerfile for models:

FROM qwen2.5-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 16384          # context length → sets KV cache memory [02]
SYSTEM "You are a terse senior Rust engineer."

ollama create rusty -f Modelfile && ollama run rusty

Two operational knobs to know: num_ctx (context length — directly drives KV memory, 02) and keep-alive (how long a model stays resident in memory after use; default a few minutes — set OLLAMA_KEEP_ALIVE or the keep_alive request field to avoid reload latency or to free memory sooner).

LM Studio: the GUI

LM Studio is a desktop app: search and download models (with a "will it fit your hardware?" indicator), a chat UI, per-model parameter sliders, and a Local Server tab that starts the same OpenAI-compatible endpoint. On Apple Silicon it can run MLX models too (05). It's ideal for non-CLI users, quick model comparison, and demos.

Where they stop: concurrency

Both are tuned for one or a few users. Under many concurrent requests, their batching is far weaker than vLLM's continuous batching (Phase 7). Ollama can serve a handful of parallel requests (OLLAMA_NUM_PARALLEL), but for production multi-tenant serving you graduate to vLLM/SGLang/TGI.

3. Mental Model

        ┌──────────────── CONVENIENCE LAYER ────────────────┐
        │  Ollama (CLI/daemon)        LM Studio (GUI)         │
        │  • pull/registry            • search + fit check    │
        │  • auto -ngl/threads        • chat UI + sliders     │
        │  • Modelfile (defaults)     • Local Server tab      │
        │  • OpenAI API :11434/v1     • OpenAI API :1234/v1   │
        └───────────────────────┬────────────────────────────┘
                                 │  both sit on…
                       GGUF + llama.cpp engine  (+ MLX in LM Studio)   [03,05]
                                 │
                    one/low concurrency  ──(scale up)──►  vLLM  [Phase 7]

Mnemonic: Ollama/LM Studio = "Docker for local models" → OpenAI API on localhost. Great for dev; vLLM for crowds.

4. Hitchhiker's Guide

What to look for first: the OpenAI-compatible base URL (:11434/v1 Ollama, :1234/v1 LM Studio) and the model tag/quant. With those you can point any existing OpenAI code at a local model.

What to ignore at first: Modelfiles, custom system prompts, and parameter tuning — defaults are fine to start. Add a Modelfile once you need a fixed persona/params.

What misleads beginners:

• Assuming num_ctx is unlimited. Ollama defaults context modestly (e.g., 4k–8k); long inputs get silently truncated unless you raise num_ctx — and raising it costs KV memory (02).
• Confusing the tag's quant. :8b is a quantized default, not FP16 — fine, but know it.
• Treating it as production-scale. It'll feel great at one user and fall over at fifty.
• Forgetting keep-alive. Models unload after idle → the next call eats a reload; or they stay resident and hold memory you wanted back.

How experts reason: they use Ollama/LM Studio as the fast local dev loop and the OpenAI-compatible seam that lets the same client hit local or cloud; they pin num_ctx to real need, set keep-alive deliberately, and switch to vLLM the moment concurrency matters.

What matters in production (if you do ship on it): low-concurrency internal tools only; pin the model version/quant, set num_ctx and OLLAMA_NUM_PARALLEL, monitor memory (KV growth), and put it behind a gateway for auth/limits (Phase 8).

How to debug/verify: ollama ps (loaded models + memory), curl /api/tags, check num_ctx vs your input length; garbage output → template/quant (08).

Questions to ask: What quant does this tag use? What's num_ctx? How many parallel requests can it take before latency degrades? Does it expose the metrics I need?

What silently gets expensive/unreliable: silent context truncation (lost input → worse answers), reload latency from keep-alive, and concurrency cliffs.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Ollama and LM Studio wrap a GGUF/llama.cpp engine (03); LM Studio also runs MLX on Macs (05).
• Both expose OpenAI-compatible APIs (Ollama :11434/v1, LM Studio :1234/v1) — change only base_url.
• Model tags encode quant (:8b ≈ a Q4_K_M default); pick explicit tags for control.
• num_ctx sets context length and thus KV memory — too low silently truncates input (02).
• keep-alive governs reload latency vs memory residency.
• They're tuned for low concurrency — production multi-user → vLLM (Phase 7).
• A Modelfile bakes base model + system prompt + parameters into a named model.
• ollama ps shows loaded models and their memory — your quick fit check.

9. Observations from Real Systems

• Ollama is the most common "first local model" experience and a frequent backend for desktop AI apps and BYOK setups (Phase 11).
• VS Code / IDE BYOK and many local-first tools point at Ollama's :11434/v1 precisely because it's OpenAI-shaped.
• LM Studio's "fit" indicator operationalizes the memory math for non-experts.
• Gateways (LiteLLM/OpenRouter-style) treat an Ollama endpoint as just another OpenAI provider, enabling local↔cloud routing/fallback (Phase 8).
• Teams outgrow Ollama at the concurrency cliff and re-platform onto vLLM — a predictable, documented transition.

10. Common Misconceptions

11. Engineering Decision Framework

PICK a convenience runner:
  Non-CLI / want a GUI / model browsing       → LM Studio
  CLI / scripting / daemon / Modelfiles       → Ollama
  Mac + want MLX speed                         → LM Studio (MLX) or MLX direct [05]
  Edge / embed in an app / no daemon           → llama.cpp directly [03]
  Many concurrent users / SLAs                 → vLLM / SGLang / TGI [Phase 7]

CONFIGURE:
  - choose explicit quant tag (fit + quality)        [02,06]
  - set num_ctx to REAL context need (KV memory)     [02]
  - set keep-alive (latency vs memory)
  - raise OLLAMA_NUM_PARALLEL only within KV budget
  - front with a gateway for auth/limits in prod     [Phase 8]

12. Hands-On Lab

Goal

Run the same model through Ollama's OpenAI-compatible API, customize it with a Modelfile, and demonstrate the num_ctx truncation trap.

Prerequisites

• Ollama installed; ~5 GB free for a 7B.

Setup

curl -fsSL https://ollama.com/install.sh | sh     # macOS: brew install ollama
ollama pull qwen2.5-coder:7b

Steps

1. Call via OpenAI SDK (base_url="http://localhost:11434/v1"); confirm it works and time tokens/sec.
2. Inspect memory: ollama ps while a request runs — note resident size.
3. Make a Modelfile (system prompt + temperature 0.2 + num_ctx 16384), ollama create rusty -f Modelfile, run it.
4. Demonstrate truncation: create a model with num_ctx 2048, send a 5,000-token document, and show the answer ignores the start (lost input). Re-create with num_ctx 16384 and show it now uses it. Tie the memory rise to 02.
5. Concurrency probe: fire 1 vs 8 parallel requests (set OLLAMA_NUM_PARALLEL=8); record per-request latency degradation.

Expected output

Working SDK calls; a custom rusty model; a clear before/after of num_ctx truncation; a latency-vs-concurrency note.

Debugging tips

• 404/connection refused → daemon not running (ollama serve) or wrong port.
• Truncation persists → num_ctx still too small or set on the wrong (uncustomized) model.

Extension task

Repeat in LM Studio: download the model, start the Local Server, and call :1234/v1 with the same client.

Production extension

Put the Ollama endpoint behind a gateway with an API key and a budget, and add a cloud fallback model (Phase 8).

What to measure

tokens/sec, resident memory (ollama ps), truncation behavior vs num_ctx, latency vs concurrency.

Deliverables

• A client snippet hitting the local OpenAI API.
• A Modelfile + custom model.
• A num_ctx truncation demo writeup with the memory implication.

13. Verification Questions

Basic

1. What do Ollama/LM Studio add on top of llama.cpp?
2. What base URLs expose their OpenAI-compatible APIs?
3. What does num_ctx control, and what happens if it's too small?

Applied 4. Write a Modelfile that pins a system prompt, temperature 0.1, and 32k context. What's the memory consequence? 5. You need local + cloud models behind one client. How do these tools make that trivial?

Debugging 6. Long documents give answers that ignore the beginning. Cause and fix. 7. First request after lunch is slow, later ones fast. Why?

System design 8. At what signal do you migrate an internal tool from Ollama to vLLM, and what changes?

Startup / product 9. Why is the OpenAI-compatible seam strategically valuable for a product that may switch between local and cloud models?

14. Takeaways

1. Ollama/LM Studio = "Docker for local models" over GGUF/llama.cpp, with an OpenAI-compatible API.
2. Swap only base_url to point existing OpenAI code at a local model.
3. Model tags encode quant; pick explicitly for fit + quality.
4. num_ctx sets context (and KV memory) — too small truncates silently (02).
5. Great for dev and low concurrency; scale to vLLM for many users.

15. Artifact Checklist

• A client snippet calling the local OpenAI-compatible endpoint.
• A Modelfile + a customized named model.
• A num_ctx truncation demo with the memory note.
• A latency-vs-concurrency measurement.
• A decision note: when you'd move to vLLM.

Up: Phase 6 Index · Next: 05 — MLX and Apple Silicon