Lab 02: Local Inference with Ollama

Goal

Run open-weight models locally. Understand GGUF quantization, hardware requirements, and performance tradeoffs. Compare local vs. cloud quality and speed.

Prerequisites

16GB+ RAM (for 7B models)
macOS, Linux, or Windows with WSL2
Ollama installed: https://ollama.com

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model (downloads ~4.5GB for Q4_K_M)
ollama pull qwen2.5:7b

# Or smaller for low RAM
ollama pull qwen2.5:3b  # ~2GB

Exercise 1: Basic Inference via Ollama CLI

# Chat via CLI
ollama run qwen2.5:7b

# One-shot query
echo "What is the KV cache?" | ollama run qwen2.5:7b

# Check what models you have
ollama list

# Check model info
ollama show qwen2.5:7b

Task: Run 5 different queries. Note which types of questions it handles well vs. poorly vs. confidently wrong.

Exercise 2: Ollama OpenAI-Compatible API

Ollama exposes an OpenAI-compatible API at http://localhost:11434.

# lab02/02_ollama_openai.py
from openai import OpenAI

# Point to local Ollama server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required but ignored by Ollama
)

response = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain self-attention in 3 sentences."}
    ],
    max_tokens=200,
    temperature=0
)

print(response.choices[0].message.content)

Key insight: The exact same OpenAI SDK code works for local models. This is why OpenAI-compatible APIs matter.

Exercise 3: Benchmark Tokens Per Second

# lab02/03_benchmark.py
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

def benchmark(model: str, prompt: str, max_tokens: int = 200) -> dict:
    start = time.time()
    first_token_time = None
    tokens = 0
    
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True
    )
    
    output = ""
    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.time()
            output += chunk.choices[0].delta.content
            tokens += 1
    
    end = time.time()
    
    return {
        "model": model,
        "ttft_ms": (first_token_time - start) * 1000 if first_token_time else 0,
        "total_s": end - start,
        "tokens": tokens,
        "tps": tokens / (end - first_token_time) if first_token_time else 0,
        "output_preview": output[:100]
    }

# Test with a prompt that requires substantial generation
prompt = "Write a detailed explanation of how the transformer attention mechanism works, covering the mathematical operations step by step."

models_to_test = ["qwen2.5:7b"]  # Add more if you have them pulled

for model in models_to_test:
    result = benchmark(model, prompt)
    print(f"\nModel: {result['model']}")
    print(f"  TTFT: {result['ttft_ms']:.0f}ms")
    print(f"  Total: {result['total_s']:.1f}s")
    print(f"  TPS: {result['tps']:.1f} tokens/sec")
    print(f"  Output: {result['output_preview']}...")

Task: Run on at least one model. Record the TPS. Is it faster or slower than you expected?

Exercise 4: Compare Quantization Quality

# Pull the same model at different quantizations
ollama pull qwen2.5:7b-q8_0   # Higher quality
ollama pull qwen2.5:7b         # Default Q4_K_M

# lab02/04_quantization_compare.py
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

test_questions = [
    "What is 17 × 23?",
    "Write a Python function to check if a number is prime.",
    "Translate 'Hello, how are you?' to French, Spanish, and German.",
]

models = ["qwen2.5:7b", "qwen2.5:7b-q8_0"]  # Adjust to what you have

for question in test_questions:
    print(f"\nQ: {question}")
    print("-" * 60)
    
    for model in models:
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": question}],
                max_tokens=200,
                temperature=0
            )
            print(f"\n[{model}]")
            print(response.choices[0].message.content)
        except Exception as e:
            print(f"[{model}] Error: {e}")

Exercise 5: Custom Modelfile (Custom System Prompt)

Ollama lets you create a custom model variant with a built-in system prompt using a Modelfile.

# Create Modelfile
cat > Modelfile << 'EOF'
FROM qwen2.5:7b

SYSTEM """You are a senior software engineer specializing in Python and distributed systems.
You answer questions concisely with code examples.
You always mention potential pitfalls and edge cases.
You never give vague or generic advice."""

PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF

# Build the custom model
ollama create my-coding-assistant -f Modelfile

# Test it
ollama run my-coding-assistant "How do I handle concurrent dict access in Python?"

Task: Create a custom model for a domain you're interested in (e.g., LLM engineering assistant, SQL expert, system design advisor).

Exercise 6: Ollama API Direct (No OpenAI SDK)

# lab02/06_direct_api.py
import httpx, json

def ollama_chat(model: str, messages: list[dict], stream: bool = False) -> str:
    """Direct Ollama API call without OpenAI SDK."""
    response = httpx.post(
        "http://localhost:11434/api/chat",
        json={
            "model": model,
            "messages": messages,
            "stream": stream
        },
        timeout=120
    )
    
    if not stream:
        data = response.json()
        return data["message"]["content"]
    else:
        # Handle streaming
        output = ""
        for line in response.iter_lines():
            if line:
                chunk = json.loads(line)
                if chunk.get("message", {}).get("content"):
                    output += chunk["message"]["content"]
                    print(chunk["message"]["content"], end="", flush=True)
                if chunk.get("done"):
                    print()
                    break
        return output

# Test
result = ollama_chat(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "What is a transformer?"}]
)
print(result)

# Check available models
models = httpx.get("http://localhost:11434/api/tags").json()
print("\nAvailable models:")
for m in models["models"]:
    print(f"  {m['name']} ({m['size'] / 1e9:.1f}GB)")

Deliverables

Ollama installed and at least one model pulled
Exercise 2: OpenAI SDK with local model working
Exercise 3: TPS benchmark recorded — note your hardware (RAM, CPU/GPU)
Exercise 5: Custom Modelfile created and tested
Comparison notes: which types of questions does the local model handle well?

Reflection Questions

Your local 7B model gives a wrong answer that GPT-4o gets right. What are the options?
A team member has 8GB RAM. Which model and quantization would you recommend?
When would you use local inference in production vs. cloud APIs?
Why is TTFT higher for local models than cloud APIs even when TPS is similar?
A startup wants to avoid sending customer data to OpenAI. Design a solution.

LLM Inside-Out Mastery