Local Inference — Overview and the Local Stack

Phase 6 · Document 00 · Local Inference Prev: Phase 5.10 — Provider Variance · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Local inference means running a model's forward pass (what-happens §1.C) on hardware you control — your laptop, a workstation GPU, an on-prem server — instead of calling a cloud API. It is the single biggest lever an LLM engineer has over privacy, cost at scale, offline capability, and control over the exact weights being served.

Why a senior engineer must own this, even if you mostly use APIs:

• Data governance. Some data legally cannot leave your environment (health, finance, defense, EU residency). Local is sometimes the only option (Phase 5.02).
• Unit economics at scale. Above some volume, owned hardware beats per-token API pricing — but only if you can estimate memory and throughput correctly (this phase) and keep GPUs busy (Phase 7).
• Reading model pages. Hugging Face / Unsloth / GGUF pages assume you understand quantization, VRAM, GGUF variants, and tokens/sec. Without this phase you can't tell whether a model fits your box (Phase 3.04, 3.05).
• Serving fidelity. Running it yourself is the ground truth against which you detect provider quantization/context-capping (Phase 5.10).

This document is the map of Phase 6: the stack, the one decision that governs everything (does it fit and is it fast enough?), and where each piece is covered in depth.

2. Core Concept

Plain-English primer (from zero)

A cloud API hides four things you must now provide yourself. To run a model locally you assemble a stack of exactly four layers:

1. Weights in a file format. The model's learned numbers (Phase 1.02), serialized to disk. Two formats dominate: GGUF (one self-contained file, the standard for CPU/Mac/consumer-GPU via llama.cpp — 03) and safetensors (the Hugging Face/GPU-server format, used by vLLM — Phase 7.01). The weights are usually quantized (compressed to fewer bits per number) so they fit — 06.
2. An inference engine. The program that loads the weights into memory and actually runs the forward pass: llama.cpp (03), Ollama / LM Studio (04), MLX on Apple Silicon (05), or vLLM/SGLang/TGI for GPU serving (Phase 7).
3. Memory to hold it. The weights plus the KV cache plus overhead must fit in RAM (CPU), VRAM (GPU), or unified memory (Apple Silicon) — 01, 02. This is the #1 constraint.
4. Compute to run it fast enough. A CPU, GPU, or NPU with enough memory bandwidth (decode is bandwidth-bound — what-happens §1.D) to give acceptable tokens/second — 01.

The one decision that governs everything

Every local-inference question reduces to two checks:

(1) DOES IT FIT?      weights + KV cache + overhead  ≤  available memory − headroom
(2) IS IT FAST ENOUGH?  achieved tokens/sec & TTFT  ≥  your task's requirement

If (1) fails, you quantize harder or pick a smaller model. If (2) fails, you get a faster backend / more bandwidth, enable speculative decoding/MTP (07), or accept it. Everything in this phase is machinery for answering these two questions precisely instead of guessing.

The technical layers, in one picture

The forward pass is identical to the cloud (Phase 2); only who runs it and on what changes. The skill is mapping a model's parameter count + precision to a memory footprint and a throughput, then matching that to hardware.

3. Mental Model

            ┌─────────────────────────────────────────────────────┐
            │            THE LOCAL INFERENCE STACK                 │
            ├─────────────────────────────────────────────────────┤
  4. COMPUTE │  CPU / GPU / Apple Silicon / NPU   (bandwidth = speed)│ → tokens/sec  [01]
  3. MEMORY  │  RAM / VRAM / Unified   (must fit weights+KV+OH)      │ → does it fit? [01,02]
  2. ENGINE  │  llama.cpp · Ollama · LM Studio · MLX · vLLM          │ → loads & runs [03,04,05]
  1. WEIGHTS │  GGUF / safetensors, quantized (Q4_K_M, AWQ, …)       │ → size on disk [03,06]
            └─────────────────────────────────────────────────────┘
                       ▲ two questions decide the whole phase:
              DOES IT FIT?  ────  IS IT FAST ENOUGH?
              fix: quantize/smaller   fix: faster backend / spec-decode / accept

Remember it as: format → engine → memory → compute, gated by fit and speed.

4. Hitchhiker's Guide

What to look for first: the model's parameter count and the quantization you'll use → compute the memory footprint (02) → compare to your available memory minus headroom. That single calculation tells you whether the rest is even possible.

What to ignore at first: exotic backends, multi-GPU, tensor parallelism, custom kernels. Start with Ollama or llama.cpp on one machine; graduate to vLLM only when you need concurrency (Phase 7).

What misleads beginners:

• "32 GB RAM means I can run a 70B." You can load a 4-bit 70B in ~40 GB, but on CPU RAM (~50–100 GB/s) you'll get 1–4 tok/s. Memory bandwidth, not just capacity, sets speed (01).
• "4-bit = bad." Q4_K_M / good AWQ are near-lossless on most tasks (06).
• "It fits, so I'm done." It fits at batch 1, short context. Add concurrency or long context and the KV cache can dwarf the weights and OOM you (02, 08).

How experts reason: they size memory before downloading; pick the highest quant that fits with headroom; choose the backend by use case (Ollama for dev, llama.cpp for embedding/edge, MLX for Mac, vLLM for multi-user); and always measure tokens/sec, TTFT, and memory rather than trusting a screenshot.

What matters in production: concurrency behavior (KV growth), thermal/throttling, model+template correctness, and a reproducible memory/throughput budget per deployment.

How to debug/verify: measure with the engine's own stats; watch memory live (nvidia-smi, Activity Monitor); if output is garbage, suspect the chat template / quant, not the model (08).

Questions to ask: What's the param count and active params (MoE)? What quant and format? What's my memory and its bandwidth? What context length and concurrency must I support? What tokens/sec does the task need?

What silently gets expensive/unreliable: KV cache under concurrency (OOM), CPU offload (falls off a cliff in speed), and an idle GPU you're paying for at low utilization (Phase 5.02).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The local stack is four layers: weights (format) → engine → memory → compute. Master the mapping between them.
• Memory is the first gate; bandwidth is the speed gate. Capacity decides fit; bandwidth decides tokens/sec (01).
• Total memory ≈ weights + KV cache + runtime overhead — and KV cache scales with context × concurrency (02).
• Quantization is what makes local practical — it shrinks weights 2–8× at small quality cost (06).
• The forward pass is identical to the cloud's (Phase 2); only the host changes.
• Engine choice is by use case: Ollama/LM Studio (dev), llama.cpp (embed/edge), MLX (Mac), vLLM (multi-user prod).
• Always measure tokens/sec, TTFT, and memory on your box — screenshots are environment-specific (Phase 4.03).

9. Observations from Real Systems

• Ollama wraps llama.cpp with model management + an OpenAI-compatible API — the default developer on-ramp (04).
• llama.cpp powers a huge fraction of local inference (it's the engine inside Ollama, LM Studio, and many apps) (03).
• vLLM is the de-facto open serving engine for GPU clusters — PagedAttention + continuous batching (Phase 7).
• Apple Silicon Macs are unusually strong local-LLM machines because unified memory lets a laptop hold a 70B 4-bit model (05).
• Unsloth/HF GGUF pages publish per-quant sizes and tokens/sec claims — exactly the numbers this phase teaches you to verify (07, Phase 3.05).

10. Common Misconceptions

11. Engineering Decision Framework

START: I want to run model M locally.
 1. SIZE IT: footprint = weights(params,quant) + KV(ctx,concurrency) + overhead   [02]
 2. FIT?  footprint ≤ available_memory × 0.8 ?
      NO → quantize harder ([06]) or smaller model → recompute. Still no → cloud/GPU upgrade.
      YES ↓
 3. PICK ENGINE by use case:
      dev/personal           → Ollama / LM Studio   [04]
      Mac, max speed         → MLX (or llama.cpp+Metal) [05]
      embed / edge / CPU     → llama.cpp (GGUF)      [03]
      many concurrent users  → vLLM / SGLang / TGI   [Phase 7]
 4. RUN + MEASURE: tokens/sec, TTFT, memory.        [12, 08]
 5. FAST ENOUGH?
      NO → faster backend / more bandwidth / speculative decoding+MTP [07] / accept.
      YES ↓
 6. HARDEN: set context limit, concurrency cap, headroom; verify template/quant. [08]

12. Hands-On Lab

Goal

Stand up the local stack end-to-end on your machine, then size and measure a model so you can answer "does it fit?" and "is it fast enough?" with numbers.

Prerequisites

• A machine with ≥8 GB RAM (more is better); admin rights to install.

Setup

# Easiest on-ramp (wraps llama.cpp):
curl -fsSL https://ollama.com/install.sh | sh    # macOS: brew install ollama

Steps

1. Size before you pull. For a 3B model at 4-bit: weights ≈ 3e9 × 0.5 bytes ≈ 1.5 GB; add ~1 GB overhead + KV. Confirm it's well under RAM × 0.8. (Full math in 02.)
2. Pull + run:

ollama pull llama3.2:3b
ollama run llama3.2:3b "Explain tokenization in 3 sentences."

3. Measure via the OpenAI-compatible API:

import time
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
t = time.time()
r = client.chat.completions.create(model="llama3.2:3b",
        messages=[{"role":"user","content":"Explain the attention mechanism."}])
dt = time.time() - t
n = r.usage.completion_tokens
print(f"{n} tok in {dt:.2f}s → {n/dt:.1f} tok/s")

4. Watch memory during generation (Activity Monitor / nvidia-smi). Note peak.
5. Stress fit: pull a 7B and an 8B; record which fit and their tokens/sec.

Expected output

A small table: model size → fits? → tokens/sec → peak memory, on your hardware.

Debugging tips

• No GPU used / very slow → wrong backend or offload; see 08.
• OOM on a model that "should fit" → KV cache / no headroom (02).

Extension task

Repeat one model under llama.cpp directly (03) and compare tokens/sec to Ollama.

Production extension

Re-run the largest model under vLLM with 10 concurrent requests and observe how KV cache and throughput scale (Phase 7).

What to measure

Fit (yes/no), tokens/sec, TTFT, peak memory — per model/quant.

Deliverables

• A hardware profile (CPU, RAM + bandwidth, GPU/unified memory).
• A fit-and-speed table across 2–3 model sizes.
• A one-paragraph recommendation for which local model your hardware can actually serve.

13. Verification Questions

Basic

1. Name the four layers of the local inference stack.
2. What two questions does every local-inference decision reduce to?
3. What three things sum to total memory required?

Applied 4. You have a 24 GB GPU. Can you run a 70B at 4-bit? At batch 1? Under concurrency? Justify with the memory model. 5. Choose an engine for: (a) a Mac dev laptop, (b) edge device, (c) 200-user internal tool. Justify.

Debugging 6. A model loads but generates at 2 tok/s. Two likely causes? 7. A model that "fit" OOMs after a long chat. Why?

System design 8. Design the decision flow from "I want model M" to a sized, measured, hardened local deployment.

Startup / product 9. At what point does self-hosting beat an API for your product, and which Phase 6 numbers drive that break-even?

14. Takeaways

1. The local stack is weights (format) → engine → memory → compute.
2. Every decision reduces to "does it fit?" (capacity) and "is it fast enough?" (bandwidth).
3. Quantization is what makes local practical; measure, don't trust screenshots.
4. KV cache under context/concurrency is the silent OOM — size for it.
5. Pick the engine by use case; graduate to vLLM only for real concurrency.

15. Artifact Checklist

• Hardware profile (CPU, RAM+bandwidth, GPU/unified memory).
• Memory-fit calculation for one target model.
• Fit-and-speed table across 2–3 models/quants.
• Engine choice with justification for your use case.
• Recommendation memo: the best local model your hardware can serve.

Up: Phase 6 Index · Next: 01 — Hardware Literacy