MLX and Apple Silicon

Phase 6 · Document 05 · Local Inference Prev: 04 — Ollama and LM Studio · Up: Phase 6 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Apple Silicon turned ordinary laptops into surprisingly capable LLM machines, and a huge share of LLM engineers develop on a Mac. The reason is architectural — unified memory (01) — and the Apple-native way to exploit it is MLX, Apple's array framework. Understanding why a $2k MacBook can run a 4-bit 70B that a $1.6k 24 GB GPU cannot, and when MLX beats llama.cpp's Metal backend, lets you make the most of the hardware most engineers already own — for development, private on-device assistants, and shipping Mac apps with embedded models.

2. Core Concept

Plain-English primer: what makes Apple Silicon special

On a typical PC, the CPU has its own RAM and the GPU has separate VRAM; to use the GPU you copy data across the PCIe bus into VRAM, and the model must fit in that (often small, 8–24 GB) VRAM. Apple Silicon (M1–M4) instead uses unified memory: one physical pool of RAM that the CPU and GPU and Neural Engine all address directly, with no copies and no split.

Two consequences matter for LLMs:

1. Capacity: the GPU can use almost all of system memory. A 64 GB Mac can give the GPU ~48+ GB for weights — enough for a 4-bit 70B (~40 GB, 02). A same-priced discrete GPU tops out at 16–24 GB VRAM.
2. Bandwidth (the speed gate): unified memory is fast, but its bandwidth is tiered by chip — roughly M-series base ~100 GB/s, Pro ~200, Max ~400, Ultra ~800 GB/s. Per the speed law tok/s ≈ bandwidth ÷ model_GB, a Max/Ultra is far quicker than a base chip on the same model. So Macs win on fit broadly, but speed depends heavily on which tier you have.

The GPU is programmed via Metal (Apple's GPU API). There is no CUDA — so the CUDA-centric ecosystem (vLLM, FlashAttention, TensorRT-LLM) doesn't run here; you use llama.cpp's Metal backend or MLX instead.

MLX: Apple's array framework

MLX is a NumPy/PyTorch-like array library built by Apple for Apple Silicon. Key design points: a familiar Python (and Swift) API, lazy evaluation + graph compilation, and unified-memory-native arrays (no host/device copies). For LLMs, the mlx-lm package gives you generation, quantization, fine-tuning (LoRA), and an OpenAI-compatible server.

pip install mlx-lm

# One-shot generation (downloads from the mlx-community HF org)
mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --prompt "Explain unified memory in 2 sentences." --max-tokens 200

# OpenAI-compatible server
mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --port 8080

# Programmatic
from mlx_lm import load, generate
model, tok = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
print(generate(model, tok, prompt="Write a haiku about the KV cache.", max_tokens=60))

MLX models use the .safetensors + MLX quantization layout (commonly published under the mlx-community org), not GGUF. MLX quantization is its own scheme (e.g., 4-bit/8-bit group quantization), so you download MLX-converted models or convert with mlx_lm.convert.

MLX vs llama.cpp (Metal) on a Mac

Both exploit unified memory; MLX squeezes more out of the latest Apple GPUs, while llama.cpp wins on portability and the GGUF ecosystem. LM Studio (04) can run both on a Mac, which makes head-to-head comparison easy.

3. Mental Model

   APPLE SILICON = ONE UNIFIED MEMORY POOL
   ┌───────────────────────────────────────────────┐
   │   CPU      GPU      Neural Engine               │  all address the SAME RAM
   │    └────────┴───────────┘  (no copies, no split)│  → GPU can use ~all memory
   └───────────────────────────────────────────────┘
        CAPACITY: huge (fit big models on a laptop)   ← Macs' superpower
        BANDWIDTH: tiered  base<Pro<Max<Ultra         ← sets tok/s [01]
        PROGRAMMED VIA: Metal (NO CUDA)

   Run it with:   MLX (Apple-native, safetensors+MLX-quant)   ← often fastest
              or  llama.cpp + Metal (GGUF, portable)          ← biggest ecosystem

Mnemonic: unified memory = fit; chip tier = speed; MLX or llama.cpp/Metal = how (no CUDA here).

4. Hitchhiker's Guide

What to look for first: your Mac's total unified memory (fit) and chip tier (bandwidth → speed). An M3 Max 64 GB and an M2 base 8 GB are completely different LLM machines.

What to ignore at first: the Neural Engine (ANE) — current LLM engines mostly use the GPU via Metal, not the ANE. Don't pick a model expecting ANE acceleration.

What misleads beginners:

• "Unified memory = unlimited." The OS and apps share the pool; macOS limits how much the GPU may "wire." Leave headroom (02).
• "Any GGUF/quant is the same on Mac." MLX needs MLX-format models (not GGUF); mixing them up wastes time.
• "My Mac is slow at LLMs" — often a base-tier chip (low bandwidth) or running FP16 when a 4-bit MLX/GGUF would fly.
• "vLLM on my Mac" — vLLM is CUDA-first; on Mac you use MLX or llama.cpp.

How experts reason: on a Mac they pick 4-bit/8-bit models sized to leave headroom, prefer MLX for max speed on recent chips (or llama.cpp/Metal for an existing GGUF), and benchmark both since the winner shifts by model and chip generation.

What matters in production (Mac apps / on-device): memory pressure and thermals on laptops, model download size for distribution, and that MLX/Metal are single-node — Macs aren't your multi-tenant server tier (Phase 7 uses CUDA GPUs).

How to debug/verify: Activity Monitor → Memory (and "Memory Pressure"); sudo powermetrics for GPU residency; compare achieved tok/s to your chip's bandwidth ceiling (01).

Questions to ask: How much unified memory, and which tier? Is there an MLX build of this model? Does my engine use the GPU (Metal)? What's my headroom at the context I need?

What silently gets expensive/unreliable: memory pressure → swapping to SSD (huge slowdown); thermal throttling on long runs; assuming Mac speed transfers to a server (it won't — different hardware/stack).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Unified memory lets the GPU use most of system RAM — Macs win on fit (capacity) vs same-priced discrete GPUs.
• Speed is tiered by chip bandwidth (base ≪ Ultra); tok/s ≈ bandwidth ÷ model_GB (01).
• Apple uses Metal, not CUDA — vLLM/FlashAttention/TensorRT-LLM don't run on Mac; use MLX or llama.cpp/Metal.
• MLX uses safetensors + MLX quantization, not GGUF — get MLX-format models (mlx-community) or convert.
• mlx-lm provides an OpenAI-compatible server (mlx_lm.server) — same base_url swap as Ollama (04).
• MLX is often faster than llama.cpp/Metal on recent chips; llama.cpp wins on portability/ecosystem — benchmark both.
• Leave headroom: the OS shares the pool and memory pressure causes SSD swapping (severe slowdown).
• MLX also does LoRA fine-tuning on-device (mlx_lm.lora) — relevant to Phase 13.

9. Observations from Real Systems

• LM Studio runs both MLX and GGUF on Macs (04) — the easiest way to compare them head-to-head.
• mlx-community publishes day-of MLX conversions of popular models in 4/8-bit — the Mac analog of GGUF release pages.
• Mac dev workstations are common for LLM engineers precisely because a laptop can hold 30–70B-class models for offline iteration.
• On-device Mac/iOS apps increasingly ship MLX or llama.cpp models for private inference with no server.
• Servers stay NVIDIA/CUDA (01, Phase 7) — Mac is the dev/edge tier, not the multi-tenant serving tier.

10. Common Misconceptions

11. Engineering Decision Framework

ON APPLE SILICON:
 1. Check fit:  model_GB(quant) + KV + overhead ≤ unified_memory × 0.7   (Mac headroom ↑) [02]
 2. Pick precision: prefer 4-bit/8-bit to fit + go fast.                                   [06]
 3. Choose engine:
      want max speed on recent chip / on-device app  → MLX (mlx-lm)
      have a GGUF / want portability / Ollama DX      → llama.cpp + Metal / Ollama   [03,04]
      want a GUI to compare                           → LM Studio (runs both)        [04]
 4. Serve: mlx_lm.server (OpenAI-compatible) or llama-server.
 5. Measure tok/s vs your chip's bandwidth ceiling; watch Memory Pressure.            [01]
 6. Need multi-tenant scale? → that's a CUDA/vLLM server job, not the Mac.            [Phase 7]

12. Hands-On Lab

Goal

Run a model with MLX, serve it via the OpenAI-compatible mlx_lm.server, and benchmark MLX vs llama.cpp/Metal on the same model class.

Prerequisites

• An Apple Silicon Mac (M1+); ~5–8 GB free unified memory for an 8B 4-bit.

Setup

pip install mlx-lm
# (for the comparison) brew install llama.cpp  OR install Ollama

Steps

1. Generate with MLX:

mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --prompt "Explain prefill vs decode." --max-tokens 200

Record tokens/sec (mlx-lm prints it). 2. Serve with MLX:

mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --port 8080

Call it with the OpenAI SDK (base_url="http://localhost:8080/v1"). 3. Compare the same model class as a GGUF under llama.cpp/Metal or Ollama (03/04); record tok/s and TTFT. 4. Watch memory: Activity Monitor → Memory Pressure during both; note peak and whether pressure stays green. 5. Check the ceiling: look up your chip's bandwidth; compute bandwidth ÷ model_GB and compare to achieved tok/s (01).

Expected output

A small table: engine (MLX vs llama.cpp/Metal) → tok/s, TTFT, peak memory — with the winner and your chip's ceiling noted.

Debugging tips

• MLX can't find the model → it expects an MLX-format repo (mlx-community), not GGUF.
• Severe slowdown / fans → memory pressure (swapping) or thermal throttling; drop quant or model size.

Extension task

Use mlx_lm.convert to quantize an HF model to 4-bit MLX yourself, then run it.

Production extension

Embed an MLX (or llama.cpp) model in a small Mac/Swift app for fully offline, private inference; measure cold-start and memory.

What to measure

tok/s, TTFT, peak unified memory, memory pressure, MLX-vs-llama.cpp delta, ceiling efficiency.

Deliverables

• A working MLX OpenAI-compatible endpoint + client snippet.
• An MLX vs llama.cpp/Metal benchmark table for your chip.
• A note on the largest model your Mac can serve with headroom.

13. Verification Questions

Basic

1. What is unified memory and why does it help LLMs?
2. Why can't you run vLLM on a Mac, and what do you use instead?
3. What format do MLX models use (and not use)?

Applied 4. Estimate the largest 4-bit model an M-series 64 GB Mac can serve with headroom (02). 5. Two Macs both have 32 GB but different chip tiers — which is faster on the same model and why?

Debugging 6. MLX errors that it can't load a GGUF. Why, and the fix? 7. tok/s craters and the fans spin up mid-run. Two likely causes.

System design 8. Design an offline, private Mac assistant; choose engine, precision, context, and headroom.

Startup / product 9. When is shipping an on-device Mac model (MLX/llama.cpp) the right product call vs a cloud API?

14. Takeaways

1. Unified memory lets a Mac fit models a same-priced discrete GPU can't.
2. Speed is set by chip-tier bandwidth — tok/s ≈ bandwidth ÷ model_GB.
3. Apple uses Metal, not CUDA — run with MLX or llama.cpp/Metal, not vLLM.
4. MLX uses safetensors + MLX quant (not GGUF); mlx_lm.server is OpenAI-compatible.
5. MLX is often fastest on recent chips; llama.cpp is most portable — benchmark both, leave headroom.

15. Artifact Checklist

• A working MLX generation + OpenAI-compatible server.
• An MLX vs llama.cpp/Metal benchmark on your chip.
• A memory-pressure / headroom note at your target context.
• The largest model your Mac serves with headroom.
• (Optional) A self-converted 4-bit MLX model via mlx_lm.convert.

Up: Phase 6 Index · Next: 06 — Quantization Guide