Local vs Cloud Deployment

Phase 5 · Document 02 · Model Selection Prev: 01 — Open-Source vs Commercial · Next: 03 — Coding Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Once you've decided to use an open-weight model (01), a second question follows: where does it run? On a user's device/your own box (local), on GPUs you operate in the cloud (self-hosted cloud), or on someone else's managed endpoint? This decision drives privacy, latency, offline capability, and a very different cost shape (fixed hardware vs per-hour GPU vs per-token). Misjudging it gives you a laptop assistant that's too slow, a cloud GPU bill for idle hardware, or a "private" deployment that quietly calls an API. This doc maps the deployment spectrum and the hardware reality, linking to the deep hardware/serving phases (6/7).

2. Core Concept

Plain-English primer (the deployment options + hardware words)

• Local (on-device / on-prem) — the model runs on hardware you physically control: a laptop, a workstation GPU, or an on-prem server. Data never leaves; works offline; you're limited by that hardware.
• Self-hosted cloud — you run the model on cloud GPUs (rented VMs/instances). You control the runtime and data path within that cloud; you pay per GPU-hour whether busy or idle.
• Managed inference endpoint — a provider runs the (often open) model for you and exposes an API; you pay per token. No ops, but data goes to them (01's "managed-open").
• VRAM / RAM / unified memory — VRAM is GPU memory (NVIDIA/AMD); RAM is system memory (CPU inference); Apple Silicon shares one unified memory pool. The model's weights + KV cache must fit (Phase 1.06, Phase 2.06).
• CUDA / ROCm / Metal — the GPU compute backends (NVIDIA / AMD / Apple) your runtime needs to support.
• Runtime — the software that runs the model: Ollama/llama.cpp/LM Studio/MLX (local), vLLM/TGI/SGLang/TensorRT-LLM (cloud GPU serving) (Phase 7).

The deployment spectrum

LOCAL (your device) ── SELF-HOSTED CLOUD (your GPUs) ── MANAGED OPEN (their GPUs) ── COMMERCIAL API (their model+GPUs)
 most control/privacy ───────────────────────────────────────────────────────────────► least ops, least control
 fixed HW cost          per-GPU-hour                    per-token                        per-token
 offline-capable        scalable but ops-heavy          no ops                           no ops

The further left, the more control, privacy, and offline capability, but the more hardware/ops you own. The further right, the less ops but more dependence and data leaves your environment.

The cost shapes are fundamentally different

• Local: a fixed, one-time hardware cost; $0 marginal per request; capped by the device.
• Self-hosted cloud: per-GPU-hour, paid whether the GPU is busy or idle → utilization is everything (an idle GPU is pure loss). Cheap per token only at high, steady utilization.
• Managed/API: per-token, scales with usage, zero idle cost.

This is why spiky traffic favors per-token (API/managed) and steady high volume favors owned GPUs — and why self-hosting economics live or die on keeping GPUs busy (Phase 1.05 batching/throughput).

Hardware reality (will it even run?)

From Phase 1.06/2.06: need = weights(at quant) + KV cache(context×concurrency) + overhead + headroom. Local devices have small memory (quantize hard, short context, low concurrency); cloud GPUs have more but you pay per hour; Apple Silicon's unified memory punches above its weight for local. Always size before promising a deployment.

3. Mental Model

WHERE DOES IT RUN?
  LOCAL          your device     → privacy + offline, fixed HW cost, limited by VRAM/unified mem
  SELF-HOST CLOUD your GPUs       → control + scale, $/GPU-hour (idle = waste → UTILIZATION rules)
  MANAGED OPEN   their GPUs       → open model, per-token, no ops, data leaves
  COMMERCIAL API their everything → frontier, per-token, no ops, data leaves

COST SHAPE decides:  spiky/low volume → per-token (API/managed) ; steady/high volume → owned GPUs (if kept busy)
PRIVACY/OFFLINE forces LEFT ; NO-OPS/FAST-TTM pulls RIGHT
ALWAYS size first:  weights + KV + overhead + headroom ≤ available memory (Phase 1.06 / 2.06)

4. Hitchhiker's Guide

What to decide first: is offline/on-device or strict data-locality required? That forces local/on-prem. Otherwise choose by cost shape (spiky vs steady) and ops capability.

What to ignore at first: exotic edge-quantization tricks; get a model running on the target hardware first, optimize later.

What misleads beginners:

• "Self-hosting in the cloud is cheap" — only at high utilization; idle GPUs burn money.
• "It runs on my laptop, so it'll serve users" — single-user local ≠ concurrent serving (KV cache).
• Forgetting KV cache/headroom in the memory budget (OOM under load).
• Calling a deployment "local/private" while it hits a cloud API.

How experts reason: offline/data-locality → local/on-prem; spiky/uncertain volume → managed/API; steady high volume + GPU-capable team → self-hosted cloud tuned for utilization. They size memory first and keep the interface OpenAI-compatible so they can move.

What matters in production: GPU utilization (self-host), memory headroom under concurrency, latency on the actual hardware, and a portable interface.

How to verify: size the memory budget; benchmark tokens/sec + p95 latency on the target hardware (Phase 1.05 lab); compute per-token cost at realistic utilization for the cloud option.

Questions to ask: What memory does it need (weights+KV+headroom)? What utilization can I sustain on cloud GPUs? Does the local device meet latency? Is the data path truly local? Which runtime/backend (CUDA/ROCm/Metal)?

What silently gets expensive/unreliable: idle cloud GPUs; promising local concurrency the hardware can't do; under-budgeting KV; ignoring cold-start/scaling for spiky self-hosted loads.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The cost shape differs: local = fixed HW; self-host cloud = per-GPU-hour (idle = waste); managed/API = per-token.
• Self-hosting is cheap per token only at high, steady utilization — idle GPUs destroy the economics.
• Spiky/low volume favors per-token (managed/API); steady high volume favors owned GPUs.
• Local single-user ≠ concurrent serving — KV cache caps concurrency (Phase 2.06).
• Size memory before promising a deployment: weights + KV + overhead + headroom.
• Apple Silicon unified memory runs surprisingly large models locally.
• Offline/strict data-locality forces local/on-prem regardless of cost.
• Keep the interface OpenAI-compatible so you can move across local/cloud/API.

9. Observations from Real Systems

• Local assistants (Ollama/LM Studio/MLX) deliver private, offline inference — limited by device memory and single-ish-user concurrency (Phase 6).
• vLLM clusters are the self-hosted-cloud standard; their whole design (PagedAttention, continuous batching) exists to maximize utilization/throughput so per-token cost is competitive (Phase 7).
• Managed-open hosts (Together/Fireworks/OpenRouter) are the no-ops middle: open models, per-token, data leaves.
• Idle-GPU bill shock is the classic self-hosting failure — paying for capacity you don't keep busy.
• OpenAI-compatible everywhere (Phase 1.06) lets the same client code hit a local Ollama, a self-hosted vLLM, or a cloud API — portability across the spectrum.

10. Common Misconceptions

11. Engineering Decision Framework

1. HARD requirement?  offline / strict data-locality → LOCAL or ON-PREM.  (else continue)
2. COST SHAPE:
     spiky / uncertain / low volume → MANAGED-OPEN or COMMERCIAL API (per-token, no idle cost).
     steady / high volume AND GPU-capable team → SELF-HOSTED CLOUD (tune utilization).
3. SIZE IT:  weights(at quant) + KV(context×concurrency) + overhead + headroom ≤ target memory? (Phase 1.06/2.06)
     doesn't fit → quantize / smaller model / shorter context / bigger hardware.
4. LATENCY:  benchmark on the ACTUAL hardware (p50/p95) — local devices may be too slow.
5. PORTABILITY:  keep an OpenAI-compatible interface so you can move as needs change.

12. Hands-On Lab

Goal

Decide deployment for one open-weight use case: size the memory, benchmark latency on a target, and compare per-token cost across local / self-hosted-cloud / managed.

Prerequisites

• A chosen open model; a local runtime (Phase 6); cloud GPU + managed-host prices.

Steps

1. Size it: compute weights(at your quant) + KV(context×concurrency) + ~20% headroom; check it fits your local device and your candidate cloud GPU (Phase 2.06).
2. Benchmark local: run the model locally; measure tokens/sec and p95 latency at your real prompt sizes (Phase 1.05 lab).
3. Cost-compare the three deployments:

def per_token_cost_selfhost(gpu_hourly, tokens_per_sec, utilization):
    tokens_per_hour = tokens_per_sec * 3600 * utilization   # only useful tokens count
    return gpu_hourly / tokens_per_hour                      # $ per token

# Example: $2/GPU-hr, 1500 tok/s, 30% vs 80% utilization
for u in (0.3, 0.8):
    c = per_token_cost_selfhost(2.0, 1500, u)
    print(f"util {u:.0%}: ${c*1e6:.2f} / 1M tokens self-hosted")
# managed/API: just read $/1M from the price page; local: ~$0/token + fixed HW

4. Decide: combine hard requirements (offline/locality), fit, latency, and cost-at-realistic-utilization into a deployment recommendation.
5. Memo it (Phase 3.07).

Expected output

• A memory-fit verdict, local latency numbers, and a per-token cost comparison showing how utilization swings self-hosting economics.

Debugging tips

• Self-host cost looks terrible? Check utilization — at 30% it's ~2.7× the 80% cost.
• Local OOM/slow? Quantize harder, shorten context, or move to cloud.

Extension task

Model spiky traffic (e.g. 5% duty cycle): show that low utilization makes self-hosting lose to per-token managed/API.

Production extension

Stand up the chosen deployment behind an OpenAI-compatible endpoint and confirm your app code is unchanged from the API version (portability proof) → feeds the Phase 8 gateway.

What to measure

Memory fit; local tokens/sec + p95; per-token cost at 30% vs 80% utilization; spiky-traffic cost (extension).

Deliverables

• A memory-sizing verdict for local + cloud.
• A 3-way cost comparison (local/self-host/managed) with utilization sensitivity.
• A deployment recommendation/memo.

13. Verification Questions

Basic

1. Name the four points on the local↔cloud↔API spectrum and their cost shapes.
2. Why does GPU utilization dominate self-hosted-cloud economics?
3. Why isn't "runs on my laptop" the same as "can serve users"?

Applied 4. Compute self-hosted $/1M tokens at $3/GPU-hr, 1000 tok/s, 50% utilization. 5. For spiky, low-volume traffic, which deployment and why?

Debugging 6. Your cloud GPU bill is huge for modest traffic. Diagnose. 7. A "private local" deployment leaked data. What likely happened?

System design 8. Design deployment for an on-prem, regulated, steady-volume workload: runtime, hardware sizing, utilization plan.

Startup / product 9. Explain how deployment choice (and utilization) affects gross margin, and when you'd move from managed to self-hosted.

14. Takeaways

1. The spectrum runs local → self-hosted cloud → managed-open → commercial API, trading ops for control/privacy.
2. Cost shapes differ: fixed HW (local), $/GPU-hour (self-host), per-token (managed/API).
3. Self-hosting is cheap only at high utilization; idle GPUs waste money.
4. Spiky/low volume → per-token; steady/high volume → owned GPUs.
5. Size memory (weights+KV+headroom) and benchmark on the real hardware before committing.
6. Stay portable (OpenAI-compatible) to move across the spectrum.

15. Artifact Checklist

• Memory-sizing verdict (local + cloud GPU).
• Local latency benchmark (tokens/sec, p95).
• 3-way cost comparison with utilization sensitivity.
• Deployment recommendation/memo.
• (Extension) spiky-traffic cost analysis.
• Notes: cost shapes + the utilization rule.

Next: 03 — Coding Models