Lab 01 — Air-Gapped LLM Serving

Goal: Run a production-quality LLM inference API on a machine with no internet access.

By the end of this lab you will have a FastAPI server wrapping Ollama's local LLM, with structured request/response schemas, request logging, and SSE streaming — all verified working offline.

Files in This Lab

Prerequisites

1. Ollama running with a model pulled

# If Ollama isn't running yet:
ollama serve &

# Pull the model (do this once — only requires internet at pull time)
ollama pull qwen3-coder:30b

# Verify the model is present
ollama list

If you only have 8–16 GB RAM, substitute a smaller model:

ollama pull llama3.1:8b
# then change DEFAULT_MODEL in server.py

2. Python environment with the right packages

This lab uses anaconda3 Python (not system or homebrew Python). The required packages are fastapi, uvicorn, openai, and httpx.

# Check which Python has these installed
/Users/s0x/anaconda3/bin/python -c "import fastapi, uvicorn, openai, httpx; print('OK')"

# If missing, install them
/Users/s0x/anaconda3/bin/pip install fastapi uvicorn openai httpx

Running the Server

cd "AI Specialist/lab-01-airgapped-serving"

/Users/s0x/anaconda3/bin/uvicorn server:app --host 0.0.0.0 --port 8000 --reload

Expected startup output:

INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

The server is now listening. Ollama still runs separately on port 11434 — the FastAPI server proxies requests to it.

Endpoints

GET /health

Liveness check. Calls Ollama's /v1/models to verify the backend is reachable.

curl http://localhost:8000/health

{"status": "ok", "model": "qwen3-coder:30b"}

POST /chat

Synchronous chat — waits for the full response.

curl -s -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a concise infrastructure assistant."},
      {"role": "user",   "content": "What is a SCADA system? Answer in one sentence."}
    ],
    "max_tokens": 100
  }' | python -m json.tool

Expected shape:

{
  "id": "54c80d2d",
  "model": "qwen3-coder:30b",
  "response": "A SCADA system is...",
  "prompt_tokens": 28,
  "completion_tokens": 34,
  "latency_ms": 10142.0,
  "timestamp": "2026-05-27T23:13:28+00:00"
}

POST /chat/stream

Same request body, tokens arrive as Server-Sent Events as they're generated.

curl -s -X POST http://localhost:8000/chat/stream \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "List 3 SCADA sensor types."}],
    "max_tokens": 150
  }'

Each line arrives individually as the model generates:

data: Temp

data: erature

data:  sensors,

data:  pressure

data:  sensors...
data: [DONE]

Running the Tests

In a second terminal (while the server is running in the first):

/Users/s0x/anaconda3/bin/python test_lab1.py

Expected output:

[PASS] Health endpoint — {"status":"ok","model":"qwen3-coder:30b"}
[PASS] Chat returns 200
[PASS] Response non-empty — A SCADA (Supervisory Control and Data Acquisition) system is...
[PASS] Latency logged — 10142 ms
[PASS] Multi-turn chat — A temperature sensor that monitors boiler temperature...
[PASS] Streaming returns chunks — 31 chunks

Result: 6/6 tests passed

Note: The first test run after a fresh ollama serve will take 60–90 seconds for the initial model load. Do not kill the process — it is working. Subsequent runs take 10–20 seconds.

Observations from Running This Lab

These are real numbers from the requests.log generated during the lab session:

Key insights:

1. Cold start dominates the first request. The 18.6 GB GGUF takes ~20s to transfer to Metal GPU layers, then another 70s to generate a 34-token response cold. Warm, the same response takes 10s.

2. Response length drives latency, not prompt length. The 905ms request had an 8-token prompt but only a 15-token answer. The 90s request had a similar prompt but a longer answer on a cold model. For latency, max_tokens is the main knob.

3. requests.log is your debugging tool. When a client complains "the AI was slow today", this log tells you exactly whether it was cold-start, a long response, or a model issue.

4. The test script needs a 300s timeout. httpx defaults to 60s which kills the test on cold start (even though Ollama is still working). test_lab1.py already has timeout=300 set.

Verifying the Air-Gap (Optional)

To confirm the server runs without internet:

# macOS — take the interface down
sudo ifconfig en0 down

# Run the tests — should still get 6/6
/Users/s0x/anaconda3/bin/python test_lab1.py

# Restore network
sudo ifconfig en0 up

What breaks when offline: nothing in this lab — all model weights are local, all packages are installed, no external DNS is required. This is the point.

Extension Exercises

Once the baseline works, try these to deepen the lab:

1. Change the model — swap DEFAULT_MODEL to llama3.1:8b and re-run tests. Compare latency. Why is it faster?

2. Add a rate limiter — use slowapi to cap at 5 requests/minute per IP. What happens when a client exceeds the limit?

3. Benchmark cold vs warm systematically — restart ollama serve, hit /chat immediately, then 5 more times. Plot latency vs request number.

4. Simulate a real Digital Twin query — write a prompt that takes a JSON blob of sensor readings and asks the model to identify anomalies. Is temperature=0 or temperature=0.7 more appropriate here?

5. Add API key auth — implement bearer token validation as described in the HITCHHIKERS-GUIDE section 7.1. Verify that a request without the token gets a 401.