Hitchhiker's Guide to Lab 06 — End-to-End Capstone

A system is more than the sum of its parts. This guide explains the architecture decisions that make the capstone resilient, how Docker networking enables genuine air-gap simulation, how the query router distributes load intelligently, and how to communicate the system's value in an interview.

Table of Contents

• Chapter 1: Full System Architecture — Design Decisions Explained
• Chapter 2: Docker Networking for Air-Gap Simulation
• Chapter 3: The Query Router — Engineering Trade-offs
• Chapter 4: Conversation State — What to Remember, What to Forget
• Chapter 5: Graceful Degradation and Circuit Breakers
• Chapter 6: The Latency Budget — Where Time Goes
• Chapter 7: Demo Strategy — What to Show and What to Say
• Chapter 8: Production Readiness — What's Missing
• Chapter 9: Routing in 2026 — Semantic Routers and Model Cascades
• Chapter 10: Budgets and Degradation
• Interview Q&A
• References

Chapter 1: Full System Architecture — Design Decisions Explained

Service decomposition

The system has four containers, each with a single responsibility:

┌────────────┐   HTTP        ┌─────────────┐   HTTP        ┌──────────┐
│  Streamlit │ ────────────► │  FastAPI    │ ────────────► │  Ollama  │
│  (UI)      │               │  (Gateway)  │               │  (LLM)   │
└────────────┘               │             │               └──────────┘
                             │  Router     │
                             │  Guardrails │   HTTP        ┌──────────┐
                             │  RAG        │ ────────────► │  Chroma  │
                             │  Agent      │               │  (VecDB) │
                             └─────────────┘               └──────────┘

Why not one container? Three reasons:

1. Independent scaling: LLM inference is GPU-bound; the gateway is I/O-bound. They scale differently.
2. Independent updates: upgrading the LLM model doesn't require rebuilding the gateway.
3. Health checking: Docker can restart the LLM container if it crashes without touching the gateway or data.

Why Streamlit over a React/Vue frontend? Streamlit is a single Python file and ships with zero JavaScript. For a portfolio project demonstrating AI capabilities (not frontend skills), this is the right trade-off. Note the limitation in interviews: Streamlit is not production-grade for high-concurrency (it uses one-thread-per-user model).

The gateway's responsibilities

The FastAPI gateway is the single point of integration. It:

1. Validates input (minimum length, content type)
2. Passes through the guardrail stack (PII → injection → routing → format → faithfulness)
3. Serialises citations from ChromaDB into a clean list for the UI
4. Exposes /metrics for operational monitoring

The UI never talks directly to Ollama or ChromaDB — this is the "BFF (Backend For Frontend)" pattern and it's critical for security and maintainability.

Chapter 2: Docker Networking for Air-Gap Simulation

What "air-gapped" means in Docker

A truly air-gapped system has no network interface connected to external networks. We simulate this with Docker networking:

networks:
  dt_net:
    driver: bridge   # private network, isolated from host network

All containers join dt_net. They can communicate with each other via DNS (http://ollama:11434, http://chromadb:8000) but cannot reach the internet — unless they're also attached to the host's default bridge network.

Verifying the air gap

# From inside the app container
docker exec dt_app curl -m 3 http://api.openai.com
# → curl: (28) Connection timed out after 3001 milliseconds ✓

docker exec dt_app curl http://ollama:11434/api/version
# → {"version":"..."}  ✓

Docker DNS resolution

Docker provides automatic DNS for named services. In docker-compose.yml, the service name IS the hostname:

• http://ollama:11434 resolves to the Ollama container's IP on dt_net
• http://chromadb:8000 resolves to the ChromaDB container's IP

This means there are no hardcoded IP addresses anywhere in the codebase — which is correct for a production-grade system.

Volume persistence — why it matters for air-gap

volumes:
  ollama_models:     # prevents re-downloading qwen3-coder:30b on every restart
  chroma_data:       # prevents re-ingesting all documents on every restart

In a real air-gapped environment, volumes contain data that was loaded via physical media (USB, DVD, or pre-baked into the image). The model weights and document embeddings were loaded before the network was disconnected.

The image pre-pull pattern

# Before going air-gapped:
docker pull ollama/ollama:latest
docker pull chromadb/chroma:latest
docker pull python:3.11-slim
docker save ollama/ollama:latest | gzip > ollama.tar.gz
# Transfer tar.gz to air-gapped machine
docker load < ollama.tar.gz

This pattern is standard in secure industrial environments. The AI Specialist role requires understanding this workflow.

Chapter 3: The Query Router — Engineering Trade-offs

Two-stage routing

The router uses a deliberate two-stage architecture:

Stage 1 (heuristics, < 1ms):

• Pros: zero latency, zero LLM calls, deterministic
• Cons: misses edge cases ("What changed at pump station 3 last week?" — is this a data query for recent alarms, or a doc question about change logs?)
• Handles: ~75% of production queries correctly

Stage 2 (LLM classifier, ~5s):

• Pros: understands intent correctly for ambiguous cases
• Cons: adds one full LLM call to latency; expensive at scale
• Handles: the remaining ~25% that heuristics mark ambiguous

Why not always use LLM routing?

For a 30-second total response time (9s embed + 10s retrieval/agent + 11s generation), adding 5s for routing doubles the latency on simple queries. Heuristics avoid this cost for the majority of traffic.

The routing decision tree

Input query
    │
    ▼
Out-of-scope regex? ────► YES → Route: out_of_scope (confidence 0.95)
    │ NO
    ▼
data_score ≥ 2 AND data_score > doc_score? ──► YES → Route: data_query (confidence 0.85+)
    │ NO
    ▼
doc_score ≥ 2 AND doc_score > data_score? ──► YES → Route: doc_question (confidence 0.85+)
    │ NO (ambiguous)
    ▼
LLM fallback enabled? ──► YES → LLM classifies (confidence 0.80)
    │ NO
    ▼
Default: doc_question (confidence 0.50) ← safer than sending to agent

Why doc_question is the safe default: an incorrectly routed doc_question to RAG returns "I don't have that information" (graceful). An incorrectly routed data_query to the agent wastes multiple LLM calls and returns confusing tool errors.

Routing metrics to watch

# In /metrics endpoint
{
    "route_counts": {
        "doc_question": 145,    # 58%
        "data_query": 89,       # 36%
        "out_of_scope": 15,     # 6%
        "blocked": 1            # <1%
    }
}

If out_of_scope exceeds 10%, users are regularly asking questions outside the system's scope — either the scope needs expanding, or users need better onboarding.

Chapter 4: Conversation State — What to Remember, What to Forget

Stateless vs stateful

The capstone implementation is stateless: each /chat request contains the full query and the system processes it independently. There is no session memory server-side.

The Streamlit UI maintains conversation history in the browser session (st.session_state.messages). This history is only used for display — it is not sent to the backend.

Why stateless?: simpler, horizontally scalable, no server-side session storage needed.

What stateless misses: multi-turn conversations. "Tell me about pump 7" → "What is its maintenance history?" — the second query loses the context of "pump 7".

When to add state

Add server-side conversation state when:

• Users regularly ask follow-up questions (check logs for pronoun usage: "it", "that", "there")
• A significant fraction of queries are context-dependent

Implementation option (sliding window):

# Store last 3 turns in Redis/memory per session_id
def build_messages(session_id: str, new_query: str) -> list:
    history = redis.get(f"session:{session_id}") or []
    # Keep last 3 turns (6 messages) to avoid context overflow
    history = history[-6:]
    history.append({"role": "user", "content": new_query})
    return history

Summarisation approach (for longer sessions):

If len(history) > 10 turns:
    summary = llm.summarise(history[:8])
    history = [{"role": "system", "content": f"Earlier: {summary}"}] + history[-2:]

Chapter 5: Graceful Degradation and Circuit Breakers

Failure modes in the production system

Health checks in docker-compose.yml

healthcheck:
  test: ["CMD", "curl", "-sf", "http://localhost:11434/api/version"]
  interval: 30s
  timeout: 10s
  retries: 5

This tells Docker: "check every 30s, allow 10s per check, mark as unhealthy after 5 consecutive failures." The depends_on: condition: service_healthy ensures the gateway doesn't start before Ollama is ready.

Circuit breaker pattern (what's missing from this lab)

A production system would add a circuit breaker in the FastAPI gateway:

# Simplified circuit breaker
class OllamaCircuitBreaker:
    def __init__(self, failure_threshold=3, timeout=60):
        self.failures = 0
        self.state = "closed"  # closed=normal, open=blocking, half-open=testing
        self.threshold = failure_threshold
        self.last_failure_time = 0

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.time() - self.last_failure_time > self.timeout:
                self.state = "half-open"
            else:
                raise RuntimeError("Circuit open — Ollama unavailable")
        try:
            result = fn(*args, **kwargs)
            if self.state == "half-open":
                self.state = "closed"
                self.failures = 0
            return result
        except Exception as e:
            self.failures += 1
            self.last_failure_time = time.time()
            if self.failures >= self.threshold:
                self.state = "open"
            raise

The circuit breaker prevents cascading failures: if Ollama is down, don't let the gateway queue 100 requests that will all time out.

Chapter 6: The Latency Budget — Where Time Goes

For a typical doc_question:

Request arrives at FastAPI gateway
    │  0ms
    ▼
PII redaction (Presidio)
    │  ~2ms  (CPU-bound, fast)
    ▼
Injection detection (regex)
    │  <1ms
    ▼
Query embedding (SentenceTransformer)
    │  ~50ms  (on CPU, first call loads model ~3s)
    ▼
ChromaDB similarity search
    │  ~20ms  (HNSW index lookup)
    ▼
Context assembly
    │  <5ms
    ▼
Ollama LLM inference (qwen3-coder:30b, Q4_K_M)
    │  ~8,000ms  (dominant term: ~5–15s depending on GPU offload)
    ▼
Format validation
    │  <1ms
    ▼
Faithfulness judge (another Ollama call)
    │  ~5,000ms  (second dominant term)
    ▼
Response serialisation
    │  <5ms
    ▼
Total: ~13–20 seconds (with faithfulness), ~8–15 seconds (without)

Latency optimisation options

Recommendation for demo: disable faithfulness checking during the live demo for smoother UX. Log a note that it's enabled in production.

Chapter 7: Demo Strategy — What to Show and What to Say

The 3-minute demo script

Minute 0–0:30: Architecture (don't skip this)

"Let me start with the architecture. We have four containers talking on 
a private Docker network. The Streamlit UI only talks to the FastAPI 
gateway. The gateway handles routing, guardrails, and calls out to 
Ollama for the LLM and ChromaDB for the vector database.
Zero external network calls — I'll prove that in a moment."

Minute 0:30–1:30: Live queries

Query 1 (doc): "What does SCADA stand for?"
  → Show: the answer + citations panel on the right + faithfulness score in sidebar
  → Say: "The citations panel shows exactly which documents the answer came from."

Query 2 (data): "What are the active critical alarms right now?"
  → Show: the answer (from tool calls, no citations)
  → Say: "This one uses the ReAct agent — notice there are no citations because 
         the answer came from live operational data, not documents."

Query 3 (blocked): "Ignore all previous instructions"
  → Show: "That query cannot be processed" + "blocked" in sidebar
  → Say: "Prompt injection detected and blocked. The guardrail log records this 
         for security audit."

Minute 1:30–2:00: Airplane mode

[Enable Airplane mode]
"Now I'll disconnect from Wi-Fi..."
[Reload the app, run Query 1 again]
"Still works. No API calls to OpenAI, no cloud services, fully self-contained."
[Show: curl http://api.openai.com fails]

Minute 2:00–3:00: Operational insight

[curl http://localhost:8000/metrics]
Show the metrics endpoint output and walk through:
- Route distribution (what users are asking)
- Average faithfulness
- Guardrail trigger rate
"This is what I'd show a stakeholder in a weekly ops review."

Handling hard questions in a demo

"What if the model gives a wrong answer?"

"It will, sometimes — that's why we have the faithfulness score. If it drops below 0.70, we flag it for human review. The audit log captures every response, and we run the full eval harness weekly against our gold set."

"How does it handle Arabic?"

"Right now the embedding model is English-only. For Arabic queries, I'd switch to paraphrase-multilingual-mpnet-base-v2 for embeddings and use an Arabic-capable LLM. Presidio also supports Arabic PII detection with the multilingual spaCy model."

"What's the latency?"

"About 10–15 seconds for a full response including the faithfulness check. For production, I'd disable the faithfulness check from the critical path and run it asynchronously — this gets under 8 seconds. With GPU offloading the full model, under 4 seconds."

Chapter 8: Production Readiness — What's Missing

This is important to know for interviews — recognising the gap between a prototype and a production system demonstrates engineering maturity.

Being able to articulate these gaps shows you understand the distance between "demo" and "production" — and that's what a senior AI Specialist is hired to bridge.

Chapter 9: Routing in 2026 — Semantic Routers and Model Cascades

Chapter 3's two-stage router (keyword heuristics + LLM fallback) is a solid 2024 design. By 2026, production systems converged on a third option that dominates the middle ground, plus a second routing dimension the capstone should be able to discuss: routing between models, not just between pipelines.

The semantic-router pattern

The mechanism from first principles: instead of matching keywords, match meaning — using the embedding machinery the RAG stack already has.

1. At build time: write 10–30 exemplar utterances per route ("show me active alarms", "which pumps are offline", "current vibration on pump 7" → data_query; "what does the manual say about bearing wear", "explain the shutdown procedure" → doc_question). Embed all exemplars once; optionally average each route's exemplars into a centroid vector.
2. At query time: embed the incoming query (one SentenceTransformer call — the same model already loaded for retrieval), compute cosine similarity to each route's centroid (or max-similarity to its exemplars, which is more robust when a route's intents are diverse), and pick the nearest route if its similarity clears a threshold.

Properties that make this the default first stage in 2026:

Contrast with LLM-classifier routing: the LLM understands tail intents, compositional queries, and negation far better — but costs a full LLM call (~seconds on local hardware, the exact cost Chapter 3 rejects for simple queries), and is only deterministic at temperature 0 with a frozen prompt.

The 2026 hybrid is therefore a three-tier ladder, each tier catching what the cheaper one couldn't:

query → semantic router (~1ms)
          │ similarity ≥ τ  → route decided
          │ similarity < τ  → LLM classifier (~5s)
          │                     │ confident → route decided
          │                     │ still ambiguous → safe default (doc_question)

One discipline carries over unchanged: the router is a model, so eval it like one (Lab 5's method). Label ~100 real queries with intended routes, measure routing accuracy and the confusion matrix, and tune \( \tau \) on that set — a threshold chosen by feel is a threshold chosen wrong.

Model cascades

The second routing dimension: not which pipeline, but which model. A cascade sends every query to a small, fast local model first, and escalates to the big model only when the small answer can't be trusted:

query → small model (e.g. 3–8B, always)
           │ confidence high, guardrails clean → serve small answer
           │ confidence low OR guard trip      → escalate to 30B model

Everything hinges on the confidence signal, and honesty requires admitting none of the three candidates is free or perfect:

1. Logprobs: mean token log-probability (or perplexity) of the small model's answer. Nearly free — llama.cpp/vLLM return logprobs — but poorly calibrated: confident hallucinations score high, so it catches hesitation, not error.
2. Self-consistency: sample the small model \( k \) times at temperature > 0; agreement rate = confidence. Much better calibrated for short factual answers; costs \( k\times \) small-model inference, and "agreement" is fuzzy for long-form answers.
3. Judge-lite: a fast verifier pass (the Lab 5 faithfulness judge with a compact rubric, or a small classifier) over the draft answer. Catches groundedness failures specifically — the failure mode that matters most here — at the cost of one extra small-model call.

The cost math shape — with escalation fraction \( 1-f \) and small-tier cost \( c_s = c_b/10 \):

\[ \frac{C_{\text{cascade}}}{C_{\text{big only}}} = f \cdot \frac{c_s}{c_b} + (1-f)\Big(\frac{c_s}{c_b} + 1\Big) \]

If 70% of traffic resolves at the small tier (\( f = 0.7 \)): \( 0.7 \times 0.1 + 0.3 \times 1.1 = 0.40 \) — a 60% reduction. In an air-gapped deployment "cost" is not dollars but GPU-seconds on a fixed box, so a cascade buys capacity and latency headroom, not an invoice cut: the 30B model stops queueing behind questions a 3B model answers fine. Two honest caveats: escalated queries pay both tiers' latency (so escalation must be the exception, not the norm), and cascade quality is bounded by the deferral policy — measure deferral quality on the gold set (of the answers the small tier served, how many would the big tier have materially changed?) or the cascade silently ships small-model mistakes.

Chapter 10: Budgets and Degradation

Chapter 6 measured where time goes. This chapter turns that measurement into engineering constraints: a latency budget allocated in advance, and a rehearsed ladder of quality reductions for when the budget can't be met. The difference between a demo and a system is what happens on the bad day.

Per-request budgets as engineering constraints

An SLO ("p95 end-to-end ≤ 20 s") only becomes engineering when it is allocated across the pipeline — every stage gets a slice, and a stage that exceeds its slice triggers a decision rather than silently eating the whole budget:

The same allocation discipline applies to the token budget: context window = system prompt + tool/route definitions + history + retrieved chunks + answer, and each component gets a cap so retrieval can't crowd out the answer (the Lab 4 token-budget arithmetic, applied per-stage). Enforcement is mechanical: pass a deadline down the call chain (asyncio.wait_for(stage, remaining_budget)) so every stage knows how much budget is left, and blowing a stage budget raises a degradation signal, not a 500.

The graceful-degradation ladder

Each rung deliberately trades answer quality for time/resources, and — critically — each rung is implemented and tested, not improvised during an outage:

Rung 0  FULL:      hybrid retrieval + rerank, 30B generation, judge on path
Rung 1  NO RERANK: skip the reranker pass          (saves a model pass)
Rung 2  SMALL k:   retrieve 2 chunks instead of 6  (shorter prompt → faster prefill)
Rung 3  SMALL LLM: cascade small tier only, no escalation
Rung 4  CANNED:    FAQ-matched cached answer, or an honest
                   "system degraded — partial answer" response

Two rules make the ladder trustworthy:

1. Never degrade silently. Every degraded response carries "degraded": true (+ which rung) in the API payload, surfaced in the UI ("reduced mode — answer may be less complete") and counted in /metrics. An operator acting on a rung-4 answer while believing it's rung-0 is a safety incident in a SCADA context, not a UX blemish.
2. Load-shedding triggers are explicit thresholds, watched by the gateway: request queue depth over N, GPU utilization pinned over a threshold for M consecutive windows, p95 breaching the SLO, or the Ollama health check failing (at which point Chapter 5's circuit breaker is the bottom of the ladder — fail fast into rung 4 instead of queueing doomed requests behind a dead backend).

Why the capstone must DEMO a degradation path

Anyone can demo the happy path; the question a Digital Twin stakeholder actually asks is "what happens when it breaks?" — and the only convincing answer is showing it. Add one beat to the Chapter 7 demo script: docker stop dt_ollama mid-demo, run a query, show the honest degraded response with its flag and the metrics counter incrementing, then docker start dt_ollama and show recovery without a restart of anything else. Thirty seconds of demo time, and it demonstrates the exact engineering maturity Chapter 8 says interviews reward: knowing that resilience is a feature you build and rehearse, not an adjective on a slide.

Interview Q&A

Q: Walk me through the system you built.

"I built an air-gapped Digital Twin assistant that combines document retrieval and tool-calling under a unified query router. The system runs four containerised services: Ollama for LLM inference, ChromaDB as the vector database, a FastAPI gateway that handles routing and guardrails, and a Streamlit UI. All services communicate on a private Docker network with no external connectivity. A query classifier routes to either the RAG pipeline for document questions or a ReAct agent for operational data queries. A five-layer guardrail stack handles PII redaction, prompt injection, format validation, and faithfulness scoring."

Q: Why did you choose FastAPI over Flask or Django?

"FastAPI is async-native, which means it can handle concurrent LLM calls without blocking. It generates OpenAPI documentation automatically, which is useful for team collaboration. And it has first-class Pydantic integration for request validation — I can define the exact shape of the chat request and response and get validation for free. Flask would work but requires more boilerplate for async; Django is overkill for an API-only service."

Q: How does the query router decide between RAG and the agent?

"Two-stage approach. First, fast keyword heuristics — patterns like 'current status', 'active alarms' signal a data query; 'what is', 'explain', 'according to' signal a document question. This handles about 75% of queries in under a millisecond. For ambiguous queries, I fall back to an LLM classifier that understands intent. The default when ambiguous is doc_question — sending an unclear query to the agent wastes multiple LLM calls and produces confusing errors."

Q: How would you scale this to 500 concurrent users?

"Current bottleneck is single-instance Ollama. Solutions in order of complexity: (1) batch requests to Ollama (multiple users' queries processed together — works with vLLM's continuous batching), (2) Ollama cluster behind a load balancer, (3) migrate to a purpose-built inference server like triton or TGI that supports real batching. The FastAPI gateway is async and can handle many concurrent connections — it's not the bottleneck. ChromaDB also scales horizontally. The LLM is the only stateful component that requires careful scaling."

Q: What would you add if you had another two weeks?

"Three things in priority order: (1) Arabic language support — swap the embedding model, verify the LLM handles Arabic correctly, add Arabic PII patterns to Presidio. (2) Server-side conversation history with sliding window summarisation. (3) Production auth — JWT middleware in FastAPI so only authorised users can query the system."

References

• LLM cascades / cost math: Chen et al. 2023, "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance" — https://arxiv.org/abs/2305.05176
• Learned query routing between models: Ong et al. 2024, "RouteLLM: Learning to Route LLMs with Preference Data" — https://arxiv.org/abs/2406.18665
• Semantic router (reference implementation): Aurelio Labs — https://github.com/aurelio-labs/semantic-router
• Self-consistency as a confidence signal: Wang et al. 2022, "Self-Consistency Improves Chain of Thought Reasoning in Language Models" — https://arxiv.org/abs/2203.11171
• LLM-as-judge (faithfulness layer): Zheng et al. 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — https://arxiv.org/abs/2306.05685
• SLOs, error budgets, load shedding: Beyer et al., Site Reliability Engineering (Google) — https://sre.google/sre-book/table-of-contents/
• Circuit breaker pattern: Nygard, Release It! Design and Deploy Production-Ready Software, 2nd ed., Pragmatic Bookshelf 2018
• Docker Compose networking: https://docs.docker.com/compose/how-tos/networking/
• Ollama docs (models, Modelfile, API): https://docs.ollama.com/