« All Roles

AI Specialist — Operational Digital Twin

Target role: AI Specialist @ Parsons (or equivalent infrastructure / GovTech AI roles)
Core differentiator: Production LLM deployment in air-gapped, on-prem environments with RAG over geospatial/asset data.

⚠️ Four roles, four tracks. This README and Labs 1–6 target jd.md — the applied air-gapped Digital Twin role (RAG, fine-tuning, guardrails). Three very different roles each have their own dedicated, principal-depth curriculum:

➡ jd2-inference-optimization-track/ — AI Model Porting, Optimization & Accelerator Solutions Architect (jd2.md)

Quantization, model conversion, vLLM/TGI/Triton serving, distributed inference, GPU/accelerator architecture, CV/video pipelines, capacity sizing. 9 knowledge modules + 5 labs + interview prep — from "how does a GPU run a matmul" to "size a cluster and defend the $/token to a CFO."

➡ jd4-azure-agentic-banking-track/ — Senior / Agentic AI Engineer, Banking on Microsoft Azure (jd4.md)

Azure-native, agentic, regulated-banking application engineering: Azure OpenAI, Azure AI Search (hybrid/semantic RAG), LangGraph / Microsoft Agent Framework / Foundry Agent Service, Document Intelligence, multimodal + Arabic, FastAPI/Docker/Cosmos/CI-CD, evaluation & Responsible AI. 10 knowledge modules + 7 labs + a 5-part interview-day battle plan. Start here if you're targeting jd4 — and if the interview is tomorrow, jump to its Interview-Day Battle Plan.

(jd3.md — the AppliedAI/Opus inference-plumbing role — is a separate JD with no dedicated track yet.)

What the Role Actually Requires

RequirementWhat to Demonstrate
LLM deployment on-prem / air-gappedServe a model with vLLM or Ollama, zero internet
RAG over internal documentsIngest PDFs/schemas → vector DB → retrieval chain
Fine-tuning / domain adaptationLoRA/QLoRA on a 7B model with domain corpus
Prompt pipelines + function-callingLLM calls tools to query structured data
Evaluation & guardrailsHallucination detection, safety filters, quality scoring
Data curationClean, chunk, and embed domain documents

Labs

Lab 1 — Air-Gapped LLM Serving

Goal: Deploy a 7B model locally with no internet dependency. Expose it as an OpenAI-compatible API.

Stack: Ollama (easiest) or vLLM (production-grade)

Steps:

  1. Pull llama3:8b or mistral:7b via Ollama before going offline. Verify ~/.ollama/models/ contains the weights.
  2. Disable network (sudo ifconfig en0 down on macOS or pull the Ethernet).
  3. Start the server: ollama serve → confirm it answers on localhost:11434.
  4. Hit it with the OpenAI SDK pointing at base_url="http://localhost:11434/v1".
  5. Wrap in a FastAPI service with a /chat endpoint that logs every request/response.

Test: curl the /chat endpoint with Wi-Fi off. Must return a response.

Resume bullet: "Deployed Llama-3 8B as an OpenAI-compatible REST service on air-gapped infrastructure using Ollama; validated zero-network-dependency operation and built a FastAPI wrapper with request logging."

Lab 2 — RAG Pipeline Over Domain Documents

Goal: Build a RAG pipeline that answers questions from a corpus of PDFs (asset manuals, schemas, specs).

Stack: LangChain or LlamaIndex · ChromaDB (local) · sentence-transformers for embeddings

Steps:

  1. Collect 5–10 PDF documents (use public infrastructure manuals or IEEE specs as stand-ins).
  2. Load → chunk (512 tokens, 50-token overlap) → embed with all-MiniLM-L6-v2 → store in ChromaDB on disk.
  3. Build a retrieval chain: user query → top-k chunk retrieval → stuff into prompt → LLM (Lab 1 server).
  4. Add a source citation step: return which document + page number each answer draws from.
  5. Test with 20 questions you know the answers to; log retrieval hit rate.

Key design decision to understand: chunk size vs retrieval precision trade-off. Try 256 vs 512 vs 1024 and report F1 on your 20 questions.

Resume bullet: "Built a fully local RAG pipeline (LangChain + ChromaDB) over 10 infrastructure PDFs; implemented source-citation, measured retrieval precision at three chunk sizes, achieved 84% answer accuracy on a 20-question eval set."

Lab 3 — Domain Fine-Tuning with QLoRA

Goal: Fine-tune a 7B model on domain Q&A pairs so it speaks the vocabulary of asset management / infrastructure operations.

Stack: transformers · peft · bitsandbytes · trl (SFTTrainer) · single GPU (16–24 GB)

Steps:

  1. Create or download ~500–1000 instruction pairs in the target domain (e.g., OpenAssistant filtered by topic, or synthesize with GPT-4o).
  2. Load mistralai/Mistral-7B-v0.1 in 4-bit with BitsAndBytesConfig.
  3. Attach LoRA adapters (r=16, lora_alpha=32, target q_proj and v_proj).
  4. Train with SFTTrainer for 1–2 epochs; log loss.
  5. Merge adapters and quantize the merged model to GGUF with llama.cpp for Ollama deployment.
  6. Compare base vs fine-tuned on 10 domain-specific questions (qualitative diff is fine).

Critical concept: Fine-tuning vs RAG — when to use which. Answer: fine-tune for style/vocabulary/format; RAG for factual retrieval from documents. Combine both for production.

Resume bullet: "QLoRA fine-tuned Mistral-7B on 800 domain instruction pairs (r=16, 4-bit NF4); merged and converted to GGUF for on-prem Ollama deployment; measured perplexity drop of 18% on held-out domain eval set."

Lab 4 — Function-Calling Over Structured Asset Data

Goal: Let the LLM query a mock asset/operations database via tool calls — no SQL knowledge required from the user.

Stack: OpenAI function-calling API (or Ollama with tools support) · SQLite mock DB · FastAPI

Steps:

  1. Create a SQLite DB with tables: assets(id, name, type, location_lat, location_lon, status) and maintenance_logs(asset_id, date, description). Populate with 100 fake rows.
  2. Define two tools: get_asset_status(asset_id) and search_assets_by_type(type, radius_km, lat, lon).
  3. Register these as OpenAI-style function schemas in your chat loop.
  4. Parse the LLM's tool-call response → execute SQL → return result → re-inject into conversation.
  5. Test: "Which pumping stations within 5 km of [lat, lon] had maintenance in the last 30 days?" — LLM should call both tools in sequence.

Talking point: This pattern (LLM as orchestrator, tools as data accessors) is exactly how the Digital Twin AI Assistant integrates with enterprise systems.

Resume bullet: "Implemented a tool-calling LLM agent over a geospatial asset database; defined 5 function schemas, built a tool-execution loop, and demonstrated multi-hop queries combining spatial search and maintenance history retrieval."

Lab 5 — Evaluation & Guardrails

Goal: Measure response quality and catch hallucinations / unsafe outputs before they reach users.

Stack: deepeval or ragas (for RAG eval) · presidio (PII detection) · custom judge LLM prompt

Steps:

  1. RAG eval (builds on Lab 2): use ragas to score your pipeline on Faithfulness, Answer Relevancy, and Context Recall against your 20-question set.
  2. Hallucination check: write a judge prompt — "Given CONTEXT: {ctx}\nANSWER: {ans}\nIs the answer fully supported by the context? Reply YES or NO and cite the unsupported claim." Run it over 50 answers.
  3. PII / safety filter: add presidio as a pre-processing step to redact any names, coordinates, or IDs from user input before it hits the LLM.
  4. Output length / format guardrail: if the model response is >1000 tokens or missing a required JSON key, retry with a stricter prompt (max 2 retries).
  5. Log all guardrail triggers to a file. Report: what % of responses needed intervention.

Resume bullet: "Built a multi-layer guardrail stack: RAGAS eval (faithfulness 0.87), LLM-as-judge hallucination detector (flagged 11% of responses), PII redaction via Microsoft Presidio, and format-enforcement retry logic."

Lab 6 — End-to-End: Air-Gapped Digital Twin Assistant

Goal: Wire all prior labs into a single deployable service — a chat assistant that retrieves from docs, queries structured data, and refuses out-of-scope questions.

Steps:

  1. Single docker-compose.yml: Ollama container + ChromaDB container + FastAPI app container. No external network required (--network none on the app container).
  2. Request router: classify incoming query as "document question" (→ Lab 2 RAG) or "data query" (→ Lab 4 tool-calling) or "out of scope" (→ polite refusal).
  3. Add a simple Streamlit UI: chat box + source citations panel.
  4. Record a 3-minute demo video showing all three query types working offline.

Resume bullet: "Delivered a containerised, fully air-gapped LLM assistant (Ollama + ChromaDB + FastAPI + Streamlit) combining RAG and tool-calling with a query router and guardrail layer; demonstrated live with Wi-Fi disabled."

Concepts You Must Be Able to Explain Out Loud

TopicOne-line answer to have ready
RAG vs fine-tuningRAG = retrieve facts at runtime; fine-tune = bake style/format into weights
Chunking strategyFixed-size with overlap; semantic chunking for higher precision
Embedding model choiceall-MiniLM-L6-v2 for speed; bge-large for accuracy; domain-specific if available
Hallucination causesLLM fills gaps when context is absent; fix with retrieval + faithfulness checks
Air-gap constraintsNo API calls, no pip install at runtime → freeze deps, ship weights, use GGUF
Function-calling vs RAGFunction-calling for live structured data; RAG for static document knowledge
LoRA rank choicer=8 fast & cheap; r=64 better quality; r=16 standard starting point
Guardrail layersInput filter → retrieval → LLM → output filter → format check

Interview Preparation

See interview-prep/ for system design walkthroughs and behavioural Q&A specific to this role.

  1. Lab 1 (serving) → Lab 2 (RAG) → Lab 4 (function-calling) → Lab 5 (eval) → Lab 3 (fine-tuning) → Lab 6 (capstone)

Fine-tuning (Lab 3) is placed after function-calling because most real deployments solve domain queries with RAG+tools before reaching for fine-tuning.