03 — QLoRA Domain Adaptation Pipeline

Role: AI Specialist
Problem: Fine-tune a 7B instruction model on infrastructure domain data using QLoRA; produce a deployable GGUF for Ollama; validate before deployment
Key challenges: Data curation in air-gap, quality gates, adapter versioning, rollback strategy

1. Clarifying Questions

Base model and hardware

• Base model? (Qwen2.5-7B-Instruct — instruction-tuned, multilingual, strong code/structured output support)
• GPU available? (NVIDIA A40, 48 GB VRAM → comfortable for 7B QLoRA at batch size 16)
• Is PyTorch CUDA stack already installed? (Yes, part of initial setup)

Training data

• What data is available? (SharePoint wiki Q&A, engineering SOPs, post-incident reports, SCADA alarm descriptions)
• How many training examples can be curated initially? (100–500; scalable to 2K over 3 months)
• What format is expected? (Instruction-input-output triples in alpaca format, or chat-formatted JSONL)
• Any privacy/classification constraints on training data? (All examples must be UNCLASSIFIED; no names of individuals in training data)

Validation and deployment

• How is success defined? (50 gold-question eval set; target: ≥ 90% format compliance, ≥ 80% answer correctness judged by domain expert)
• Deployment target? (Ollama via GGUF, same Ollama server as production base model)
• Rollback requirement? (Previous adapter version must be restorable within 1 hour)

2. Capacity Estimation

Training resource budget

• 7B model NF4 base: ~4 GB VRAM
• FP16 LoRA adapters (r=16, 4 projections × 32 layers): ~16 MB
• Activations + optimizer states (adapters only): ~1–2 GB
• Total VRAM: ~6 GB → leaves 42 GB free on A40; can run batch_size=32

Training time estimate

• 500 examples × 3 epochs × 512 max tokens per example = 768K tokens processed
• A40 throughput at batch_size=16: ~8K tokens/sec → 768K / 8K = 96 seconds per epoch
• 3 epochs: ~5 minutes total (the bottleneck is data curation, not training)

Adapter size

• r=16, target modules = q_proj, k_proj, v_proj, o_proj (4 per layer × 32 layers = 128 adapter matrices)
• Each adapter: A (r×d_head) + B (d_head×r) = 2 × 16 × 2048 = 65,536 params × 2 bytes (fp16) = 128 KB
• 128 matrices × 128 KB = 16 MB — trivially small, versioned in git

GGUF conversion overhead

• Merge adapters (add BA to W₀ for each matrix): ~5 minutes on CPU
• Quantize to Q4_K_M with llama.cpp: ~15 minutes for 7B
• Total deployment preparation: ~20 minutes

3. High-Level Architecture

DATA CURATION PIPELINE
══════════════════════════════════════════════════════════════════════

SharePoint                  Domain Expert          Synthetic Augmentation
Wiki articles               Reviews 50 Q&As        (optional, if data < 300 pairs)
SOP documents               marks correct/wrong    base model generates variants
Incident reports                 │                         │
      │                          │                         │
      ▼                          ▼                         ▼
  ┌──────────────────────────────────────────────────────────────┐
  │                  Data Curation Tools                         │
  │  ├── extract_qa_from_wiki.py  (regex + LLM to mine Q&A)      │
  │  ├── format_to_alpaca.py      (→ instruction/input/output)   │
  │  ├── deduplicate.py           (exact + near-duplicate filter) │
  │  ├── pii_scrub.py             (Presidio — no names/IDs)      │
  │  └── quality_filter.py        (length, language, format)     │
  └──────────────────────────────────────┬───────────────────────┘
                                         │ train.jsonl (500 pairs)
                                         │ eval.jsonl (50 pairs — gold set)
                                         ▼

TRAINING PIPELINE
══════════════════════════════════════════════════════════════════════

  ┌─────────────────────────────────────────────────┐
  │  finetune.py (QLoRA on A40)                     │
  │  ├── Load base: Qwen2.5-7B-Instruct (NF4)       │
  │  ├── Attach LoRA: r=16, α=32, q/k/v/o_proj      │
  │  ├── SFTTrainer: 3 epochs, lr=2e-4, warmup 50   │
  │  ├── Log: loss curve, eval_loss per epoch        │
  │  └── Save: ./adapters/v1.0/                      │
  └──────────────────────┬──────────────────────────┘
                         │ ./adapters/v1.0/ (16 MB)
                         ▼

VALIDATION GATE
══════════════════════════════════════════════════════════════════════

  ┌─────────────────────────────────────────────────┐
  │  validate.py                                    │
  │  ├── Perplexity: eval_loss vs baseline          │
  │  ├── Task accuracy: 50 gold Qs, LLM judge       │
  │  ├── Format compliance: JSON key presence       │
  │  └── Regression: accuracy ≥ base model - 2%    │
  └──────────────────────┬──────────────────────────┘
                         │ PASS → proceed to build
                         │ FAIL → block, alert, debug
                         ▼

DEPLOYMENT BUILD
══════════════════════════════════════════════════════════════════════

  ┌─────────────────────────────────────────────────┐
  │  build_gguf.py                                  │
  │  ├── merge_adapters.py  →  merged model (fp16)  │
  │  ├── convert-hf-to-gguf.py (llama.cpp)         │
  │  ├── llama-quantize Q4_K_M                     │
  │  └── Modelfile with system prompt + params     │
  └──────────────────────┬──────────────────────────┘
                         │ digital-twin-7b-v1.0.gguf + Modelfile
                         ▼

  ollama create digital-twin-v1.0 -f Modelfile
  ollama run digital-twin-v1.0    ← smoke test
  → production

4. Deep Dives

4.1 Data Curation — The Most Important Step

Most fine-tuning failures come from bad data, not bad hyperparameters.

The target format is alpaca-style JSONL:

{"instruction": "What is the maximum allowable vibration for a centrifugal pump?",
 "input": "",
 "output": "According to the maintenance manual, the vibration warning threshold is 7.5 mm/s and the critical threshold is 10.0 mm/s. Exceeding the critical threshold indicates imminent bearing failure and requires immediate inspection."}

Or chat-format JSONL (preferred for instruction-tuned base models like Qwen):

{"messages": [
  {"role": "system", "content": "You are the Infrastructure Digital Twin AI Assistant. Answer questions about the water network using only documented information. Always state the source."},
  {"role": "user", "content": "What is the maximum allowable vibration for a centrifugal pump?"},
  {"role": "assistant", "content": "The vibration warning threshold is 7.5 mm/s and the critical threshold is 10.0 mm/s per the pump maintenance guide. Exceeding the critical threshold indicates imminent bearing failure."}
]}

Why chat format for an instruction-tuned base? The base model was fine-tuned on chat format. Training on the same format means the LoRA adapters learn the domain shift without fighting the existing chat template learned by the base.

Data quality checks:

def is_good_example(example: dict) -> bool:
    q = example["messages"][1]["content"]  # user turn
    a = example["messages"][2]["content"]  # assistant turn
    
    # Basic length checks
    if len(q) < 20 or len(q) > 500: return False
    if len(a) < 50 or len(a) > 1000: return False
    
    # No PII in training data
    if has_pii(q) or has_pii(a): return False
    
    # Answer must address the question (not a generic response)
    question_words = set(q.lower().split()) - STOPWORDS
    answer_words = set(a.lower().split()) - STOPWORDS
    overlap = len(question_words & answer_words) / len(question_words)
    if overlap < 0.2: return False  # answer doesn't address question
    
    return True

Near-duplicate removal (prevents the model from memorising repeated examples):

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

def deduplicate(examples: list[dict], threshold: float = 0.92) -> list[dict]:
    questions = [e["messages"][1]["content"] for e in examples]
    embeddings = embedder.encode(questions, show_progress_bar=True)
    
    keep = []
    for i, emb in enumerate(embeddings):
        # Check if this question is too similar to any already-kept question
        if not keep:
            keep.append(i)
            continue
        kept_embs = embeddings[[k for k in keep]]
        sims = np.dot(kept_embs, emb)  # embeddings are normalised
        if np.max(sims) < threshold:
            keep.append(i)
    
    return [examples[i] for i in keep]

4.2 LoRA Configuration Explained

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                          # rank — controls adapter capacity
    lora_alpha=32,                 # scaling: effective_lr = alpha/r = 2 (standard)
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # which matrices to adapt
    lora_dropout=0.1,              # prevent adapter overfitting on small datasets
    bias="none",                   # don't adapt bias terms
    task_type="CAUSAL_LM",
)

Why these target modules? The four attention projections (q, k, v, o) handle the model's "attention routing" — which parts of prior context to attend to when generating each token. Fine-tuning these teaches the model to attend to domain-relevant patterns. The MLP layers encode "what to say"; attention layers encode "what to look at" — for domain adaptation, attention is more impactful per parameter.

r=16 trade-off:

• r=8: 2M trainable params; converges fast; good for format/style changes; may underfit for heavy vocabulary shifts
• r=16: 4M trainable params; standard for domain adaptation; right choice for 500–2K examples
• r=32: 8M trainable params; better for complex multi-task; risk of overfitting on < 500 examples
• r=64: 16M params; full "dataset-scale" fine-tuning; only for > 5K high-quality examples

4.3 QLoRA Configuration (CUDA path)

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,             # quantize base model to 4-bit
    bnb_4bit_quant_type="nf4",     # NormalFloat 4 — optimal for normal-dist weights
    bnb_4bit_compute_dtype=torch.float16,  # dequantize to fp16 for computation
    bnb_4bit_use_double_quant=True, # quantize the quantization scales too (~0.4 bits saved)
)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",             # auto-shard across available GPUs
    torch_dtype=torch.float16,
)

# Prepare for gradient checkpointing (required for QLoRA)
model = prepare_model_for_kbit_training(model)

What prepare_model_for_kbit_training does:

1. Freezes all base model parameters (no gradients flow to quantized weights)
2. Casts all layer norms to float32 (they must stay full precision for stable training)
3. Casts the LM head to float32
4. Enables gradient checkpointing (recomputes activations rather than storing them → 60–80% activation memory savings at the cost of 30% slower training)

4.4 Training Loop Configuration

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./adapters/v1.0",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,   # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",      # smooth decay
    warmup_steps=50,                 # avoid early catastrophic forgetting
    max_seq_length=512,              # Q&A pairs are short; 512 is sufficient
    fp16=True,                       # adapters train in fp16
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
)

trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=lora_config,
    args=training_args,
)

trainer.train()
trainer.save_model()  # saves adapter only — base model not saved

Learning rate 2e-4: LoRA adapters are small and need a higher LR than full fine-tuning (where 1e-5 is typical). The adapter weights start at near-zero; they need a fast escape from the zero initialisation. 2e-4 is the standard recommendation from the LoRA paper for 7B-class models.

Cosine scheduler: decays learning rate smoothly to near-zero by epoch 3. Prevents the model from making large updates at the end of training (which would overfit on the last few batches) while still allowing meaningful learning throughout.

4.5 Validation Gate

Gate 1 — Loss regression:

# eval_loss should improve vs base model's eval_loss on the domain eval set
baseline_loss = evaluate_base_model(eval_dataset)
finetuned_loss = trainer.evaluate()["eval_loss"]
assert finetuned_loss < baseline_loss * 0.95, f"No meaningful improvement: {finetuned_loss:.3f} vs {baseline_loss:.3f}"

Gate 2 — Task accuracy (the only gate that matters to the business):

JUDGE_PROMPT = """
Question: {question}
Reference Answer: {reference}
Model Answer: {model_answer}

Is the model answer:
1. Factually correct (matches reference)?
2. Properly formatted (cites source, uses correct terminology)?
3. Complete (doesn't truncate)?

Score: 0 (wrong/incomplete), 1 (partially correct), 2 (fully correct)
Return JSON: {{"score": 0|1|2, "reason": "..."}}
"""

def task_accuracy(model, eval_set: list[dict]) -> float:
    scores = []
    for example in eval_set:
        answer = model.generate(example["question"])
        result = llm_judge(JUDGE_PROMPT.format(
            question=example["question"],
            reference=example["reference_answer"],
            model_answer=answer
        ))
        scores.append(result["score"] / 2.0)
    return sum(scores) / len(scores)

accuracy = task_accuracy(finetuned_model, gold_eval_set)
assert accuracy >= 0.80, f"Task accuracy below threshold: {accuracy:.2f}"

Gate 3 — Format compliance:

def format_compliance(model, test_prompts: list[str]) -> float:
    """Fraction of responses that include source citation."""
    correct = 0
    for prompt in test_prompts:
        response = model.generate(prompt)
        if "[Source:" in response or "According to" in response:
            correct += 1
    return correct / len(test_prompts)

assert format_compliance(finetuned_model, TEST_PROMPTS) >= 0.90

4.6 Deployment: Adapter → GGUF → Ollama

# Step 1: Merge adapters into base model (in fp16)
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base, "./adapters/v1.0")
merged = model.merge_and_unload()   # adds BA into W₀ for each adapter matrix
merged.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

# Step 2: Convert to GGUF (llama.cpp, pre-built offline)
python llama.cpp/convert-hf-to-gguf.py ./merged-model --outfile dt-assistant-v1.0.gguf --outtype f16

# Step 3: Quantize to Q4_K_M
./llama.cpp/llama-quantize dt-assistant-v1.0.gguf dt-assistant-v1.0-q4km.gguf Q4_K_M

# Step 4: Create Modelfile
cat > Modelfile << 'EOF'
FROM ./dt-assistant-v1.0-q4km.gguf

SYSTEM """
You are the AI Assistant for the National Infrastructure Digital Twin.
Answer ONLY from the provided context. Always cite your source.
If the information is not available, say so clearly.
Answer in the same language as the question.
"""

PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER stop "<|im_end|>"
EOF

# Step 5: Load into Ollama
ollama create digital-twin-v1.0 -f Modelfile

# Step 6: Smoke test
ollama run digital-twin-v1.0 "What is the vibration threshold for a centrifugal pump?"

4.7 Adapter Versioning and Rollback

Version scheme: v{major}.{minor} in ./adapters/ directory.

adapters/
├── v1.0/          ← current production
│   ├── adapter_model.safetensors
│   ├── adapter_config.json
│   └── eval_results.json       ← accuracy: 0.82, faithfulness: 0.91
├── v1.1/          ← candidate (training with 200 new examples)
│   ├── adapter_model.safetensors
│   └── adapter_config.json
└── baseline/      ← base model eval results (no adapter)
    └── eval_results.json       ← accuracy: 0.64, faithfulness: 0.87

Rollback procedure (< 5 minutes):

# 1. Remove current production model from Ollama
ollama rm digital-twin-v1.0

# 2. Check which version to roll back to
cat adapters/v0.9/eval_results.json

# 3. Rebuild GGUF from previous adapter
./scripts/build_gguf.sh v0.9

# 4. Reload into Ollama
ollama create digital-twin-v0.9 -f adapters/v0.9/Modelfile

# 5. Update the service config to point to v0.9
# (Environment variable: OLLAMA_MODEL=digital-twin-v0.9)

Always keep two versions in Ollama (current + previous) so rollback is just a config change without the 20-minute GGUF rebuild.

5. Trade-offs and Alternatives

6. Interview Talking Points

1. "Data is the hard part, not the code." Fine-tuning code is 200 lines. Curating 500 high-quality, domain-specific instruction pairs takes 2–4 weeks of domain expert time. The pipeline is worth nothing if the data is wrong. I budget 60% of the fine-tuning effort for data curation.

2. "Never deploy without a validation gate." I've seen teams ship a fine-tuned model that improved perplexity but reduced task accuracy (the model became more fluent but less factual). The gold eval set with LLM-as-judge is the only signal that matters to the user.

3. "The GGUF path is mandatory for air-gap Ollama deployment." Adapters alone don't work in production Ollama without the base model, and managing two separate artefacts (base + adapter) at a deployment site is operationally risky. Merge and quantize once; carry one GGUF file. Simpler, faster, verifiable.