Lab 03 — LoRA / QLoRA Fine-Tuning

Goal: Fine-tune a small language model on a domain-specific infrastructure Q&A dataset using LoRA. On Apple Silicon (MPS) this runs without 4-bit quantization. On a CUDA GPU it runs as full QLoRA (4-bit base model + fp16 adapters). The same adapter file deploys anywhere.

Files

Prerequisites

1. Python packages

/Users/s0x/anaconda3/bin/pip install \
  transformers>=4.40.0 \
  peft>=0.10.0 \
  trl>=0.8.0 \
  datasets>=2.18.0 \
  accelerate>=0.27.0 \
  torch>=2.2.0

For CUDA QLoRA (optional, GPU only):

/Users/s0x/anaconda3/bin/pip install bitsandbytes>=0.42.0

2. Base model cached

The default model is TinyLlama/TinyLlama-1.1B-Chat-v1.0 (~2.2 GB download). A faster alternative is Qwen/Qwen2-0.5B-Instruct (~1 GB).

# Pre-cache the model (one-time, requires internet)
/Users/s0x/anaconda3/bin/python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
m = 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'
AutoTokenizer.from_pretrained(m)
AutoModelForCausalLM.from_pretrained(m)
print('cached')
"

Step 1 — Dry Run (verify config)

cd "AI Specialist/lab-03-qlora-finetuning"
/Users/s0x/anaconda3/bin/python finetune.py --dry-run

Expected output:

2026-05-27 23:00:00  INFO  MPS (Apple Silicon) detected — LoRA in fp16 (no 4-bit)
2026-05-27 23:00:00  INFO  Loaded 60 examples from train.jsonl
2026-05-27 23:00:00  INFO  Loaded 20 examples from eval.jsonl
2026-05-27 23:00:02  INFO  Loading tokenizer: TinyLlama/TinyLlama-1.1B-Chat-v1.0
2026-05-27 23:00:05  INFO  Loading model: TinyLlama/TinyLlama-1.1B-Chat-v1.0  ...
2026-05-27 23:00:18  INFO  Trainable params: 2,097,152 / 1,100,048,384  (0.19%)
2026-05-27 23:00:18  INFO  --dry-run: skipping training

The 0.19% trainable parameters is the LoRA magic — you're training only 2M weights out of 1.1B.

Step 2 — Train

/Users/s0x/anaconda3/bin/python finetune.py

Expected training output:

{'loss': 1.8432, 'learning_rate': 0.0002, 'epoch': 0.27}
{'loss': 1.3210, 'learning_rate': 0.00018, 'epoch': 0.53}
{'loss': 1.0841, 'learning_rate': 0.00015, 'epoch': 0.80}
{'eval_loss': 0.9342, 'epoch': 1.0}
{'loss': 0.8105, 'learning_rate': 0.00010, 'epoch': 1.33}
...
{'eval_loss': 0.7821, 'epoch': 3.0}

2026-05-27 23:08:41  INFO  LoRA adapters saved to lora-adapters/
2026-05-27 23:08:41  INFO  Adapter size: 8.4 MB

Expected wall time:

• MPS (Apple Silicon M2/M3): ~5–12 minutes for 3 epochs
• CUDA (single A100): ~45 seconds
• CPU: ~45–90 minutes (use --epochs 1 --questions 10 for a quick test)

Using a smaller model (recommended for CPU or quick iteration):

/Users/s0x/anaconda3/bin/python finetune.py --model Qwen/Qwen2-0.5B-Instruct --epochs 5
# ~3 min on MPS, ~30s on CUDA

Adapters saved to lora-adapters/:

lora-adapters/
  adapter_config.json      ← LoRA hyperparameters + base model name
  adapter_model.safetensors ← the actual adapter weights (~8 MB)
  tokenizer.json
  tokenizer_config.json

Step 3 — Compare Base vs Fine-Tuned

/Users/s0x/anaconda3/bin/python inference.py --questions 5

Expected output (excerpt):

====================================================================================================

Q01: What does RTU stand for in a SCADA system?

[BASE model  8.3s]
    RTU can stand for several things depending on the context. In telecommunications it
    might refer to a Remote Trunk Unit. In computing contexts...

[FINE-TUNED  7.9s]
    RTU stands for Remote Terminal Unit. It is a field device that interfaces physical
    sensors and actuators with the SCADA master station, converting analogue measurements
    to digital data for transmission over communication networks.

[REFERENCE]
    RTU stands for Remote Terminal Unit. It is a field device that interfaces physical
    sensors and actuators with the SCADA master station, converting analogue measurements
    to digital data for transmission over communication networks.

----------------------------------------------------------------------------------------------------

Observations:

• The base model gives a hedged, generic answer ("can stand for several things")
• The fine-tuned model confidently gives the domain-specific answer
• Domain-specific numbers and terminology (PSI thresholds, NPSH, 4–20 mA) show the clearest improvement

Step 4 — Merge Adapters (optional)

Merge the LoRA adapter weights back into the base model to create a standalone model with no PEFT dependency:

/Users/s0x/anaconda3/bin/python merge.py

Expected output:

Merging adapters into base weights ...
Saving merged model to merged-model/ ...
Done in 18.4s. Merged model size: 2.19 GB

The merged model can be run directly:

/Users/s0x/anaconda3/bin/python -c "
from transformers import pipeline
p = pipeline('text-generation', model='merged-model', max_new_tokens=100)
result = p('<|user|>\nWhat does SCADA stand for?\n<|assistant|>\n')
print(result[0]['generated_text'])
"

Experimenting

Try different LoRA ranks

Lower rank = smaller adapter, faster training, less capacity:

python finetune.py --rank 4    # tiny, ~2MB adapter
python finetune.py --rank 16   # more expressive, ~16MB adapter
python finetune.py --rank 32   # approaching full fine-tune territory

Try adding MLP projections

The default targets only attention projections. Adding MLP layers increases expressiveness:

Edit finetune.py line with target_modules:

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

Effect of training data size

Edit finetune.py to slice the dataset:

train_ds = Dataset.from_list(train_records[:20])   # 20 examples
# vs default 60 examples

Observe the difference in eval_loss at each checkpoint.

Extension Exercises

1. Export to GGUF for Ollama: after merging, use llama.cpp/convert_hf_to_gguf.py to convert to GGUF format and add it to Ollama as a custom model. This chains Lab 1, Lab 2, and Lab 3 into a complete air-gapped AI assistant.

2. Use Lab 2 answers as training data: generate the RAG answers for all 80 Q&A pairs, use those as training targets. This makes the fine-tuned model behave like a distilled version of the RAG pipeline — no retrieval needed at inference time.

3. Evaluate with perplexity: compute perplexity of the base vs fine-tuned model on eval.jsonl using evaluate library (pip install evaluate).

4. Try DPO: collect base model responses on the training questions, manually rank them, and use trl.DPOTrainer to fine-tune towards preferred responses. This is how RLHF preference alignment works.

5. Add target modules for MLP projections and compare eval loss with attention-only targeting at the same rank.