Lab 13 — LoRA / QLoRA Fine-Tune

Needs a GPU (consumer GPU is fine with QLoRA; or a free cloud GPU notebook). Concepts: Phase 13.01–13.02, cheatsheet 06.

Goal

Run a QLoRA SFT on a small model, eval before/after, check for catastrophic forgetting, and demonstrate the adapter is a small swappable file — proving fine-tuning teaches behavior.

Prerequisites

pip install unsloth trl peft datasets   # Unsloth = fastest, lowest-VRAM path

Steps

  1. Pick a narrow behavior the base model does poorly (a specific format/style/tone) — not facts (use RAG for facts; the ladder, Phase 13.00).
  2. Build a small SFT dataset of (input → ideal output) pairs — quality over quantity (a few hundred clean, consistent, diverse, leak-free; Phase 13.06).
  3. QLoRA load: 4-bit base + LoRA adapter (rank 16, attention target modules). Note VRAM vs full precision.
  4. Train (SFT): confirm only the adapter trains (<1% params); watch eval loss; avoid overfitting (few epochs).
  5. Eval before/after on a held-out golden set (the target behavior) — and an out-of-distribution task to check forgetting (lab-05 scorers.py).
  6. Adapter file: save it; note its size (MB) vs the base (GB) — the swappability win (Phase 13.07).
  7. (Stretch) merge + export GGUF to run locally (cheatsheet 10).

Key snippet (shape)

from unsloth import FastLanguageModel
model, tok = FastLanguageModel.from_pretrained("unsloth/llama-3.1-8b-bnb-4bit",
                                               load_in_4bit=True, max_seq_length=2048)
model = FastLanguageModel.get_peft_model(model, r=16,
            target_modules=["q_proj","k_proj","v_proj","o_proj"], lora_alpha=16)
# ... SFTTrainer on your (input -> ideal output) dataset, 1-2 epochs ...
model.save_pretrained("adapter")   # small file (MB)

Deliverables

  • A QLoRA adapter (record VRAM + trainable-param %).
  • Before/after eval on the target behavior.
  • A forgetting check (OOD task before/after).
  • Adapter file size vs base; a "when NOT to fine-tune" note.

Why it matters (interview)

Demonstrates you know fine-tuning is behavior-not-facts, can run LoRA/QLoRA, and eval-gate it — the adaptation signal (interview-prep 09).