Lab 02 — QLoRA Adapter

Phase: 10 — GenAI on Edge | Difficulty: ⭐⭐⭐⭐⭐ | Time: 4–5 hours

QLoRA (Dettmers et al., 2023) fine-tunes 65B LLMs on a single 48GB GPU using 4-bit NF4 quantization + LoRA adapters.

What you build

  • LoRALinear — frozen base weight + trainable low-rank adapters A, B (B=0 init)
  • quantize_nf4 / dequantize_nf4 — NF4 (Normal Float 4-bit) block quantization (block_size=64)
  • QLoRALinear — NF4 quantized base + trainable LoRA adapter
  • train_qlora — freeze base, train A/B/bias only with AdamW
  • merge_lora — W_merged = W_base_deq + B@A * (α/r) for deployment

Key concepts

ConceptWhat to understand
LoRAW = W_base + B@A * (α/r); rank r << d — few trainable params
NF4Quantization grid optimized for normally-distributed weights
Block quantizationIndependent scale per 64 values — better accuracy than per-tensor
Double quantQuantize the quantization constants themselves (saves ~0.4 bpw)
Memory savings7B model: FP16=14GB, INT8=7GB, NF4=3.5GB VRAM

Files

FilePurpose
lab.pyStubs with TODO markers
solution.pyComplete working implementation
test_lab.pypytest suite
requirements.txtDependencies

Run

pip install -r requirements.txt
python lab.py
pytest test_lab.py -v