LoRA and QLoRA

Phase 13 · Document 02 · Fine-Tuning and Adaptation Prev: 01 — Supervised Fine-Tuning (SFT) · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

LoRA (and QLoRA) is what made fine-tuning practical — it's the reason you can fine-tune a 7B–70B model on a single consumer or modest cloud GPU instead of a cluster, in hours instead of days, for dollars instead of thousands. It's the production default for fine-tuning open-weight models (00/01): the method (SFT, 01) is unchanged — LoRA just changes which parameters you train, training a tiny adapter while freezing the giant base. Beyond efficiency, LoRA gives two superpowers: swappable adapters (one base model, many tiny task-specific adapters, hot-swapped at serving time — 07) and less catastrophic forgetting (the base is frozen, 01). Understanding LoRA/QLoRA is core to any open-weight fine-tuning.

2. Core Concept

Plain-English primer: train a small adapter, freeze the giant base

Full fine-tuning updates all of a model's billions of weights — huge memory (you need optimizer state for every parameter), slow, and it can degrade the base (01). LoRA (Low-Rank Adaptation) instead freezes the entire base model and trains a tiny set of new, low-rank "adapter" matrices inserted alongside certain weight matrices. Only the adapter (often <1% of the parameters) is trained.

The trick (the "low-rank" part): a weight update ΔW to a big d×d matrix is approximated as the product of two skinny matrices A (d×r) and B (r×d) where the rank r is small (e.g., 8–64). So instead of training d×d numbers you train 2×d×r — orders of magnitude fewer. At inference, the output is base(x) + (A·B)(x) — the frozen base plus the small learned adjustment.

full FT:  update ALL weights W  (billions of params, optimizer state for each → huge VRAM)
LoRA:     freeze W; learn ΔW ≈ A·B  (A: d×r, B: r×d, r small) → train <1% of params
          inference: y = W·x + (A·B)·x      (base + adapter)

Why it works: the update a model needs for a specific task lives in a low-dimensional subspace — you don't need to move every weight to teach a behavior. So a low-rank approximation captures most of the benefit at a fraction of the cost.

QLoRA: LoRA on a quantized base

QLoRA = LoRA where the frozen base is loaded in 4-bit quantization (Phase 6.06) while the small LoRA adapter trains in higher precision. Since the base is frozen (not updated), quantizing it costs little quality but slashes VRAM — this is what lets you fine-tune a 70B on a single 48 GB GPU (or a 7B on a consumer card). QLoRA = quantized base + LoRA adapter → the most VRAM-efficient fine-tuning (Phase 6.02).

The LoRA knobs

• Rank r — adapter capacity. Higher r = more trainable params = more capacity (and more memory), but diminishing returns and overfitting risk. 8–32 is a common range; 16 a frequent default.
• lora_alpha — a scaling factor on the adapter's contribution (effective scale ≈ alpha/r); often set ≈ r or 2r.
• Target modules — which weight matrices get adapters. Commonly the attention projections (q_proj, k_proj, v_proj, o_proj) and sometimes the MLP. Adapting more modules = more capacity/cost.
• lora_dropout — regularization on the adapter.
• LoRA tolerates higher learning rates than full FT (e.g., ~1e-4–2e-4) since far fewer params train.

The two superpowers (beyond efficiency)

1. Swappable adapters. A LoRA adapter is a small file (megabytes, vs the multi-GB base). One base model can host many adapters (one per task/customer), hot-swapped or even batched together at serving time (vLLM multi-LoRA, 07/Phase 7). This is a massive deployment win — 50 fine-tunes = 1 base + 50 tiny adapters, not 50 full models.
2. Less catastrophic forgetting. Because the base is frozen, the model retains its general abilities better than full FT (01) — you're adding a behavior, not overwriting the model.

Merge vs keep-as-adapter

After training you can either keep the adapter separate (load base + adapter at inference — enables swapping, multi-LoRA) or merge it into the base weights (W' = W + A·B) to produce a standalone fine-tuned model (simpler single-model serving, can export to GGUF for local, Phase 6.03). Merging loses swappability but simplifies deployment (07).

Quality vs full fine-tuning

LoRA/QLoRA usually reaches most of full fine-tuning's quality for typical adaptation tasks, at a fraction of the cost — which is why it's the default. Full FT can edge it out when you need the absolute ceiling or very large behavioral changes, but for the common cases (format/style/narrow task, 00) LoRA is the right tool. The method is still SFT (01) (or DPO, 03) — LoRA only changes which parameters train.

Tooling

Unsloth (fastest, lowest-VRAM, great for consumer GPUs), Hugging Face PEFT + TRL, and Axolotl are the standard LoRA/QLoRA stacks (01). They handle the adapter insertion, quantization (QLoRA), and export.

3. Mental Model

   full FT: update ALL weights (huge VRAM/optimizer state, slow, more forgetting)
   LoRA: FREEZE base; learn ΔW ≈ A·B (A: d×r, B: r×d, rank r small ~8–32) → train <1% of params
         inference: y = W·x + (A·B)·x   (frozen base + tiny adapter)   [update lives in a low-rank subspace]
   QLoRA: LoRA + base loaded in 4-bit [6.06] → slashes VRAM (base is frozen, quantize it cheaply) → 70B on one GPU [6.02]

   KNOBS: rank r (capacity) · lora_alpha (scale ≈ alpha/r) · target_modules (q/k/v/o_proj…) · dropout · higher LR ok
   ★ SUPERPOWERS: SWAPPABLE adapters (MB files; 1 base + N task adapters, hot-swap/batched [07/Phase 7]) ·
                  LESS catastrophic FORGETTING (base frozen [01])
   MERGE (W+A·B → standalone, GGUF [6.03]) vs KEEP-ADAPTER (swappable/multi-LoRA)
   method is still SFT [01]/DPO [03]; LoRA only changes WHICH params train. Tools: Unsloth/PEFT+TRL/Axolotl

Mnemonic: LoRA freezes the giant base and trains a tiny low-rank adapter (<1% of params) — same SFT method, far cheaper. QLoRA adds a 4-bit base to slash VRAM. Bonus: adapters are swappable files and the frozen base forgets less.

4. Hitchhiker's Guide

What to look for first: can you fit the base in VRAM for QLoRA (4-bit), and what rank/target-modules do you need? Those determine feasibility and capacity. Then decide merge vs adapter for serving.

What to ignore at first: exhaustive hyperparameter search. Start with QLoRA, rank 16, attention target modules, alpha 16/32; tune only if eval shows under/overfitting (Phase 12).

What misleads beginners:

• Thinking LoRA is a different training method. It's SFT/DPO with fewer trainable params — the data/loss/eval discipline (01) is identical.
• Cranking rank too high. More rank = more memory + overfitting risk with little gain; start ~16.
• Forgetting the base must still fit (QLoRA). 4-bit base + adapter + activations must fit VRAM (Phase 6.02).
• Merging when you wanted swappability. Merge for single-model simplicity; keep-adapter for multi-LoRA serving (07).
• Expecting full-FT ceiling for huge behavioral changes. LoRA covers most cases; full FT for the extreme.

How experts reason: they default to QLoRA for VRAM efficiency, set modest rank (~16) on attention (and sometimes MLP) modules, run the same SFT discipline (01) (clean data, eval loss, held-out eval), exploit swappable adapters for multi-task/multi-tenant serving (07), rely on the frozen base to limit forgetting, and choose merge vs adapter by their serving needs. They reach for full FT only when LoRA demonstrably caps out.

What matters in production: VRAM fit (QLoRA), quality vs the base/full-FT baseline (eval), adapter swappability for serving many tasks, and the merge-vs-adapter decision (07).

How to debug/verify: OOM during training → QLoRA (4-bit base), lower rank, smaller batch/grad-accum (Phase 6.02); underfitting → raise rank / add target modules / more epochs; overfitting → lower rank / fewer epochs / more data (01); quality below full-FT → consider higher rank or (rarely) full FT.

Questions to ask: QLoRA (4-bit base) to fit VRAM? what rank/target modules? merge or keep-adapter? does it match the full-FT/base baseline on eval? will I serve multiple adapters (multi-LoRA)?

What silently gets expensive/unreliable: too-high rank (memory/overfit), OOM from non-quantized base, merging away needed swappability, and treating LoRA as exempt from the SFT data/eval discipline (01).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• LoRA freezes the base and trains a tiny low-rank adapter (ΔW ≈ A·B, rank r small) — often <1% of parameters — because the needed update lives in a low-rank subspace.
• It's the same SFT/DPO method (01/03) — LoRA only changes which parameters train; data/eval discipline is identical.
• QLoRA = LoRA on a 4-bit-quantized frozen base (Phase 6.06) → slashes VRAM (fine-tune large models on one GPU, Phase 6.02).
• Knobs: rank r (~16), lora_alpha (≈ alpha/r scale), target modules (attention first), dropout, higher LR ok.
• Adapters are small, swappable files — one base + many adapters, hot-swapped/batched at serving (multi-LoRA) (07, Phase 7).
• The frozen base means less catastrophic forgetting than full FT (01).
• Merge (W+A·B → standalone, GGUF-exportable) vs keep-adapter (swappable/multi-LoRA) — choose by serving needs (07, Phase 6.03).
• LoRA reaches most of full-FT quality for typical adaptation — the production default; Unsloth/PEFT+TRL/Axolotl are the tools.

9. Observations from Real Systems

• LoRA/QLoRA democratized fine-tuning — Unsloth/PEFT let people fine-tune 7B–70B on consumer/modest cloud GPUs (Phase 6).
• Multi-LoRA serving (vLLM) lets a platform host many customer/task adapters on one base — the deployment superpower for multi-tenant fine-tuning (07, Phase 7).
• QLoRA's nf4 + paged optimizers are why "fine-tune a 70B on a single 48 GB GPU" is real (Phase 6.02).
• Most production fine-tunes are LoRA SFT — full fine-tuning is reserved for cases needing the ceiling (01).
• Merging to GGUF is the common path to ship a fine-tune for local serving (Phase 6.03).

10. Common Misconceptions

11. Engineering Decision Framework

RUN LoRA/QLoRA (efficient SFT [01]):
 1. VRAM: can the base fit in higher precision? If tight → QLoRA (4-bit base [6.06/6.02]).
 2. CONFIG: rank ~16 (raise if underfit), target_modules = attention (q/k/v/o_proj) (+MLP for more capacity),
    lora_alpha ≈ r–2r, dropout small, LR ~1e-4–2e-4.
 3. TRAIN with the SAME SFT discipline [01]: clean data [06], eval loss, held-out eval (no leakage [12.01]).
 4. EVAL before/after vs base (and full-FT if you can): gains + OOD regressions (less with frozen base) [12].
 5. SERVE: keep-adapter (swappable / multi-LoRA [07/Phase 7]) OR merge (standalone, GGUF [6.03]) — by serving need.
 6. Reach for FULL FT only if LoRA demonstrably caps out.

12. Hands-On Lab

Goal

Run a QLoRA SFT on a small model, confirm it fits modest VRAM, and demonstrate adapter swappability + less forgetting vs full FT.

Prerequisites

• A GPU (consumer is fine with QLoRA); pip install unsloth trl peft datasets; the SFT dataset from 01.

Steps

1. QLoRA load: load a small instruct model in 4-bit (load_in_4bit=True) and add a LoRA adapter (rank 16, attention target modules) — Unsloth/PEFT (Phase 6.06). Note the VRAM vs loading full-precision.
2. Train (SFT): run the SFT from 01 with the adapter; confirm only the adapter trains (param count <1%); watch eval loss.
3. Adapter file: save the adapter; note its size (MB) vs the base (GB) — the swappability win.
4. Swap demo: load the base + adapter A, then base + a different adapter B; show one base serves multiple behaviors (the multi-LoRA idea, 07).
5. Forgetting check: eval the LoRA model on an out-of-distribution task vs the base; compare to (if feasible) a full-FT run — LoRA should forget less (01).
6. Merge + GGUF (stretch): merge the adapter and export GGUF for local serving (Phase 6.03); compare to keeping it as an adapter.

Expected output

A QLoRA-trained adapter (small file) fitting modest VRAM, a swap demonstration (one base, two behaviors), a forgetting comparison (LoRA vs full FT), and (stretch) a merged GGUF — showing why LoRA/QLoRA is the default.

Debugging tips

• OOM → ensure 4-bit base, lower rank/batch, grad-accum (Phase 6.02).
• Underfitting → raise rank / add MLP target modules / more epochs.

Extension task

Compare rank 8 vs 16 vs 64 on quality, memory, and overfitting; and QLoRA vs LoRA (full-precision base) on quality/VRAM.

Production extension

Serve multiple adapters on one base via vLLM multi-LoRA (07, Phase 7); decide merge-vs-adapter per serving need; eval-gate the result (Phase 12).

What to measure

VRAM (QLoRA vs full), trainable-param %, adapter file size, quality vs base/full-FT, OOD forgetting, swap correctness.

Deliverables

• A QLoRA adapter (small file) fitting modest VRAM.
• A swap demo (one base, multiple adapters).
• A forgetting comparison (LoRA vs full FT) + (stretch) a merged GGUF.

13. Verification Questions

Basic

1. What does LoRA train, and what does it freeze?
2. What is QLoRA, and why does quantizing the base cost little quality?
3. Why are LoRA adapters small and swappable?

Applied 4. What do rank r and target modules control? 5. When do you merge the adapter vs keep it separate?

Debugging 6. You OOM during fine-tuning. Three fixes. 7. LoRA underfits the task. What do you adjust?

System design 8. Design fine-tuning + serving for 20 per-customer adapters on one base model.

Startup / product 9. Why does multi-LoRA serving change the economics of offering per-customer fine-tunes?

14. Takeaways

1. LoRA freezes the base and trains a tiny low-rank adapter (<1% of params) — same SFT/DPO method (01/03), far cheaper.
2. QLoRA = LoRA on a 4-bit base → slashes VRAM (large models on one GPU) (Phase 6.06/6.02).
3. Knobs: rank (~16), alpha, target modules (attention first) — start small, raise on underfit.
4. Superpowers: swappable adapters (multi-LoRA serving) + less catastrophic forgetting (frozen base) (07/01).
5. Merge (standalone/GGUF) vs keep-adapter (swappable); LoRA reaches most of full-FT quality — the production default.

15. Artifact Checklist

• A QLoRA adapter trained on a small model (VRAM noted).
• Trainable-param % and adapter file size vs the base.
• A swap demo (one base, multiple adapters).
• A forgetting comparison (LoRA vs full FT) on OOD.
• A merge-vs-adapter decision (+ optional GGUF export).

Up: Phase 13 Index · Next: 03 — DPO and Preference Tuning