Hitchhiker's Guide to Lab 03 — LoRA / QLoRA Fine-Tuning

"Training 100% of a model is like repainting every room in a house every time you want to change one wall. LoRA repaints only the wall."

Table of Contents

• The 30-Second Mental Model
• Section 1 — LoRA Mathematics
  • The low-rank decomposition
  • Initialisation
  • The alpha scaling factor
  • Why does it work?
• Section 2 — QLoRA: Adding Quantization
  • The memory problem
  • NF4 — NormalFloat 4-bit
  • Double quantization
  • Paged optimizers
  • What happens on MPS (this lab)?
• Section 3 — Training Data Format
  • Instruction tuning format
  • Why the chat template matters
  • Data quality vs data quantity
• Section 4 — Key Hyperparameters
  • LoRA rank r
  • Learning rate
  • Gradient accumulation
  • Target modules
• Section 5 — Training Dynamics
  • Reading the loss curves
  • Cosine learning rate schedule
• Section 6 — Evaluation
  • Loss-based evaluation
  • Perplexity
  • Qualitative evaluation via inference.py
  • BLEU/ROUGE for automated evaluation
• Section 7 — Merging and Deployment
  • What merge.py does
  • Export to Ollama (the Lab 1 bridge)
• Section 8 — LoRA vs QLoRA vs Full Fine-Tuning vs RAG
  • The verdict for infrastructure AI assistants
• Section 9 — Digital Twin Integration
• Section 10 — Interview Cheat-Sheet
  • 5 Phrases That Land Well in Interviews
• Section 11 — LoRA's Descendants
  • rsLoRA — rank-stabilised scaling
  • DoRA — magnitude and direction, decoupled
  • LoRA+ — asymmetric learning rates for A and B
  • Honest guidance — what actually moves the needle
• Section 12 — The Preference-Tuning Ladder: Beyond SFT
  • Why SFT alone plateaus
  • RLHF in two sentences
  • DPO — the closed-form shortcut
  • ORPO and KTO — one-paragraph variants
  • GRPO and RLVR — advantage without a value model
  • The decision table
• Section 13 — Data Is the Model
  • The synthetic-data flywheel
  • Catastrophic forgetting — the practical mitigation
  • Chat-template pitfalls — the silent quality killer
• Section 14 — When NOT to Fine-Tune
• References

The 30-Second Mental Model

FROZEN BASE MODEL (1.1B params, ~2.2 GB)
──────────────────────────────────────────────────────────
  Embedding layer         │ frozen
  Transformer block 0     │ frozen
    Attention:            │
      q_proj  W (d×d)     │ frozen ←── W_new = W + s·BA
      k_proj  W (d×d)     │ frozen ←── W_new = W + s·BA
      v_proj  W (d×d)     │ frozen ←── W_new = W + s·BA
      o_proj  W (d×d)     │ frozen ←── W_new = W + s·BA
    MLP:      ...         │ frozen
  Transformer block 1...N │ frozen
  LM head                 │ frozen
──────────────────────────────────────────────────────────
TRAINABLE: only A and B matrices (~2M params, ~8 MB)

During the forward pass, each adapted weight computes $W + sBA$ on the fly. During the backward pass, only $A$ and $B$ receive gradients. The base model weights never change.

Section 1 — LoRA Mathematics

The low-rank decomposition

For a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA constrains the weight update to a low-rank factorisation:

$$W_0 + \Delta W = W_0 + BA$$

where:

$$B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k)$$

The parameter count for the full weight matrix is $d \times k$. The LoRA parameter count for the same matrix is $r(d + k)$.

For TinyLlama's attention projections with $d = k = 2048$ and $r = 8$:

$$\text{Full:} \quad 2048 \times 2048 = 4{,}194{,}304 \text{ params per matrix}$$ $$\text{LoRA:} \quad 8 \times (2048 + 2048) = 32{,}768 \text{ params per matrix}$$

A reduction of 128× per matrix. With 4 attention projections × 22 transformer layers = 88 matrices, the total trainable parameters for rank-8 LoRA is ~2.9M out of 1.1B — about 0.26%.

Initialisation

$A$ is initialised with a random Gaussian distribution: $A \sim \mathcal{N}(0, \sigma^2)$. $B$ is initialised to zero, so $\Delta W = BA = 0$ at the start of training. This ensures the LoRA model starts identical to the base model, making training stable.

The alpha scaling factor

The output of the LoRA path is scaled by $s = \alpha / r$:

$$h = W_0 x + \frac{\alpha}{r} B A x$$

Setting $\alpha = 2r$ (the default used in this lab) gives $s = 2$, which is a stable convention from the original LoRA paper. A higher $\alpha/r$ ratio applies stronger LoRA influence relative to the frozen base.

Why does it work?

The hypothesis (validated empirically) is that the weight updates needed for task-specific adaptation have a low intrinsic rank — most of the useful gradient signal lives in a low-dimensional subspace of the full weight matrix space. Pre-trained models already represent language well; fine-tuning only needs to shift a small number of "directions" in weight space.

This is analogous to PCA: a high-dimensional data matrix can often be well-approximated by its top-$r$ principal components.

Section 2 — QLoRA: Adding Quantization

The memory problem

Fine-tuning a 7B model requires:

QLoRA makes 7B fine-tuning feasible on a single 24 GB consumer GPU.

NF4 — NormalFloat 4-bit

Standard INT4 quantization maps values uniformly across the numeric range. NF4 (Normal Float 4, from the QLoRA paper) instead places quantization levels at the quantiles of a standard normal distribution:

$$q_i = \frac{1}{2}\left(Q_{\mathcal{N}}^{-1}\left(\frac{i}{2^k + 1}\right) + Q_{\mathcal{N}}^{-1}\left(\frac{i+1}{2^k + 1}\right)\right)$$

where $Q_{\mathcal{N}}^{-1}$ is the inverse normal CDF and $k = 4$.

The intuition: pre-trained LLM weights are approximately normally distributed. NF4 allocates more quantization levels to the dense centre of this distribution (where most weights live) and fewer to the sparse tails, minimising quantization error.

In code (this lab, CUDA path only):

BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NF4 vs INT4: ~0.1–0.3 ppl improvement
    bnb_4bit_compute_dtype=torch.float16, # upcast to fp16 for matmul
    bnb_4bit_use_double_quant=True,      # quantize quantization constants too
)

Double quantization

NF4 requires storing quantization constants (one per 64-weight block). Double quantization quantizes those constants too — saving ~0.37 bits per weight (from 32-bit constants in fp32 to ~8-bit). For a 1.1B model, this saves ~50 MB.

Paged optimizers

When training on long sequences, optimizer states (Adam momentum and variance tensors) can spike VRAM and cause OOM errors. bitsandbytes implements paged Adam: optimizer states are stored in CPU RAM and paged into GPU VRAM on-demand, similar to OS virtual memory. This prevents OOM during gradient accumulation over variable-length batches.

What happens on MPS (this lab)?

On Apple Silicon, bitsandbytes 4-bit quantization is not supported (it uses CUDA-specific CUDA kernels). This lab therefore runs LoRA without quantization on MPS — the base model loads in fp16, using ~2.2 GB of unified memory. The key benefit of LoRA (0.2% trainable parameters) still fully applies; you just don't get the 4× memory reduction from NF4.

Section 3 — Training Data Format

Instruction tuning format

The training data pairs a natural-language instruction with a desired response:

{
  "instruction": "What does NPSH stand for?",
  "output": "NPSH stands for Net Positive Suction Head. It quantifies the absolute pressure available at the pump inlet above the vapour pressure of the liquid..."
}

The finetune.py formats this into the model's chat template — a model-specific string format that the model was pre-trained to expect:

<|system|>
You are an expert in industrial infrastructure systems...
<|user|>
What does NPSH stand for?
<|assistant|>
NPSH stands for Net Positive Suction Head...

Why the chat template matters

Using the correct chat template is critical. If you deviate from the format the model was pre-trained with, the model's special token embeddings for <|user|> and <|assistant|> don't fire correctly, and fine-tuning trains the model to respond to a different format than it expects at inference time. tokenizer.apply_chat_template() handles this correctly — always use it.

Data quality vs data quantity

60 high-quality examples consistently outperform 500 noisy or inconsistent examples for instruction-tuning a pre-trained model. Quality criteria:

1. Factually accurate: wrong training facts create confident incorrect responses
2. Consistent style: if some answers use bullet points and others use prose, the model learns inconsistency
3. Right length for the task: long answers for conceptual questions, short answers for definitions

Catastrophic forgetting: fine-tuning on a narrow domain can cause the model to forget general-purpose capabilities. LoRA mitigates this significantly (frozen base preserves general knowledge), but very aggressive fine-tuning (high rank, many epochs) on a small dataset can still overfit.

Section 4 — Key Hyperparameters

LoRA rank $r$

Higher rank does not always mean better results. On 60 examples, rank-8 will likely match or beat rank-32 because the dataset is too small to fill the additional rank's capacity, leading to overfitting.

Learning rate

The optimal range for LoRA is typically $1 \times 10^{-4}$ to $3 \times 10^{-4}$ (10–100× higher than full fine-tuning). Because only 0.2% of parameters are trained, the gradient noise from a small batch doesn't destabilise the larger frozen weights.

Gradient accumulation

With batch size 1 and gradient accumulation 4, the effective batch size is 4. This is important because:

• Small batch → high gradient variance → unstable training
• Large batch requires proportionally larger memory
• Gradient accumulation simulates a larger batch by accumulating gradients over multiple forward passes before taking one optimizer step

Target modules

Adding MLP projections (gate_proj, up_proj, down_proj) increases the number of LoRA matrices from 88 to 220 for a 22-layer model, roughly tripling the adapter size. This is worth trying when attention-only LoRA underfits (eval loss plateaus too early).

Section 5 — Training Dynamics

Reading the loss curves

A healthy training run shows:

Epoch 1:  train_loss 1.8 → 1.0    eval_loss 0.93
Epoch 2:  train_loss 0.85 → 0.65  eval_loss 0.82
Epoch 3:  train_loss 0.55 → 0.45  eval_loss 0.80   ← best eval checkpoint
Epoch 4:  train_loss 0.38 → 0.30  eval_loss 0.86   ← overfitting begins

Signs of overfitting: eval_loss begins rising while train_loss continues falling. load_best_model_at_end=True in TrainingArguments automatically saves the best checkpoint.

Signs of underfitting: both losses plateau above 1.0. Try: increasing rank, more epochs, higher learning rate, or adding more training data.

Cosine learning rate schedule

The training uses a cosine decay schedule:

$$\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$$

Starting at warmup_ratio=0.05 of total steps, the LR ramps linearly to the peak, then decays smoothly to near zero. This prevents large gradient steps near the end of training when the model is sensitive to small changes.

Section 6 — Evaluation

Loss-based evaluation

The training harness reports eval_loss — cross-entropy loss on the held-out eval.jsonl. A final eval loss of 0.75–0.90 represents a well-trained model on this 60-example corpus. Lower loss correlates with more accurate, fluent answers.

Perplexity

Perplexity is the exponentiated average cross-entropy loss:

$$\text{PPL} = \exp\left(\frac{1}{N}\sum_{i=1}^{N} \mathcal{L}_i\right)$$

A model with eval_loss = 0.80 has perplexity $e^{0.80} \approx 2.23$. This means the model is, on average, choosing from ~2.2 equally likely next tokens — relatively confident. Base model perplexity on the domain dataset is typically 15–40 before fine-tuning.

Qualitative evaluation via inference.py

inference.py shows side-by-side comparisons. Key things to look for:

1. Domain confidence: does the fine-tuned model give a specific domain answer vs the base model's hedged generic response?
2. Factual accuracy: are the numbers correct (40–80 PSI, 10 mm/s vibration threshold, 95°C motor temperature)?
3. Hallucination rate: does the fine-tuned model make up plausible-sounding but wrong facts?
4. Format consistency: is the answer style consistent with the training data?

BLEU/ROUGE for automated evaluation

import evaluate
bleu = evaluate.load("sacrebleu")
rouge = evaluate.load("rouge")
predictions = [fine_tuned_answer]
references  = [reference_answer]
bleu.compute(predictions=predictions, references=[references])
rouge.compute(predictions=predictions, references=references)

BLEU measures n-gram precision (useful for checking factual term inclusion). ROUGE-L measures longest common subsequence (useful for checking structural similarity). Neither is a complete substitute for human evaluation but they enable automated regression testing.

Section 7 — Merging and Deployment

What merge.py does

During training, the model runs $W_0 x + sBAx$ at every forward pass — two matrix multiplications per adapted layer. Merging computes $W_{\text{merged}} = W_0 + sBA$ once and stores the result. After merging:

• Inference is the same speed as the base model (no extra LoRA computation)
• No PEFT library needed at inference time
• The model file is the same size as the base model (not +8 MB)

This is the correct path for production deployment.

Export to Ollama (the Lab 1 bridge)

After merging, export to GGUF to use with Ollama:

# 1. Clone llama.cpp (one-time)
git clone https://github.com/ggerganov/llama.cpp
pip install -r llama.cpp/requirements.txt

# 2. Convert to GGUF (Q8_0 = 8-bit quantization, good quality)
python llama.cpp/convert_hf_to_gguf.py merged-model/ \
  --outfile infra-assistant-q8.gguf \
  --outtype q8_0

# 3. Create Ollama Modelfile
cat > Modelfile << 'EOF'
FROM ./infra-assistant-q8.gguf
SYSTEM "You are an expert in industrial infrastructure systems including SCADA, pump stations, pipeline monitoring, sensors, and digital twins."
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
EOF

# 4. Register with Ollama
ollama create infra-assistant -f Modelfile

# 5. Test
ollama run infra-assistant "What does NPSH stand for?"

This model can then be used as the LLM backend for Lab 2's RAG pipeline — a complete domain-adapted AI assistant running entirely air-gapped.

Section 8 — LoRA vs QLoRA vs Full Fine-Tuning vs RAG

The fundamental decision tree for adapting a language model to a new domain:

Need to add new FACTS not in training data?
  YES → RAG (knowledge is in documents, can be updated instantly)
  NO  → Fine-tuning (knowledge needs to be in model weights)

Is the model being fine-tuned > 7B params on a single GPU?
  YES → QLoRA (4-bit base, fp16 adapters, fits in 24 GB)
  NO  → LoRA (no quantization needed, MPS or small CUDA GPU)

Is the adapter going to production with latency requirements?
  YES → Merge + quantize to GGUF (no PEFT overhead)
  NO  → Keep as separate adapter (easier to version and swap)

The verdict for infrastructure AI assistants

RAG is the right primary tool when:

• Documents are updated frequently (new SOPs, equipment manuals)
• Source attribution is required (safety documentation, compliance)
• The domain facts are already written down

Fine-tuning is the right complement when:

• The model should have confident domain vocabulary (acronyms, terminology)
• The deployment is truly air-gapped with no document corpus
• Response style needs to match domain conventions (e.g., always use metric units)

The combination — a domain-fine-tuned model as the LLM backend of a RAG pipeline — outperforms either approach alone. The fine-tuned model better understands the retrieved context because it already knows the domain vocabulary.

Section 9 — Digital Twin Integration

In the 4-layer digital twin architecture, the fine-tuned model enables:

Richer RAG answers: a domain-fine-tuned model used as the LLM in Lab 2's RAG pipeline produces more technically accurate answers because it already understands NPSH, VFD, Modbus, Kalman filter — it doesn't need to explain these terms to itself from the retrieved context.

Embedded operator training: the fine-tuned model can answer procedural questions ("how do I commission a new pressure transducer?") without a document corpus, using knowledge baked into the adapter weights.

Structured data extraction: a fine-tuned model reliably extracts structured data from SCADA alarm text logs: {"alarm": "high_pressure", "location": "pump3", "value": "112 PSI"} — critical for automating incident report generation.

Alarm explanation synthesis: given a batch of sensor readings around an alarm event, the fine-tuned model synthesises a root-cause hypothesis ("vibration spike on pump 3 is consistent with bearing wear at 92°C motor temperature and 8.4 mm/s RMS velocity") without a RAG corpus.

Section 10 — Interview Cheat-Sheet

5 Phrases That Land Well in Interviews

1. "LoRA initialises B to zero so $\Delta W = BA = 0$ at the start of training — the model is identical to the pre-trained base before any gradient steps. This makes it extremely stable."

2. "The critical distinction between QLoRA's NF4 and INT4 is that NF4 uses quantile quantization calibrated to the empirical distribution of LLM weights — which happen to be approximately Gaussian. It's not a new data type; it's a smarter allocation of the 16 representable values."

3. "On 60 training examples, rank-8 will match rank-32 because there's not enough signal to fill the additional capacity. Rank is capacity, not quality — you need the data to justify it."

4. "The adapter file is only 8 MB for a 1.1B model. The 100× size reduction isn't just about training — it means you can version, A/B test, and swap domain adaptations in seconds without redeploying the 2 GB base."

5. "For infrastructure AI, the best architecture is a domain-fine-tuned model as the RAG pipeline's LLM backend. The fine-tuned model already knows the vocabulary — NPSH, VFD, Modbus, Kalman filter — so it requires less context to understand the retrieved passages and produces more confident, accurate answers."

Section 11 — LoRA's Descendants

Between 2024 and 2026, three refinements of vanilla LoRA graduated from papers into one-line flags in peft. Each fixes a specific, identifiable weakness in the original formulation — and each is worth understanding at the mechanism level, because "which LoRA variant and why" is now a standard interview probe.

rsLoRA — rank-stabilised scaling

Section 1 established the LoRA forward pass \( h = W_0 x + \frac{\alpha}{r} BAx \). The problem hides in that innocent-looking \( \alpha / r \).

Think of \( BAx \) as a sum of \( r \) rank-one contributions — one per LoRA dimension. Sums of \( r \) roughly-independent terms don't grow like \( r \); they grow like \( \sqrt{r} \) (the same statistics as a random walk: fluctuations scale with the square root of the number of steps). So the adapter's natural output magnitude scales as \( \sqrt{r} \), and after multiplying by \( \alpha/r \), the effective contribution scales as:

\[ \frac{\alpha}{r} \cdot \sqrt{r} ;=; \frac{\alpha}{\sqrt{r}} ;\longrightarrow; 0 \quad \text{as } r \text{ grows} \]

The gradients flowing back through the adapter are scaled by the same factor. Consequence: at r = 8 nothing is visibly wrong, but a rank-64 adapter runs with its output — and its learning signal — suppressed ~2.8× relative to rank-8. High ranks are effectively trained with a throttled learning rate: you paid for capacity that the scaling starves. This is a big part of why "higher rank didn't help" became folk wisdom.

rsLoRA (rank-stabilised LoRA) changes one thing: \( s = \alpha / \sqrt{r} \) instead of \( \alpha / r \), making the adapter's effective scale rank-independent. Now r = 64 actually uses 64 ranks. In peft: LoraConfig(use_rslora=True). At r ≤ 16 the difference is negligible; from r ≥ 32 it is the difference between capacity and decoration.

DoRA — magnitude and direction, decoupled

Any weight matrix column can be factored into a magnitude (its norm — "how loud") and a direction (its unit vector — "what feature"). The DoRA authors instrumented full fine-tuning and vanilla LoRA and measured how each changes these two quantities over training. The finding: full fine-tuning makes substantial directional changes with relatively independent, often small, magnitude changes — the two move separately. Vanilla LoRA cannot do that: a single additive update \( BA \) shifts direction and magnitude in one coupled motion, so LoRA tends to trade them off proportionally. Different learning dynamics, same parameter budget.

DoRA (Weight-Decomposed Low-Rank Adaptation) makes the decomposition explicit and trains the parts separately:

\[ W' = \underbrace{m}{\text{trainable magnitude vector}} \cdot \underbrace{\frac{W_0 + BA}{\lVert W_0 + BA \rVert_c}}{\text{direction, LoRA-adapted then normalised}} \]

where \( \lVert \cdot \rVert_c \) is the per-column norm and \( m \) is initialised to \( \lVert W_0 \rVert_c \) (so training starts at the identity, same stability property as B = 0). The LoRA matrices now steer direction only, while \( m \) learns magnitude on its own schedule — matching the decoupled dynamics observed in full fine-tuning. Empirically DoRA closes part of the LoRA↔full-FT gap, most visibly at low rank (r = 8-16), at the cost of some extra training compute and memory (the normalisation needs the composed weight). It merges into plain weights afterwards — zero inference cost, same merge.py path as Section 7. In peft: LoraConfig(use_dora=True).

LoRA+ — asymmetric learning rates for A and B

Recall the initialisation asymmetry from Section 1: A starts random, B starts at zero. Look at what that does to the first gradient steps. The adapter output is \( sBAx \); the gradient reaching A is proportional to \( B^\top \)(upstream gradient) — which is exactly zero at step 0, because B is zero. A literally cannot learn until B has moved away from zero, and while B is still small, A's learning signal stays proportionally faint. The two matrices sit in structurally different positions, yet standard training hands them the same learning rate.

LoRA+ formalises this (via an infinite-width analysis of feature learning) and prescribes the fix: give B a learning rate ~16× larger than A's. B has to grow from nothing — give it longer strides; A is already a reasonable random projection — let it adjust gently. That's the entire method: one optimizer parameter-group tweak (loraplus_lr_ratio=16 in the TRL/peft helper), reported ~1-2× faster convergence and ~1% quality on hard tasks, and essentially free to try.

Honest guidance — what actually moves the needle

The variants are real improvements — and they are second-order. The first-order levers remain the ones this lab already teaches, in roughly this order of impact:

The professional recipe: run vanilla LoRA first — it is the baseline every paper compares against and the config every tool supports. Reach for rsLoRA when you deliberately move to r ≥ 32; try DoRA when a stubborn gap to full fine-tuning persists at the rank you can afford; flip on LoRA+ whenever it's a one-liner in your stack. If someone's first suggestion for a struggling fine-tune is "switch to DoRA" rather than "show me twenty training examples", they are optimising the wrong layer.

Section 12 — The Preference-Tuning Ladder: Beyond SFT

Everything in this lab so far is supervised fine-tuning (SFT): show the model demonstrations, minimise cross-entropy. This section is about what SFT cannot do, and the 2023-2026 ladder of methods built on top of it.

Why SFT alone plateaus

SFT's loss is imitation: for each prompt, push probability mass toward one reference answer, token by token. Two structural blind spots follow:

1. It cannot express "better". Cross-entropy knows one correct token per position; every alternative is equally wrong. But most quality judgments are comparative — this answer is more grounded, that one hedges less, this one's format is cleaner. A dataset of single demonstrations has no channel for that information; a (chosen, rejected) pair does.
2. It cannot express "never do this". You can't show the model an example of not hallucinating a pressure threshold. Avoidance targets — the exact thing guardrails care about — need a signal that says "of these two behaviours, this one is disqualifying", which imitation loss cannot encode.

Hence the ladder: SFT teaches the format and skill; preference methods teach the ranking over behaviours.

RLHF in two sentences

Classic RLHF: (1) train a separate reward model on human preference pairs to score answers; (2) optimise the policy with PPO to maximise that reward, with a KL-divergence penalty tethering it to the SFT reference so it doesn't wander into reward-hacked gibberish. It works — it built ChatGPT — but it holds four models in memory (policy, reference, reward, value), and PPO's instability knobs made it a specialist sport. Everything after 2023 is an attempt to keep the alignment while deleting the machinery.

DPO — the closed-form shortcut

The DPO insight, derived in one paragraph: the RLHF objective \( \max_\pi , \mathbb{E}[r(x,y)] - \beta , \text{KL}(\pi ,|, \pi_{\text{ref}}) \) has a known closed-form optimum — \( \pi^(y|x) \propto \pi_{\text{ref}}(y|x) , e^{r(x,y)/\beta} \). Solve that for the reward: \( r(x,y) = \beta \log \frac{\pi^(y|x)}{\pi_{\text{ref}}(y|x)} + \text{const} \). Now substitute this expression into the Bradley–Terry model of pairwise preferences, \( P(y_w \succ y_l) = \sigma(r_w - r_l) \) — the constants cancel, the reward model disappears from the equation entirely, and maximising preference likelihood becomes a plain classification loss on the policy itself:

\[ \mathcal{L}{\text{DPO}} = -,\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma!\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} ;-; \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right] \]

Reading each part: the log-ratio \( \beta \log \frac{\pi_\theta}{\pi_{\text{ref}}} \) is the implicit reward — how much more likely the policy makes this answer than the frozen reference does; the loss says the chosen answer's implicit reward must exceed the rejected one's (that's the \( \sigma \) of a difference — logistic classification); \( \beta \) sets the strength of the invisible KL tether (small β = allowed to drift far from the reference); and \( \pi_{\text{ref}} \) — a frozen copy of your SFT model — anchors everything, which is why DPO comes after SFT, never instead of it. No reward model, no sampling during training, no PPO: just batches of (prompt, chosen, rejected) triples through two models.

It composes perfectly with this lab's stack: TRL's DPOTrainer on a QLoRA base, where the "frozen reference" is simply the same model with the adapter disabled — no second copy in memory. A 7-8B DPO run fits the same 24 GB GPU as the SFT run.

ORPO and KTO — one-paragraph variants

ORPO (odds-ratio preference optimisation) deletes the reference model too: it runs plain SFT on the chosen answers and adds a penalty term based on the odds ratio \( \text{odds}(y) = \frac{P(y)}{1-P(y)} \), pushing the odds of chosen above rejected within a single loss — one stage, one model, straight from the base (no separate SFT phase). The trade: the fixed anchor is gone, so data quality matters even more. Use it when memory or pipeline simplicity is the binding constraint.

KTO (Kahneman-Tversky Optimisation) attacks the data requirement instead: DPO needs pairs — two answers to the same prompt, comparatively judged, which is expensive to collect. KTO needs only per-example binary labels — this answer was good / this one was bad, inspired by prospect theory's asymmetric treatment of gains and losses. That is exactly the shape of real operational logs: thumbs-up/down from your Digital Twin operators is KTO-ready data, no pairing step required. Slightly weaker signal per example than a true pair, but data you actually have beats data you wish you had.

GRPO and RLVR — advantage without a value model

PPO needs a trained value network to estimate "how good is the average answer from here?" — the baseline against which each sample is judged. GRPO (Group Relative Policy Optimization, from DeepSeekMath) replaces that entire model with a batch statistic: sample G answers (say 8-16) to the same prompt, score them all with a reward, and compute each one's advantage relative to its own group:

\[ A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)} \]

"Better than your siblings" replaces "better than a learned prediction". One model deleted, memory roughly halved, and the baseline is exact for the prompt at hand rather than approximated by a network.

GRPO's natural habitat is verifiable rewards — this is what RLVR (Reinforcement Learning with Verifiable Rewards) means: the reward is computed by a checker program, not a learned model or a human. Did the math answer match? Do the unit tests pass? Does the output parse as the required JSON schema? A verifier is cheap, deterministic, and immune to reward-model gaming — and note the air-gap fit: a Python checker needs no labelling team and no cloud judge. DeepSeek-R1 is the existence proof that this recipe scales: R1-Zero ran pure GRPO with verifiable math/code rewards on a base model — no SFT stage at all — and long-form reasoning (self-checking, backtracking, extended chains of thought) emerged from the reward signal. The production R1 added a small SFT "cold start" and multi-stage polish, but the engine is RLVR.

For this track: SCADA query generation, structured extraction (Section 9's alarm-parsing use case), and function-call formatting all have trivially checkable outputs — json.loads + schema + a SQL dry-run is a verifier. Those are GRPO-shaped problems.

The decision table

The ladder collapses into one question: what supervision do you actually have?

All of the preference methods run as LoRA/QLoRA adapter training via TRL (DPOTrainer, KTOTrainer, ORPOTrainer, GRPOTrainer) — the ladder extends this lab's hardware budget, it doesn't obsolete it.

Section 13 — Data Is the Model

Sections 11-12 tune the algorithm. In practice, the dataset decides more than everything above combined. Three disciplines separate a working domain fine-tune from a quietly broken one.

The synthetic-data flywheel

The chicken-and-egg of domain adaptation: you need hundreds of instruction-response pairs about your pumps and your SOPs, and no such dataset exists. The 2025-era answer is to manufacture it — carefully — from the documents you already have, using the bigger local model from Lab 1 as the generator:

your SOPs / manuals (human-written ground truth)
        │
        ▼
  GENERATE  (30B model, Lab 1): "given this passage, write N question-
        │    answer pairs a field engineer would actually ask"
        ▼
  JUDGE-FILTER (same or second model + rubric): grounded in the passage?
        │    factually consistent? non-trivial? correct format?  → drop failures
        ▼
  DEDUPE: embedding cosine > 0.95 or MinHash near-duplicates → drop
        │
        ▼
  train.jsonl → QLoRA (this lab) → eval → failures become new generation targets

Every step runs air-gapped — generator, judge, and trainee are all local models. Two rules keep the flywheel honest:

• The grounding rule: every synthetic example must be traceable to a human-written source passage — the generation prompt includes the passage, and the judge explicitly checks the answer is entailed by it. You are converting document knowledge into behavioural training data, not asking a model to imagine facts about your plant.
• The model-collapse caveat: recursively training models on model output degrades them — generated distributions lose their tails, errors amplify generation over generation (Shumailov et al. call it "the curse of recursion"). One grounded generation pass is safe; a loop of models feeding models is not. Concretely: keep a human-written eval set that never passes through any generator, and never promote model outputs into "ground truth" without human or verifier review.

Catastrophic forgetting — the practical mitigation

Section 3 named the risk; here is the 2026 working practice. Even with a frozen base, the adapter modulates every forward pass — an aggressively domain-tuned model can lose general instruction-following, and you will not notice from the domain eval, because the domain eval only asks domain questions. The mitigation is two-sided:

1. Mix 1-5% general instruction data into the domain training set (open general-purpose instruction datasets, downloaded to the air-gap alongside your models). This is rehearsal: the optimizer keeps seeing "normal" behaviour, so the adapter is pulled toward adding the domain rather than replacing everything else. On this lab's scale: 60 domain examples + ~3-5 general ones already measurably stabilises tone and refusal behaviour.
2. Extend the eval.jsonl discipline in both directions: keep the domain eval and a small general-capability probe (a dozen everyday instructions, or an MMLU-style slice), and run both before and after training. A fine-tune that gains 20 points on pump questions and silently loses summarisation is a regression, and only a before/after general benchmark makes that visible.

If the general probe drops: increase the general mix, lower epochs, or lower rank — in that order.

Chat-template pitfalls — the silent quality killer

Section 3 said "always use apply_chat_template()". Here is the machinery underneath, because this is the single most common way real fine-tunes get silently destroyed.

What a chat template actually is: during its own post-training, the base model saw conversations serialised in one exact token markup — e.g. <|im_start|>user\n...<|im_end|> — where the role markers are dedicated special tokens with their own embeddings, trained to mean "the user's turn starts here". The chat template is a Jinja recipe stored in tokenizer_config.json that reproduces exactly that serialisation. It is not formatting preference; it is the trained interface contract of the model.

How it fails silently: build the string by hand with the wrong markup, and the "role markers" tokenize as ordinary text fragments — <|im_start|> becomes several meaningless subword tokens instead of one special token. The model never sees the boundary signal it was trained on. And here's the trap: training loss still goes down — the model happily learns your malformed format — so nothing crashes and no metric screams. The damage surfaces only at inference: degraded quality, role confusion, responses that won't terminate. Corollary for this track: the template used in training must match the one used at serving — when you export to Ollama (Section 7), the Modelfile's TEMPLATE must agree with what you trained on, or you re-create the same mismatch in the other direction.

Loss masking on prompt tokens: the other half of correct data preparation. The training example is one token sequence (system + user + assistant), but you only want to learn the assistant's part — so the labels for all prompt tokens are set to −100 (PyTorch's ignore-index for cross-entropy):

tokens:  <|sys|> You are… <|user|> What is NPSH? <|asst|> NPSH stands for…  <eos>
labels:   -100    -100     -100    -100  -100     NPSH stands for…  <eos>

Without masking, the model spends capacity learning to predict the user's question — and on short-answer datasets, prompt tokens outnumber answer tokens, so most of your gradient budget trains the wrong thing. TRL's SFTTrainer handles this ("completion-only" / assistant-only loss), but verify it: decode one collated batch and check with your own eyes which tokens carry labels. The full pre-flight checklist: correct template applied on both train and inference sides, no doubled BOS token (template and tokenizer each adding one), EOS present at the end of every assistant turn (or the model never learns to stop), and labels masked to −100 on everything before the assistant response.

Section 14 — When NOT to Fine-Tune

The most senior answer to "should we fine-tune?" is usually "not yet". The escalation ladder, each rung roughly 10× cheaper to iterate than the next:

1. Prompt engineering      — minutes per iteration, zero infrastructure
2. Few-shot examples       — put 3-5 gold examples in the prompt; often
                             delivers the format/style win you wanted from SFT
3. RAG                     — when the failures are MISSING KNOWLEDGE (Lab 2)
4. Fine-tuning             — when the failures are BEHAVIOUR that survives
                             rungs 1-3: format discipline, terminology, tone,
                             structured-output reliability, refusal style

The dividing principle (Section 8 said it; it deserves restating as the decision rule): fine-tuning teaches behaviour — style, format, vocabulary, procedure. Facts belong in retrieval (Lab 2 guide). Facts baked into weights can't be updated without retraining, can't be audited, can't cite a source, and the model can't tell which fine-tuned "fact" is stale. A model fine-tuned on 2025 pressure thresholds will confidently recite them in 2027 — the RAG pipeline would have retrieved the revised SOP.

And the cost nobody budgets: maintenance. A LoRA adapter is welded to the exact base checkpoint it was trained on — same architecture, same weights, same tokenizer. Every base-model upgrade (and in 2024-2026, meaningful upgrades arrived roughly every six months) orphans the adapter: you retrain and re-evaluate from scratch. The durable asset is therefore not the adapter file — it's the pipeline that produced it: the curated train/eval data, the generation and filtering scripts (Section 13), and the eval harness. Those transfer to every future base model; the 8 MB of weights do not. Fine-tune when the behaviour gap is real, persistent, and worth owning that recurring bill — which, for a long-lived air-gapped deployment with a stable model registry (a Digital Twin site being the canonical example), it often genuinely is.

References

LoRA family

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models — arXiv:2106.09685 — https://arxiv.org/abs/2106.09685
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs — arXiv:2305.14314 — https://arxiv.org/abs/2305.14314
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA) — arXiv:2312.03732 — https://arxiv.org/abs/2312.03732
• Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation — arXiv:2402.09353 — https://arxiv.org/abs/2402.09353
• Hayou, Ghosh & Yu, LoRA+: Efficient Low Rank Adaptation of Large Models — arXiv:2402.12354 — https://arxiv.org/abs/2402.12354

Preference tuning & RL

• Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT/RLHF) — arXiv:2203.02155 — https://arxiv.org/abs/2203.02155
• Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model — arXiv:2305.18290 — https://arxiv.org/abs/2305.18290
• Hong, Lee & Thorne, ORPO: Monolithic Preference Optimization without Reference Model — arXiv:2403.07687 — https://arxiv.org/abs/2403.07687
• Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization — arXiv:2402.01306 — https://arxiv.org/abs/2402.01306
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO) — arXiv:2402.03300 — https://arxiv.org/abs/2402.03300
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — arXiv:2501.12948 — https://arxiv.org/abs/2501.12948

Data

• Zhou et al., LIMA: Less Is More for Alignment — arXiv:2305.11206 — https://arxiv.org/abs/2305.11206
• Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget — arXiv:2305.17493 — https://arxiv.org/abs/2305.17493

Tools & docs

• HuggingFace PEFT documentation (rsLoRA, DoRA flags) — https://huggingface.co/docs/peft
• HuggingFace TRL documentation (SFT/DPO/KTO/ORPO/GRPO trainers) — https://huggingface.co/docs/trl
• Chat templating guide — https://huggingface.co/docs/transformers/chat_templating