DPO and Preference Tuning

Phase 13 · Document 03 · Fine-Tuning and Adaptation Prev: 02 — LoRA and QLoRA · Up: Phase 13 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

SFT teaches the model to imitate ideal outputs (01) — but some things are easier to express as "A is better than B" than to write the single perfect answer: tone, helpfulness, safety, "don't be verbose," nuanced judgment. Preference tuning (RLHF, and its simpler successor DPO) teaches the model from comparisons rather than demonstrations, aligning it to preferred behavior. It's how instruct/chat models are made helpful and safe after SFT, and how you push a fine-tune beyond imitation toward judgment. DPO matters especially because it delivers much of RLHF's benefit without the reward-model + RL infrastructure — making preference tuning practical for ordinary teams. Knowing when SFT is enough vs when you need preferences is a real seniority signal.

2. Core Concept

Plain-English primer: learn from "better vs worse," not just "the answer"

SFT data is (input → one ideal output); the model imitates it (01). Preference data is (input → a chosen (better) output, a rejected (worse) output) — pairs where a human (or model) judged one response better. Preference tuning trains the model to make the chosen response more likely and the rejected one less likely, so it learns the direction of "better" — useful for qualities (helpfulness, tone, safety, conciseness) that are easier to compare than to specify.

SFT:        input → IDEAL output            (imitate)        [01]
PREFERENCE: input → CHOSEN ≻ REJECTED        (prefer better over worse)

Where it sits: the alignment pipeline

The standard recipe (Phase 3.03): pre-train → SFT → preference tuning. SFT first teaches the model to follow instructions at all; preference tuning then refines it toward preferred behavior (helpful, harmless, honest). You typically SFT before DPO/RLHF, not instead of — preferences sharpen a model that already produces reasonable outputs.

RLHF (the original) — powerful but heavy

RLHF (Reinforcement Learning from Human Feedback):

1. Collect preference pairs (humans rank model outputs).
2. Train a reward model to predict human preference (score a response).
3. RL-optimize the LLM (PPO) to maximize the reward model's score (with a KL penalty to stay near the SFT model).

It works (it's how ChatGPT/Claude were aligned) but is complex and unstable: a separate reward model, an RL loop, careful tuning. Heavy infrastructure most teams can't run.

DPO (the practical successor)

DPO (Direct Preference Optimization) achieves the same goal without a reward model or RL. The insight: you can optimize the preference objective directly on the (chosen, rejected) pairs with a simple classification-style loss — make the policy assign higher relative probability to chosen than rejected, while a reference model (the frozen SFT model) keeps it from drifting too far. It's just a supervised-style training loop on preference pairs — far simpler and more stable than RLHF, and runs with the same tooling/LoRA you already use (02, HF TRL DPOTrainer). For most teams, DPO is how you do preference tuning.

DPO: train policy so  P(chosen) ↑ relative to P(rejected),  anchored to a frozen reference (SFT) model
     → no reward model, no RL — a stable supervised-style loop on (chosen, rejected) pairs

(Variants: IPO, KTO, ORPO — refinements/simplifications; KTO even works with thumbs-up/down rather than pairs. DPO is the common default.)

When to use preference tuning (vs SFT)

• SFT is enough when you can write the ideal output (format, style you can demonstrate). Start here.
• Preference tuning helps when "better" is easier to judge than to write — nuanced helpfulness, tone, safety/refusals, reducing verbosity/sycophancy, picking among plausible answers. Also when you have comparison data (e.g., from user thumbs or A/B feedback) but not gold answers.
• Order: SFT first (teach the behavior), then DPO (refine toward preferred) — preferences polish, they don't bootstrap from nothing.

The data: preference pairs

Preference data is (prompt, chosen, rejected). Sources: human labelers ranking outputs (gold but expensive), production signals (thumbs up/down, accepted vs rejected — the feedback flywheel, Phase 12.01), or AI feedback (a strong model judges — "RLAIF"/Constitutional-AI style, with the LLM-judge caveats: bias, calibration). Quality and consistency of the preferences matter as much as SFT data quality (06).

Caveats

• Don't drift too far — DPO's reference model / RLHF's KL penalty keep the model close to the SFT base; over-optimizing preferences can degrade general ability or game the signal (reward hacking in RLHF).
• Preferences encode whoever labeled them — biases/values of the labelers (or the AI judge) get baked in; curate carefully.
• Eval before/after for the target preference and regressions (Phase 12), same discipline as SFT (01).

3. Mental Model

   SFT = imitate ONE ideal output [01]   |   PREFERENCE = learn from CHOSEN ≻ REJECTED (better vs worse)
        use preferences when "better" is easier to JUDGE than to WRITE (tone/helpfulness/safety/conciseness)

   PIPELINE: pre-train → SFT → PREFERENCE TUNING (refine toward preferred); SFT FIRST, then preference

   RLHF: preference pairs → train REWARD MODEL → RL (PPO) optimize + KL penalty   [powerful, complex/unstable, heavy infra]
   DPO: optimize preference objective DIRECTLY on (chosen, rejected) pairs, anchored to a frozen REFERENCE (SFT) model
        → NO reward model, NO RL → simple/stable supervised-style loop (TRL DPOTrainer, works with LoRA [02])  ★ the practical default
        (variants: IPO/KTO/ORPO)

   DATA = (prompt, chosen, rejected): human labels · production thumbs/accept-reject · AI feedback (judge caveats [12.02])
   CAVEATS: don't drift (reference/KL) · reward hacking · preferences encode labelers' bias · eval before/after [12]

Mnemonic: preference tuning learns from "better vs worse" (chosen ≻ rejected) — for qualities easier to judge than to write. RLHF (reward model + RL) is heavy; DPO does it directly on pairs, no RL — the practical default. SFT first, then DPO.

4. Hitchhiker's Guide

What to look for first: is "better" easier to judge than to write (→ preference tuning) or can you demonstrate the ideal (→ SFT, 01)? And do you have preference pairs? Those decide whether you need DPO at all.

What to ignore at first: RLHF/PPO. For almost all teams, DPO (or a variant) gives the benefit without the RL infrastructure — start there.

What misleads beginners:

• Preference tuning instead of SFT. It refines a model that already produces reasonable outputs — SFT first, then DPO.
• Reaching for RLHF. The reward-model + RL loop is complex/unstable; DPO is simpler and usually sufficient.
• Ignoring drift. Over-optimizing preferences degrades general ability — the reference model / KL anchor matters; eval regressions (Phase 12).
• Unexamined preference data. Preferences bake in labelers' (or the judge's) bias — curate; AI-feedback has the judge caveats.
• Thinking it teaches facts. Like SFT, it's behavior/judgment, not knowledge (00).

How experts reason: they SFT first, then use DPO (not RLHF) when the target quality is comparative (tone/helpfulness/safety/conciseness) and they have preference pairs (human, production, or AI-feedback with calibration); they keep the model anchored to a reference to avoid drift, run it with the same LoRA/eval discipline (02/01), and eval before/after for the preference and regressions (Phase 12).

What matters in production: whether DPO improved the target preference (eval), no general-ability regression (drift), the quality/bias of the preference data, and the same maintenance burden as any fine-tune (07).

How to debug/verify: "model got worse overall after DPO" → drifted too far (stronger reference/KL, fewer steps); "preference not learned" → weak/inconsistent preference data (06); "it games the signal" → reward hacking (RLHF) or noisy preferences. Always eval before/after (Phase 12).

Questions to ask: is the target quality comparative (judge > write)? did I SFT first? DPO or RLHF (almost always DPO)? where do preference pairs come from (bias)? am I anchoring to a reference? eval before/after + regressions?

What silently gets expensive/unreliable: skipping SFT, RLHF complexity/instability, drift (degraded general ability), biased preference data, and AI-feedback with an uncalibrated judge (Phase 12.02).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Preference tuning learns from (chosen ≻ rejected) comparisons, not single ideal outputs — for qualities easier to judge than to write (tone/helpfulness/safety/conciseness).
• Pipeline: pre-train → SFT → preference tuning — SFT first, then refine; preferences polish, they don't bootstrap (01).
• RLHF = reward model + RL (PPO) + KL penalty — powerful but complex/unstable, heavy infra.
• DPO optimizes the preference objective directly on pairs (no reward model, no RL), anchored to a frozen reference (SFT) model — simpler, stable, runs with LoRA (02); the practical default (variants: IPO/KTO/ORPO).
• Data = (prompt, chosen, rejected) from human labels, production thumbs/accept-reject, or AI feedback (RLAIF) — with judge caveats.
• Stay anchored (reference/KL) to avoid drift / degraded general ability; watch reward hacking (RLHF).
• Preferences encode labelers'/judge's bias — curate carefully (06).
• Eval before/after for the target preference and regressions (Phase 12) — same discipline as SFT.

9. Observations from Real Systems

• Every major chat model uses preference tuning after SFT (RLHF for the labs; the SFT→preference recipe is standard) (Phase 3.03).
• DPO replaced RLHF for most teams — same benefit, no reward model/RL; TRL's DPOTrainer made it routine (and it composes with LoRA, 02).
• Constitutional AI / RLAIF use AI feedback to scale preference labels — practical, but inherits judge bias (Phase 12.02).
• Production preference data comes from the feedback flywheel (thumbs/accept-reject) — the same signals that grow eval sets (Phase 12.01).
• Over-optimization is a known failure — too much preference tuning degrades general ability or games the reward; the reference/KL anchor and eval guard against it.

10. Common Misconceptions

11. Engineering Decision Framework

SFT OR PREFERENCE TUNING?
  can you WRITE the ideal output (format/style)? → SFT [01] (start here).
  is "better" easier to JUDGE than write (tone/helpful/safe/concise), or do you only have comparisons? → preference tuning.
  ORDER: SFT first → then preference (refine).

RUN PREFERENCE TUNING:
 1. DATA: (prompt, chosen, rejected) from human labels / production thumbs-accept / AI feedback (calibrate judge [12.02]); curate for bias [06].
 2. METHOD: DPO (default; TRL DPOTrainer, + LoRA [02]) — not RLHF unless you truly need it. Consider KTO/ORPO for simpler signals.
 3. ANCHOR to a frozen REFERENCE (SFT) model to prevent drift; modest steps.
 4. EVAL before/after for the target preference AND general-ability REGRESSIONS [12].
 5. DEPLOY + maintain like any fine-tune [07].

12. Hands-On Lab

Goal

Run DPO (with LoRA) on preference pairs to shift a behavior (e.g., conciseness or tone), and eval before/after for the preference and regressions — feeling preference-vs-imitation.

Prerequisites

• An SFT'd or instruct model (01); pip install trl peft datasets; a small set of (prompt, chosen, rejected) pairs (hand-made, from thumbs data, or AI-judged).

Steps

1. Build preference pairs: for ~100 prompts, create a chosen (preferred, e.g., concise/on-tone) and rejected (worse, e.g., verbose) response — from human judgment or an AI judge (Phase 12.02).
2. Baseline: eval the SFT/instruct model on the target preference (e.g., a conciseness/tone metric or pairwise judge) (Phase 12).
3. DPO (LoRA): run DPOTrainer with a LoRA adapter (02) and the frozen model as the reference; train modestly.
4. Eval the preference: re-measure the target preference — chosen-style responses should increase.
5. Regression check: eval general ability (out-of-distribution tasks) to confirm no drift/degradation (Phase 12); if it dropped, reduce steps / strengthen the reference anchor.
6. Contrast with SFT: note this used comparisons (chosen/rejected), not single ideal outputs — the preference-vs-imitation distinction (01).

Expected output

A DPO-tuned (LoRA) model showing an improved target preference with a regression check for drift, demonstrating preference tuning beyond SFT imitation — without RLHF infrastructure.

Debugging tips

• General ability dropped → drifted; fewer steps / stronger reference anchor.
• Preference didn't shift → weak/inconsistent preference data (06) or chosen/rejected too similar.

Extension task

Compare DPO vs SFT on the same goal (single ideal outputs vs pairs); try KTO with thumbs-up/down signals.

Production extension

Source preference pairs from the production feedback flywheel (thumbs/accept-reject, Phase 12.01); eval-gate the DPO result (Phase 12.08); deploy + maintain (07).

What to measure

Target-preference metric before/after, OOD general-ability regression (drift), preference-data consistency, DPO vs SFT comparison.

Deliverables

• A set of (prompt, chosen, rejected) pairs.
• A DPO (LoRA) run + a before/after on the target preference.
• A regression (drift) check + a note on preference-vs-imitation.

13. Verification Questions

Basic

1. How does preference data differ from SFT data?
2. What are the three steps of RLHF?
3. How does DPO avoid the reward model and RL?

Applied 4. When do you use preference tuning instead of (or after) SFT? 5. Why must the model stay anchored to a reference?

Debugging 6. After DPO the model got worse at unrelated tasks. Cause and fix. 7. The preference didn't take. What about the data might be wrong?

System design 8. Design a preference-tuning pipeline using production feedback (thumbs) + DPO + eval gating.

Startup / product 9. Why does DPO (vs RLHF) make preference alignment practical for a small team, and where do preference pairs come from?

14. Takeaways

1. Preference tuning learns from (chosen ≻ rejected) — for qualities easier to judge than to write; it goes beyond SFT imitation.
2. Pipeline: SFT first, then preference tuning (refine toward preferred).
3. RLHF (reward model + RL) is heavy; DPO does it directly on pairs (no RL) — the practical default, runs with LoRA (02).
4. Anchor to a reference to avoid drift; watch reward hacking; preference data encodes labelers' bias (06).
5. Data from human/production/AI feedback (calibrate the judge Phase 12.02); eval before/after for the preference and regressions (Phase 12).

15. Artifact Checklist

• A set of (prompt, chosen, rejected) preference pairs.
• A DPO (LoRA) run anchored to a reference model.
• A before/after on the target preference.
• A regression/drift check (general ability).
• A note on data source + bias of the preferences.

Up: Phase 13 Index · Next: 04 — Distillation