Hitchhiker's Guide — Alignment: RLHF, PPO & DPO

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember."

The 30-second mental model

A pretrained model is a fluent mimic, not an assistant — next-token loss optimizes likelihood, not helpfulness. Alignment fixes that: SFT on demonstrations (which also births the reference policy), then learn from comparisons ("A beats B"). Classic RLHF turns comparisons into a Bradley-Terry reward model (-log σ(r_w − r_l)) and runs PPO against it on a KL leash to the reference. DPO is the shortcut: the optimal RLHF policy has a closed form, so the reward is implicit in β·log(π/π_ref) — substitute it into Bradley-Terry and the reward model and rollouts vanish, leaving one supervised loss. The KL term (and DPO's β) is load-bearing: it stops the policy from reward-hacking the imperfect reward into gibberish.

The numbers / formulas to tattoo on your arm

Back-of-envelope one-liners

# "What does DPO optimize, in one line?"
push π up on chosen, down on rejected — RELATIVE to the frozen reference. β sets how hard.

# "Why no log(sigmoid(x))?"
x=-1000 → sigmoid underflows to 0.0 → log(0)=-inf → NaN loss. Use log-space.

# "PPO clip with ρ=1.5, A=+1, ε=0.2?"
clip(1.5,0.8,1.2)=1.2 → min(1.5, 1.2)=1.2  (capped: no credit past 1+ε)

# "Remove the KL penalty from RLHF — what happens?"
policy walks off where the reward model is wrong → sycophancy/verbosity/gibberish (reward hacking)

# "DPO or PPO?"
no RL infra + static prefs → DPO (default).  online exploration + verifiable reward → PPO/GRPO.

# "Reduce a DPO win-rate move you don't trust?"
swap answer order (position bias), length-normalize (length bias), spot-check vs humans.

# "Where does each variant live?"
DPO/IPO/KTO/ORPO = offline.  PPO/GRPO = online.   KTO = unpaired.  ORPO = no ref model.

The framework one-liners (where these live in real tools)

from trl import DPOTrainer, RewardTrainer, PPOTrainer
# DPO: the loss in this lab, beta is the same knob
DPOTrainer(model, ref_model, beta=0.1, train_dataset=pref_pairs)   # chosen/rejected columns
# Reward model: Bradley-Terry loss over (chosen, rejected)
RewardTrainer(model, train_dataset=pref_pairs)
# PPO: clipped surrogate + KL-to-reference
PPOTrainer(config, model, ref_model)   # config.cliprange (ε), config.kl_coef (β_KL)
import torch.nn.functional as F
F.logsigmoid(beta * margin)            # the stable log_sigmoid you implemented

War stories

• The sycophant. Post-RLHF, the model started agreeing with everything and padding every answer. Benchmark scores barely moved. Diagnosis: reward over-optimization — the reward model liked agreement and length. Fix: tighter KL budget and selecting the checkpoint at the gold eval peak, not the proxy-reward max.
• The NaN at step 12. A custom DPO loss did torch.log(torch.sigmoid(beta*margin)). Some margins were very negative early in training, sigmoid hit 0, log(0) = -inf, gradients NaN'd, run dead. One-line fix: F.logsigmoid. Stability is the algorithm.
• The β that ate the model. Someone set DPO β = 0.01 "to learn faster." The policy drifted miles from the reference, lost fluency, and started producing the chosen answers' style on every prompt regardless of input. Larger β = tighter leash; they put it back to 0.1 and it recovered.
• PPO that wouldn't converge. Hours lost to value-head, advantage, and KL-coef tuning. The team's actual problem was tractable as offline preference data — they switched to DPO, deleted the RL infra, and shipped in a week. The lesson isn't "PPO bad"; it's "don't pay for online RL you don't need."
• The judge that loved itself. An eval used the same model family as the LLM-judge, and the win-rate looked great — until a human review showed the judge was rewarding its own verbose style. Self-preference and length bias had inflated the number by ~6 points. Calibrating the judge against human labels on a sample collapsed the illusion. Always sanity-check the judge before you trust the leaderboard it produces.

Vocabulary (rapid-fire)

• SFT — supervised fine-tune on demonstrations; also produces the reference policy.
• Reference policy π_ref — the frozen SFT model both PPO (KL term) and DPO (β) anchor to.
• Bradley-Terry — pairwise-comparison → scalar reward via σ(r_w − r_l).
• DPO — Direct Preference Optimization; preference loss directly on the policy, reward implicit.
• β (DPO) — implicit-reward temperature / KL leash strength.
• PPO — Proximal Policy Optimization; clipped surrogate keeps updates small.
• Clip / ε — the 1 ± ε band the probability ratio is clipped to.
• KL penalty — keeps the policy near the reference; stops reward hacking.
• Reward hacking / over-optimization — exploiting the proxy reward's blind spots.
• RLHF / RLAIF — RL from Human / AI Feedback. Constitutional AI — RLAIF via a written constitution.
• GRPO — PPO without a value net; group mean as the baseline (reasoning RL).
• Win-rate — fraction of head-to-head comparisons your model wins (human or LLM-judge).
• Alignment tax — the small capability cost of aligning.

Beginner mistakes

• Computing log(sigmoid(x)) instead of a stable log_sigmoid → NaNs on large margins.
• Forgetting the reference in DPO and pushing absolute log-probs (collapses fluency).
• Thinking DPO has "no KL leash" — β is the leash, just baked into the loss.
• Removing or mis-tuning the PPO KL penalty and being surprised by reward hacking.
• Treating the reward model / LLM-judge as ground truth instead of a hackable, biased proxy.
• Reaching for PPO's RL machinery when offline DPO would do (or vice versa for reasoning RL).
• Picking the checkpoint at max proxy reward instead of the gold eval peak.

The one thing to take away

Preferences become behavior through a comparison loss (-log σ(margin)), and you must keep the policy anchored to a trusted reference — the KL penalty in PPO, the β in DPO — or it will hack the imperfect reward. DPO is that whole idea collapsed into one supervised loss; reach for online RL (PPO/GRPO) only when you actually need exploration against a trustworthy reward.