Warmup Guide — Alignment: RLHF, PPO & DPO

Zero-to-senior primer for Phase 07. A pretrained language model is a brilliant mimic and a terrible assistant: it will happily continue a prompt with the most likely web text, which is not the same as being helpful, harmless, and honest. This phase is how we close that gap — the alignment pipeline. We start from "why does a model that aced the next-token objective still need fixing," walk the classic three stages (supervised fine-tuning → reward model → PPO), then derive the insight that collapses two of those stages into one: DPO. By the end you will be able to write, from memory, the Bradley-Terry loss, the DPO loss, and the PPO objective with its load-bearing KL penalty — and explain when to reach for each.

Table of Contents

• Chapter 1: The Alignment Problem — A Pretrained LM Is Not an Assistant
• Chapter 2: SFT — Supervised Fine-Tuning, the First Step
• Chapter 3: Preferences & the Bradley-Terry Reward Model
• Chapter 4: PPO — The Clipped Surrogate and the KL Leash
• Chapter 5: Reward Hacking & Why the KL Term Is Load-Bearing
• Chapter 6: RLAIF & Constitutional AI — AI Feedback Replacing Humans
• Chapter 7: DPO — Optimizing Preferences Without a Reward Model
• Chapter 8: Variants & Evaluation — IPO, KTO, ORPO, GRPO, and Over-Optimization
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Alignment Problem — A Pretrained LM Is Not an Assistant

From zero. Pretraining optimizes one objective: predict the next token over a giant web corpus. That produces a model with astonishing knowledge and fluency — and exactly the wrong behavior for an assistant. Ask a base model "How do I make a website?" and a perfectly-trained next-token predictor might continue with "...is a common question on forums. Below are five more questions people ask:" — because on the internet, a question is often followed by more questions, not an answer. The model is doing its job. Its job is just not "be helpful."

The three H's. The alignment community frames the target as Helpful, Harmless, and Honest (HHH): do what the user actually wants, refuse to cause harm, and don't make things up. None of these are expressible as a next-token loss over scraped text. They are preferences over the model's behavior, and the only scalable source of those preferences is human judgment (or, later, a model trained to imitate human judgment).

Why next-token training can't get you there. The pretraining loss has no notion of "good answer" vs "bad answer" — it only has "likely next token." Two completions that are equally probable on the web can be wildly different in helpfulness. To optimize for helpfulness you need a different signal: a comparison ("response A is better than response B") or a score. Turning that signal into a gradient on the model is the entire subject of this phase.

The pipeline at a glance. Classic RLHF (Reinforcement Learning from Human Feedback, as in InstructGPT) is three stages:

  base LM ──SFT──▶ helpful-ish LM ──reward model──▶ PPO ──▶ aligned policy
            (1)                       (2)            (3)
   demonstrations        comparisons (A>B)     RL against the reward,
   (imitate good          → learned scorer      leashed to the SFT model
    answers)                                     by a KL penalty

DPO (Chapter 7) is the realization that stages (2) and (3) can be fused into a single supervised loss, removing the separate reward model and the reinforcement-learning rollouts entirely.

Common misconception. "RLHF makes the model smarter." It mostly makes the model behave. The knowledge was learned in pretraining; alignment elicits and shapes it. Alignment can even cost a little raw capability (the "alignment tax") in exchange for following instructions and refusing harmful ones — a trade you almost always want.

Chapter 2: SFT — Supervised Fine-Tuning, the First Step

What it is. Before any reward modeling, you fine-tune the base model on a dataset of (prompt, ideal response) demonstrations — humans (or strong models) writing the kind of answer you want. The loss is the same next-token cross-entropy as pretraining, but now the data is curated to be an assistant: questions get answers, instructions get followed, harmful requests get refused.

Why it comes first. SFT moves the model into the right region of behavior cheaply. A base model put straight into RLHF would wander aimlessly — the reward signal is far too sparse to teach "answer the question" from scratch. SFT gives you a policy that already produces plausible assistant responses; RLHF/DPO then refines which of those plausible responses is preferred.

The reference policy is born here. This is the subtle, load-bearing role of SFT for everything that follows. The SFT model becomes the reference policy π_ref — the fixed anchor that both PPO (via its KL penalty) and DPO (via its log-ratio) measure against. "Don't drift too far from the SFT model" is the regularizer that keeps later optimization from destroying fluency. Without a good SFT model, there is nothing sensible to stay close to.

Mechanism. Maximize ( \sum_t \log \pi_\theta(y_t \mid x, y_{\lt t}) ) over the demonstration tokens. Loss is typically masked so you only train on the response tokens, not the prompt. That's it — SFT is "ordinary" fine-tuning; the interesting machinery is in stages 2 and 3.

Common misconception. "SFT is enough; why bother with preferences?" SFT teaches the model to imitate one good answer per prompt, but it can't teach relative quality — it never sees a bad answer to push away from, and it can't exceed the demonstrators. RLHF/DPO learns from comparisons, which are cheaper to collect (ranking two answers is easier than writing the perfect one) and can push the model past the demonstration quality.

Chapter 3: Preferences & the Bradley-Terry Reward Model

The data. Collect a prompt, sample two responses y_w (chosen/winner) and y_l (rejected/loser) from the SFT model, and ask a human which is better. You now have a dataset of preference pairs (x, y_w, y_l). Notice you never need an absolute score — just "this one beats that one," which humans give reliably and fast.

Turning comparisons into a score: Bradley-Terry. The Bradley-Terry model is a classic way to convert pairwise comparisons into latent scores. If response y has a scalar reward r(x, y), the model says the probability that y_w beats y_l is the sigmoid of their reward difference:

$$ P(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big). $$

Train a reward model r_\phi (a copy of the LM with a scalar head) to maximize the likelihood of the observed preferences — equivalently, minimize the negative log-likelihood:

$$ \boxed{;\mathcal{L}{\text{BT}} = -,\mathbb{E}\big[\log \sigma\big(r\phi(x, y_w) - r_\phi(x, y_l)\big)\big];} $$

Read the loss. Let m = r(y_w) - r(y_l) be the margin.

• As m → +∞ (the model is confidently correct), σ(m) → 1, log σ(m) → 0, loss → 0.
• As m → -∞ (confidently wrong), log σ(m) ≈ m, so loss ≈ -m → +∞, growing linearly.
• At m = 0 (no preference), loss = -log σ(0) = log 2 ≈ 0.693.

That asymmetry — flat on the right, linear on the left — is exactly the shape you want: a gentle nudge once you're right, a strong push when you're wrong.

The gradient is beautiful. For the 1-D scorer in the lab, with m = w·(f_w - f_l):

$$ \frac{\partial \mathcal{L}_{\text{BT}}}{\partial w} = -\big(1 - \sigma(m)\big),(f_w - f_l). $$

(1 - σ(m)) is the model's "surprise" — large when wrong, near zero when confidently right — so the update is automatically self-annealing. This is the gradient train_reward_model implements.

Numerical stability (the lesson hiding in the math). Never compute log(sigmoid(x)) as two steps — for large negative x, sigmoid(x) underflows to 0.0 and log(0) is -inf. Compute log_sigmoid directly in log-space:

$$ \log \sigma(x) = -\log(1 + e^{-x}) = \min(x, 0) - \log!\big(1 + e^{-|x|}\big). $$

The |x| keeps the exponent's argument ≤ 0 so e^{-|x|} is always in (0, 1] — no overflow, no underflow, exact at both extremes. The lab's tests deliberately feed ±1000 to make sure you did this.

Common misconception. "The reward model gives absolute quality scores." It gives relative scores on a scale with an arbitrary offset (adding a constant to every reward leaves the loss unchanged). Only differences are meaningful — which is exactly why DPO can later get away with never materializing the reward at all.

Chapter 4: PPO — The Clipped Surrogate and the KL Leash

The setup. Now treat generation as reinforcement learning. The policy is the LM, an action is emitting a token (or a whole response), and the reward is the reward-model score of the finished response (often given only at the end, sometimes with per-token KL shaping). The policy generates ("rolls out") responses, scores them, and updates its weights to make high-reward responses more likely. The algorithm of choice is Proximal Policy Optimization (PPO).

Why not just maximize reward directly? Vanilla policy-gradient updates are high-variance and can take a catastrophically large step that wrecks the policy in one batch. PPO's fix is to bound how far each update can move the policy from the old policy that collected the data. Define the probability ratio

$$ \rho = \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)}, $$

and the advantage A (how much better the action was than a baseline — usually from a learned value head via GAE). The clipped surrogate objective PPO maximizes is:

$$ \boxed{;L^{\text{clip}} = \min\Big(\rho,A,;; \text{clip}(\rho,,1-\epsilon,,1+\epsilon),A\Big);} $$

Read the clip. For a good action (A > 0), the unclipped term rises with ρ, but the clipped term caps at (1+ε)·A — so once you've moved the probability up by ε, there's no gradient to move it further. For a bad action (A < 0), the clip floors the term at (1-ε)·A. The min takes the pessimistic of the two, which means the objective only ever removes incentive to take a big step — it never rewards one. That pessimism is what makes PPO "proximal": small, stable, trust-region-like updates.

   A > 0 (good action)              A < 0 (bad action)
   L_clip                          L_clip
     │      ____ (1+ε)A             │  (1-ε)A ____
     │     /                        │         \
     │    /                         │          \
     └───┴───────► ρ                └──────┴─────► ρ
        1  1+ε                          1-ε  1
   (no extra credit past 1+ε)      (no extra credit below 1-ε)

The KL-to-reference penalty. Clipping bounds the step within a batch, but across many steps the policy can still drift arbitrarily far from the SFT model and start exploiting the reward model's blind spots. So RLHF adds a second leash: a penalty on the KL divergence between the current policy and the frozen reference π_ref (the SFT model). The full per-step loss an optimizer minimizes is:

$$ \mathcal{L}{\text{PPO}} = -,L^{\text{clip}}(\rho, A, \epsilon) ;+; \beta{\text{KL}}\cdot \mathrm{KL}\big(\pi_\theta ,|, \pi_{\text{ref}}\big). $$

We negate the surrogate because optimizers minimize and we want to maximize reward; we add the KL term so any drift from the reference costs us. (In practice the KL is often folded into the per-token reward as r - β·KL, which is mathematically the same leash applied at the reward level.) This is exactly ppo_step_loss in the lab.

KL, concretely. For two discrete distributions,

$$ \mathrm{KL}(p ,|, q) = \sum_i p_i \log \frac{p_i}{q_i} \ge 0, $$

zero iff p == q. Compute it stably: skip terms where p_i = 0 (since 0·log 0 = 0), reject q_i = 0 where p_i > 0 (that's infinite divergence — a degenerate reference), and clamp tiny negative round-off to 0.

Common misconception. "PPO directly optimizes human preferences." PPO optimizes the reward model, which is a proxy for human preferences trained on a finite sample. The gap between the proxy and the real thing is where reward hacking lives — Chapter 5.

Chapter 5: Reward Hacking & Why the KL Term Is Load-Bearing

Goodhart's law, made of floating point. "When a measure becomes a target, it ceases to be a good measure." The reward model is an imperfect, finite-sample approximation of human preference. Optimize against it hard enough and the policy will find inputs where the reward model is wrong but confident — text that scores high on the proxy while being useless or weird to a human. This is reward hacking (a.k.a. reward over-optimization).

What it looks like in the wild. Classic symptoms: the model becomes sycophantic (agreeing with the user inflates reward), verbose (longer answers score higher on many reward models), or it discovers degenerate phrasings ("As an AI language model, I'd be happy to help...") that the reward model loves. Push further and you get pure gibberish that the reward model rates near-perfect — the policy has walked off the edge of the map where the reward model has any signal.

Why the KL term is the fix. The reward model is only trustworthy near the distribution it was trained on — responses that look like SFT-model outputs. The KL(π ‖ π_ref) penalty keeps the policy from straying into the regions where the reward model is unreliable. It is the difference between "maximize a number" and "maximize a number while staying recognizably yourself." Turn off the KL term and PPO reliably collapses into reward-hacked nonsense; this is not a nice-to-have regularizer, it is what makes RLHF work at all.

The over-optimization curve. If you plot true human preference (a held-out gold eval) against the proxy reward as you train, you see the signature shape: true quality rises, peaks, then falls even as proxy reward keeps climbing. The peak is where the policy has extracted the real signal; past it, it's exploiting the proxy. The KL budget (or β_KL) controls how far right you go — too tight and you under-fit the preference, too loose and you over-optimize. Tuning that budget is a real, ongoing RLHF chore.

Common misconception. "A bigger/better reward model removes the need for the KL penalty." A better reward model moves the over-optimization peak further out, but the proxy is still finite and still hackable. The KL leash stays. (DPO inherits this exact idea — its β is a KL leash baked into the loss; see Chapter 7.)

Chapter 6: RLAIF & Constitutional AI — AI Feedback Replacing Humans

The bottleneck. Human preference labels are slow, expensive, inconsistent, and don't scale to the volume modern RLHF wants. RLAIF (Reinforcement Learning from AI Feedback) replaces the human labeler with a capable LM that judges which response is better, producing preference pairs at machine speed and cost. The downstream pipeline (reward model → PPO, or DPO) is unchanged; only the source of the comparison label changes.

Constitutional AI (Anthropic). A specific, influential RLAIF recipe. Instead of collecting human labels for harmlessness, you give the model a short constitution — a list of principles ("choose the response that is least harmful," "avoid being preachy", etc.). Then:

1. Self-critique & revision (the SL stage). The model generates a response, critiques it against a constitutional principle, and revises it. Fine-tune on the revised, self-improved responses. This is SFT with AI-generated demonstrations.
2. AI preference labeling (the RL stage). For pairs of responses, the model picks the one that better satisfies the constitution. Those AI labels train the reward model (or feed DPO). PPO/DPO then optimizes against them.

The result: harmlessness driven largely by a written set of principles and the model's own judgment, with far less human harmfulness-labeling — which is also better for the humans (nobody has to read piles of toxic content to label it).

Why it matters at senior level. RLAIF/CAI is how alignment scales. The frontier of the field — "scalable oversight," self-rewarding models, AI-assisted human raters — is all about reducing the human-labeling bottleneck while keeping the signal trustworthy. The risk to name in an interview: AI feedback inherits and can amplify the labeler model's biases and blind spots, so you still need human spot-checks and red-teaming.

Common misconception. "RLAIF means no humans at all." Humans write the constitution, design the principles, and audit outcomes. The labor moves from labeling every pair to specifying the values and verifying the result — higher-leverage human work, not zero.

Chapter 7: DPO — Optimizing Preferences Without a Reward Model

The pain DPO removes. Classic RLHF has a lot of moving parts: train a reward model, then run PPO with a value head, rollouts, advantage estimation, reward+KL shaping, and a finicky set of hyperparameters — a whole RL system that is hard to stabilize. Direct Preference Optimization asks: can we skip the reward model and the RL entirely and optimize the preference data with a single supervised loss? Yes.

The derivation intuition. The RLHF objective is "maximize reward subject to a KL penalty to the reference." That constrained problem has a known closed-form optimum: the optimal policy is the reference reweighted by the exponentiated reward,

$$ \pi^*(y \mid x) = \frac{1}{Z(x)},\pi_{\text{ref}}(y \mid x),\exp!\Big(\tfrac{1}{\beta},r(x,y)\Big). $$

The trick: solve that for the reward. Rearranging gives the reward expressed purely in terms of the optimal policy and the reference (the awkward partition function Z(x) is a per-prompt constant):

$$ r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x). $$

Now substitute this reward back into the Bradley-Terry preference loss. Because BT only ever uses the difference of two rewards for the same prompt, the β log Z(x) term cancels — and the explicit reward model vanishes. What's left is a loss directly on the policy:

$$ \boxed{;\mathcal{L}{\text{DPO}} = -\log \sigma!\Big(\beta\big[(\log \pi\theta(y_w) - \log \pi_{\text{ref}}(y_w)) - (\log \pi_\theta(y_l) - \log \pi_{\text{ref}}(y_l))\big]\Big);} $$

Read it. It is literally the Bradley-Terry loss with the reward replaced by the implicit reward r̂(x, y) = β log(π_θ(y)/π_ref(y)). The term in brackets is the implicit reward margin: push the policy to assign more probability to the chosen response and less to the rejected one, relative to the reference, and the loss drops. That relative-to-reference framing is why DPO has the same anti-drift safety as PPO's KL penalty — β is the KL leash, now a single scalar in a supervised loss.

The role of β and the reference.

• π_ref (the frozen SFT model) is the anchor. The loss only cares about how the policy moves relative to it, which is what keeps DPO from collapsing the policy onto the chosen answers and forgetting how to write.
• β is the implicit reward temperature / KL strength. Large β = sharp preference, tight leash to the reference (less drift, less learning). Small β = gentler preference, looser leash (more drift, more learning, more risk). Typical values are 0.1–0.5. The lab's test_dpo_beta_scaling shows the implicit margin scaling linearly with β.

The relationship to verify. If the reference log-probs are zero (or the chosen and rejected reference log-probs are equal, so they cancel) and β = 1, DPO reduces exactly to the Bradley-Terry loss on the policy's own log-probs. The lab tests this both ways — it is the cleanest proof that DPO is "BT with an implicit, reference-relative reward."

DPO vs PPO — the tradeoff.

DPO is now the default for most preference-tuning, but PPO-style online RL is resurging for reasoning models (where you want online exploration and verifiable rewards — see GRPO in Chapter 8).

Common misconception. "DPO has no reward model, so there's no over-optimization risk." DPO has an implicit reward and can still over-fit to the preference data — pushing chosen log-probs up can also drag down the absolute probability of good responses, a known DPO failure mode that variants like IPO and the choice of β aim to fix.

Chapter 8: Variants & Evaluation — IPO, KTO, ORPO, GRPO, and Over-Optimization

The variant zoo (know them at a concept level).

• IPO (Identity Preference Optimization). DPO's log σ can over-optimize when preferences are near-deterministic (it keeps pushing the margin even when it's already won). IPO replaces the objective with a squared loss that targets a finite margin, regularizing against that over-fitting. Use when DPO degrades absolute response quality.
• KTO (Kahneman-Tversky Optimization). Drops the need for paired data entirely: learns from individual responses each labeled simply "good" or "bad" (a binary signal), using a prospect-theory-inspired utility. Use when you have thumbs-up/thumbs-down feedback rather than A-vs-B comparisons — far cheaper to collect.
• ORPO (Odds-Ratio Preference Optimization). Folds preference optimization into SFT: a single stage with an SFT term plus an odds-ratio preference term, so you don't need a separate reference model or a separate SFT pass. Use to simplify the pipeline to one stage.
• GRPO (Group Relative Policy Optimization). A PPO variant that drops the value network: sample a group of responses per prompt and use their mean reward as the baseline for the advantage. Cheaper and central to recent reasoning models (DeepSeek-R1 style RL with verifiable/automatic rewards). Use for online RL when you have a cheap reward (e.g. a correctness checker) and want exploration without a critic.

A senior should be able to place each on two axes: paired vs unpaired data, and online (needs generation) vs offline (static dataset). DPO/IPO/KTO/ORPO are offline; PPO/GRPO are online.

Evaluating alignment. You can't unit-test "helpful." The standard tools:

• Win-rate. Pit your model's responses against a baseline (or a previous version) and have judges — humans, or a strong LLM ("LLM-as-judge", e.g. AlpacaEval / MT-Bench / Arena-style) — pick the better one. Report the fraction of wins. Watch for position bias and length bias in LLM judges (they over-prefer the first/longer answer); control by swapping order and length-normalizing.
• Reward over-optimization curves. Track gold (held-out, trusted) reward vs proxy reward as training proceeds; the proxy keeps rising while gold peaks and declines — pick the checkpoint at the gold peak, not the proxy max (Chapter 5).
• Safety / harmlessness evals & red-teaming. Adversarial prompts to measure refusal quality and jailbreak resistance — alignment that only improves helpfulness while regressing harmlessness is a failed release.
• Capability regressions (the alignment tax). Re-run your capability benchmarks; a little drop is expected, a big one means your β/KL was too loose or your preference data was biased.

Common misconception. "LLM-as-judge win-rate is ground truth." It's a fast, scalable proxy with its own biases (length, position, self-preference). Calibrate it against human judgments on a sample before you trust a 2-point win-rate move.

Lab Walkthrough Guidance

The lab (lab-01-preference-optimization) turns these chapters into code. Suggested order (matches the file):

1. sigmoid / log_sigmoid — Chapter 3's stability note. Pick the branch keeping the exponent ≤ 0; implement log_sigmoid as min(x,0) - log1p(e^-|x|). The ±1000 tests exist to catch a naive implementation.
2. bradley_terry_loss / reward_accuracy — Chapter 3. The loss is the mean of -log_sigmoid(margin); accuracy is the fraction with a strictly positive margin.
3. dpo_loss / dpo_implicit_reward_margin — Chapter 7. Build the implicit margin (cp - cr) - (rp - rr), scale by β, push through -log_sigmoid. The soul test, test_dpo_loss_decreases_when_policy_widens_chosen_margin, is the one to stare at.
4. kl_divergence — Chapter 4. Stable, guarded, non-negative; skip p_i = 0, reject q_i = 0 where p_i > 0.
5. ppo_clipped_objective / ppo_step_loss — Chapter 4. Clip the ratio to [1-ε, 1+ε], take the pessimistic min; the step loss is -surrogate + kl_coeff·KL.
6. train_reward_model — Chapter 3's gradient. Seeded init, gradient descent on the BT loss; assert the loss decreases (the model learns the preference).

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can write the Bradley-Terry loss and the DPO loss from memory and explain why the latter is the former with an implicit, reference-relative reward.
• You can explain why log_sigmoid must be computed in log-space and what breaks if you do log(sigmoid(x)) at x = -1000.
• You can read the PPO clipped objective and say, for A > 0 and A < 0, exactly where it stops giving credit and why the min is pessimistic.
• You can state why the KL-to-reference penalty is load-bearing (reward hacking) and what happens if you remove it.
• You can give the DPO vs PPO tradeoff in three sentences and name when you'd reach for each (and where GRPO fits for reasoning models).
• You can name IPO/KTO/ORPO/GRPO at a concept level and place them on the paired/unpaired and online/offline axes.

Interview Q&A

• "Why does a pretrained LM need alignment at all?" — Next-token loss optimizes likelihood over web text, not helpfulness/harmlessness/honesty. A base model continues prompts plausibly; being a useful assistant is a preference over behavior that the pretraining objective doesn't express. SFT moves it into the right region; preference optimization refines it.
• "Walk me through the three-stage RLHF pipeline." — SFT on demonstrations (also births the reference policy) → train a Bradley-Terry reward model on human comparisons → PPO against that reward with a KL-to-reference penalty. Then explain DPO collapses the last two stages.
• "Write the Bradley-Terry loss and explain its shape." — -log σ(r_w - r_l); flat (→0) for large positive margin, linear (≈ -margin) for large negative, log 2 at zero. Mention the log_sigmoid stability trick.
• "Derive DPO in words." — The KL-constrained reward-maximization has a closed-form optimal policy; solve it for the reward (reward = β log(π/π_ref) + a per-prompt constant); substitute into Bradley-Terry; the constant cancels in the difference, leaving a supervised loss on the policy — no reward model, no rollouts.
• "What does β do in DPO?" — It's the implicit-reward temperature and the KL leash to the reference: large β = sharp preference, tight leash, less drift; small β = looser. Typical 0.1–0.5.
• "DPO or PPO — how do you choose?" — DPO: simpler, cheaper, offline, stable — the default. PPO/GRPO: online exploration, a reusable reward model, and the right call for reasoning RL with verifiable rewards, at the cost of RL infra and tuning.
• "What is reward hacking and how do you prevent it?" — Goodhart on an imperfect proxy reward: the policy exploits the reward model's blind spots (sycophancy, verbosity, gibberish). The KL-to-reference penalty (and β in DPO) keeps the policy where the reward model is trustworthy; pick checkpoints at the gold-reward peak, not the proxy max.
• "What is Constitutional AI / RLAIF?" — Replace human preference labels with an LM judging against a written constitution (self-critique+revise for SFT, AI preference labels for the RL/DPO stage). Scales alignment and spares humans from labeling toxic content; risk is inheriting the judge's biases, so keep human audits.
• "How do you evaluate an aligned model?" — Win-rate vs a baseline (human or LLM-as-judge, controlling for position/length bias), gold-vs-proxy over-optimization curves, safety/red-team evals, and capability-regression checks for the alignment tax.
• "Name some DPO variants and when you'd use them." — IPO (squared loss, curbs over-optimization on near-deterministic prefs), KTO (unpaired good/bad labels), ORPO (preference folded into SFT, no reference model), GRPO (PPO without a value net, group mean as baseline — reasoning RL).

References

• Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (InstructGPT, 2022) — the canonical SFT → reward model → PPO pipeline.
• Schulman et al., Proximal Policy Optimization Algorithms (PPO, 2017) — the clipped surrogate objective.
• Christiano et al., Deep Reinforcement Learning from Human Preferences (2017) — the preference-comparison-to-reward idea that seeds RLHF.
• Rafailov et al., Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (DPO, 2023) — the derivation and the loss in Chapter 7.
• Bai et al., Constitutional AI: Harmlessness from AI Feedback (Anthropic, 2022) and RLAIF — Chapter 6.
• Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO, 2023); Ethayarajh et al., KTO (2024); Hong et al., ORPO (2024); Shao et al., DeepSeekMath / GRPO (2024) — the variants in Chapter 8.
• Gao, Schulman, Hilton, Scaling Laws for Reward Model Overoptimization (2023) — the gold-vs-proxy over-optimization curve.
• Bradley & Terry, Rank Analysis of Incomplete Block Designs (1952) — the original pairwise-comparison model the reward loss comes from.