Phase 07 — Alignment: RLHF, PPO & DPO

The phase that turns a fluent mimic into a usable assistant. A pretrained model aced one objective — predict the next token of the web — which is emphatically not "be helpful, harmless, and honest." Alignment is how we install behavior the pretraining loss never expressed: from supervised demonstrations, to a reward model built from human comparisons, to the optimization step (PPO) that chases that reward on a leash — and finally to DPO, the insight that you can optimize the preference directly, with no reward model and no reinforcement learning at all. Every modern chat model you've used went through some version of this pipeline.

Why this phase exists

The gap between "a model that knows things" and "a model you'd ship" is alignment. Pretraining gives you knowledge and fluency; it gives you nothing about which of the many plausible continuations is the one a human actually wanted. Closing that gap is its own discipline, and it rests on a small set of ideas a Senior AI Engineer must hold cold:

1. The alignment problem — helpful/harmless/honest are preferences over behavior, not a next-token loss. You need a different signal: human (or AI) comparisons.
2. The Bradley-Terry reward model — convert "A is better than B" into a scalar reward via -log σ(r_chosen − r_rejected). The whole field's preference loss, and the one you must be able to write on a whiteboard.
3. PPO with a KL leash — the clipped surrogate that bounds each update, plus the KL-to-reference penalty that is load-bearing: without it the policy reward-hacks the imperfect reward model into gibberish.
4. DPO — the optimal RLHF policy has a closed form, so the reward is implicit in the policy's own log-ratio to the reference. Substitute it into Bradley-Terry and the reward model and the rollouts both vanish: preference optimization as one supervised loss. The default for most teams today.
5. Variants & evaluation — RLAIF/Constitutional AI (AI feedback at scale), the variant zoo (IPO, KTO, ORPO, GRPO), and how you actually measure alignment without over-optimizing the proxy.

Get fluent here and you can read any alignment paper, defend a preference-tuning design, and debug the moment a model starts agreeing with everything you say.

Concept map

              ┌────────────────────────────────────────────────────┐
              │  Base LM aced next-token; it is not yet an ASSISTANT │
              └────────────────────────────────────────────────────┘
                                     │
                    ┌──── SFT (demonstrations) ────┐
                    ▼                               ▼
            helpful-ish policy            REFERENCE policy π_ref (frozen anchor)
                    │                               │  (both PPO & DPO leash to it)
                    ▼                               │
        ┌───────────────────────┐                  │
        │ PREFERENCE PAIRS (A>B) │ ◀── humans / AI (RLAIF, Constitutional AI)
        └───────────────────────┘                  │
            │                  │                    │
   ┌────────┘                  └──────────┐         │
   ▼                                      ▼         │
 REWARD MODEL (Bradley-Terry)          DPO ─────────┘
 -log σ(r_w - r_l)                     -log σ(β·[Δlogπ_w - Δlogπ_l])
   │                                   (implicit reward = β·log π/π_ref)
   ▼                                      │  no reward model, no rollouts
 PPO  L_clip + β_KL·KL(π‖π_ref) ──────────┤
   │  clip bounds the step;               │
   │  KL term stops REWARD HACKING        │
   └──────────────┬──────────────────────┘
                  ▼
         ALIGNED POLICY  →  eval: win-rate, gold-vs-proxy, safety, alignment tax

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Defend "DPO, not PPO" (or the reverse). Given a team with no RL infra and a static preference dataset, argue for DPO with the simplicity/cost/stability tradeoff; then flip it — a reasoning task with a verifiable reward — and argue for GRPO and online exploration. Same decision, opposite winners.
• Diagnose a reward-hacked model. A model post-RLHF has gotten sycophantic and verbose while benchmark scores barely moved. Explain it as proxy over-optimization, point at the KL budget, and propose the gold-vs-proxy checkpoint-selection fix.
• Tune β. Show what a too-large vs too-small DPO β does to drift and learning, and how you'd pick it from a win-rate-vs-β sweep.
• Design an RLAIF loop. Replace human labels with a constitution + an LM judge; name the bias risk and the human audits you keep.
• Choose unpaired over paired data. You have a flood of thumbs-up/thumbs-down product signal but few clean A-vs-B comparisons. Argue for KTO (unpaired) over DPO (paired), and what you'd give up.
• Pick the right checkpoint. Given a gold-vs-proxy training curve, point to the checkpoint you ship and explain why it's the gold peak, not the last (highest-proxy) step — the over-optimization story in one decision.

How this fits the track

Phase 00 taught you that inference cost is paid forever, which is why you ship the smallest model that clears the bar. This phase is how you make a small model clear the bar on behavior — alignment is what turns raw capability into a shippable assistant. It leans on the fine-tuning machinery from earlier phases (SFT is ordinary fine-tuning; the reference policy is the SFT checkpoint) and feeds directly into the next one: an aligned policy still has to decode tokens, and Phase 08 is where the sampler that turns the aligned logits into text lives. Alignment shapes the distribution; decoding chooses from it.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can write the Bradley-Terry loss and the DPO loss from memory and explain how the latter is the former with an implicit, reference-relative reward.
• You can derive DPO in words (closed-form optimal policy → solve for reward → substitute → the partition function cancels).
• You can read the PPO clipped objective and say where it stops giving credit for A > 0 and A < 0, and why the min is pessimistic.
• You can state why the KL-to-reference penalty is load-bearing (reward hacking) and what β does in DPO.
• You can place IPO/KTO/ORPO/GRPO on the paired/unpaired and online/offline axes.

Key takeaways

• Alignment installs behavior, not knowledge. Pretraining learned the facts; SFT + preference optimization teach the model which response a human wanted. Helpful/harmless/ honest are preferences over behavior that the next-token loss can't express.
• The reward signal is a comparison, and Bradley-Terry turns it into a gradient. -log σ(r_w − r_l) is the loss the entire field is built on — and it's the same loss DPO uses, with the reward made implicit.
• The KL-to-reference penalty is what makes RLHF work. The reward model is a finite, hackable proxy; the KL leash (and DPO's β) keeps the policy where the proxy is trustworthy. Remove it and you get reward-hacked nonsense.
• DPO collapsed a whole RL system into one supervised loss and became the default — but online RL (PPO/GRPO) is back for reasoning models, because sometimes you want exploration against a verifiable reward. "Best method" is, as always, "best for these constraints."

Next: Phase 08 — Sampling, Decoding & Constrained Generation.