Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

A new paper argues that DPO matches RLHF only under a narrow assumption: the RLHF-optimal policy must already prefer human-approved responses.

When that assumption fails, DPO can optimize a different objective and converge to undesirable behavior, exposing important failure modes for alignment.

The authors reinterpret DPO as a soft margin ranking method that may use negative targets, helping explain why the equivalence can break.

They propose Constrained Preference Optimization (CPO), a simpler method that adds constraints to preserve alignment guarantees while staying practical.