You do not need reinforcement learning for preference alignment — DPO achieves equivalent results with a simple binary cross-entropy loss, dramatically simplifying the alignment pipeline.
Shows that the RLHF objective can be optimized directly without training a separate reward model or using RL. DPO reparameterizes the reward function as a simple classification loss on preference pairs, making alignment simpler and more stable.