Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary|November 3, 2026arXiv

Key Takeaway

DNO can substantially improve open-weight language models through iterative preference-based fine-tuning, and the expensive rating step can use cheaper open-weight models without losing most of the performance gains.

Summary

This paper studies Direct Nash Optimization (DNO), a method that improves language models by treating alignment as a two-player preference game. The authors implement and test DNO on Llama and Mistral models, showing significant improvements on AlpacaEval 2.0 benchmarks.

alignment training evaluation

Key Terms

direct-preference-optimization nash-equilibrium on-policy-sampling preference-learning