RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

Chenglong Wang, Ziming Zhu, Yifu Huo, Bei Li, Qiaozhi He et al.|August 6, 2026arXiv

Key Takeaway

Generative reward models work better in RL when you extract rewards from their ranking capabilities rather than forcing them to produce scalar scores—use pairwise or reference-based comparisons instead.

Summary

This paper shows how to use generative reward models (which rank responses) effectively in reinforcement learning for language models. The key insight is that generative models naturally compare responses rather than score them individually.

training reasoning

Key Terms

generative-reward-model ranking-based-reward-construction self-competitive-ranking anchor-guided-ranking