Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

Yijing Ke, Tianle Cai, Ruosong Wang|October 23, 2026arXiv

Key Takeaway

Training instability in LLM RL often stems from mismatched rounding locations between inference and training—fixing this in just a few critical computation paths can prevent collapse while keeping everything else unchanged.

Summary

This paper identifies how rounding differences between inference and training destabilize reinforcement learning in large language models, even when using identical weights.

training efficiency

Key Terms

bf16-format rope-modulation on-policy-learning rmsn-orm