Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen et al.|September 1, 2026arXiv

Key Takeaway

You don't need to find the perfect SFT-RL budget split—a wide range of allocations work nearly equally well, and you can identify this range using small models and apply it to large ones, saving computation.

Summary

This paper solves a practical problem in LLM training: how to split your annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL). Instead of finding one perfect ratio, the authors identify a 'near-optimal region'—a range of budget splits that all perform nearly as well.

training efficiency scaling

Key Terms

supervised-fine-tuning reinforcement-learning-from-human-feedback annotation-budget proxy-model near-optimal-region