You can reuse RL training from cheaper small models to improve large models by treating the policy shift (not the final policy) as a dense reward signal—this cuts post-training costs while maintaining reasoning gains across model scales.
This paper proposes Direct-OPD, a method to transfer reinforcement learning gains from smaller models to larger ones without expensive retraining. Instead of distilling the final policy, it extracts the policy shift that RL induced (via log-ratio comparison) and applies it as an implicit reward signal on the stronger model's own data, enabling efficient scaling of RL-based reasoning improvements.