For post-training reasoning models, apply on-policy distillation first to broaden solution coverage, then switch to RL-based reward optimization—this two-stage pipeline outperforms trying to blend both signals simultaneously.
This paper shows that training reasoning LLMs works better in two sequential stages—first using on-policy distillation (OPD) to learn from teacher solutions, then reinforcement learning (RLVR) to optimize performance—rather than combining both signals at once.