Injecting reasoning prefixes from weaker models during RLVR training preserves exploration diversity and improves pass@k performance without extra training or complex reward engineering.
This paper addresses a key problem in reinforcement learning for language models: when training LLMs to solve reasoning tasks with verifiable rewards, the models become overconfident and stop exploring diverse solution paths.