Combining trajectory-level intervention (teacher corrections inserted into rollouts) with token-level supervision (learning from corrected segments) is more effective than either approach alone for training on hard reasoning tasks.
This paper addresses a key limitation in reinforcement learning from verifiable rewards (RLVR): when all attempted solutions fail, the model gets no learning signal. PIVOT combines two strategies—inserting teacher corrections into student attempts and using supervised learning on those corrections—to help models learn from harder problems where pure reward signals are too sparse.