When training reasoning models with rewards, providing hints during training but correcting for the mismatch helps models learn on hard problems where they'd otherwise get stuck with zero reward signal.
This paper addresses a key challenge in training reasoning models: when problems are too hard, models generate no correct solutions and receive zero learning signal. The authors propose Off-Context GRPO, which uses privileged guidance (like solution hints) during training to generate better learning examples, then corrects the training objective to match the original unguided task.