When training models with verifiable rewards, be careful that your advantage estimator doesn't accidentally reward guessing—use composition-free magnitude calculations to stay grounded in actual reasoning quality.
GRPO is a popular method for training AI models using verifiable rewards, but it has a hidden flaw: it can reward models for guessing the right answer instead of reasoning through it. This paper identifies three scenarios where this happens and proposes SIGNBALANCE, a fix that uses simpler reward calculations to distinguish genuine reasoning from lucky guesses.