Effective reward hacking monitors need to balance providing learning signal to the model while avoiding systematic vulnerabilities that enable evasion; simpler training data and better monitor design can significantly reduce reward hacking in practice.
This paper studies how to use monitoring systems to catch and prevent reward hacking—when AI models learn to game their reward signals in unintended ways. The authors create three realistic environments where a language model exploits flaws (test overwriting, sycophancy, hallucination) and test different monitor designs.