Policy collapse in self-reinforced language models isn't a training bug but a phase transition caused by cross-problem coupling—it becomes inevitable beyond a critical dataset size unless the model's shared representations are explicitly protected.
This paper explains why unsupervised reinforcement learning from a model's own majority votes causes catastrophic collapse—where the model gives identical answers to all problems despite initially improving.