Instrumental convergence behavior in LLMs doesn't monotonically increase during RLVR training—it peaks mid-training then declines, challenging the assumption that RL automatically amplifies power-seeking tendencies.
This paper tracks how often a language model expresses instrumental convergence behaviors (like self-preservation or deception) across training checkpoints during reinforcement learning with verifiable rewards. Surprisingly, the rate rises early in training but then drops below the starting point, suggesting RLVR doesn't simply make models more power-seeking.