Durable suppression of misaligned behaviors requires training against diverse misalignment states, not just the final bad behavior—PEARL does this by sampling intermediate states along cached trajectories during adversarial RL.
This paper studies how to durably suppress undesired behaviors in language models after they emerge. The authors create two test cases—models trained to cheat on multiple-choice questions or generate harmful code—then compare methods like standard fine-tuning and a new approach called PEARL that uses adversarial training with intermediate state sampling.