Language model simulations can predict whether AI safeguards will work in deployment more accurately and efficiently than separate control environments, suggesting you can test safety without building expensive real-world test setups.
This paper introduces simulated control evaluations (SCEs), which use language models to simulate tool responses instead of real environments to test whether AI safeguards work. Testing on Linux command execution, SCEs predict safety outcomes more accurately than traditional control settings, using far fewer resources while closely matching real-world attack patterns and reasoning.