Off-policy environment difficulty scaling beats on-policy co-evolution: by pre-generating harder environments in batches rather than reacting to live rollouts, you can train stronger agents more efficiently and avoid the bottleneck of waiting for models to fail.
This paper proposes environment evolution, a method to automatically create increasingly difficult training environments for AI agents without relying on real-time model feedback.