Vision-language agents struggle dramatically with real-time interaction: GPT-4o's performance drops from 75% to 9% when environments run in real time instead of paused, exposing a major limitation in current models that excel at deliberative reasoning.
This paper introduces Gym-Anything-RT, a benchmark of 100 auto-generated minigames designed to test vision-language agents' ability to interact with environments in real time. The key insight is that real-time interaction—where the environment state changes while the agent reasons—is fundamentally different from deliberative tasks.