Frontier multimodal agents show strong zero-shot robot control with good failure recovery, but action-level analysis reveals they still struggle with precise physical execution tasks—a critical insight for building more reliable robotic systems.
This paper evaluates how frontier AI agents (like GPT-4o) control robots by analyzing individual actions rather than just overall success rates. The authors introduce AgentRoboCritic, a framework where agents explain their intent before each action and a critic evaluates whether the action succeeded or failed.