Visual tool-calling agents fail primarily due to poor image understanding, not planning: 53% of failures come from misreading images despite correct task logic. This suggests different improvement strategies are needed for smaller vs. larger models.
MM-ToolSandBox is a benchmark for evaluating AI agents that use images to call tools and complete tasks. It includes 500+ tools across 16 domains, 258 test scenarios with images, and a stateful execution environment. Testing 12 models reveals that even the best achieve below 50% success—with visual perception (extracting correct info from images) being the main bottleneck, not task planning.