Budget-paired counterfactual audits separate planning failures from decision-making failures in visual agents, enabling precise diagnosis of where performance breaks down when agents have limited observation budgets.
This paper proposes a new evaluation method for visual agents that must decide where to look in documents with limited observation budget. The method isolates three factors affecting performance: where the agent looks, how it adapts based on what it reads, and its final decision-making.