Visual evidence in bug reports is largely ignored or unhelpful for current AI systems—the benchmark reveals that adding images doesn't improve localization performance, suggesting systems need better multimodal reasoning to leverage visual clues effectively.
MM-IssueLoc is a benchmark for evaluating how well AI systems locate bugs in code repositories using both text and visual evidence like screenshots and error dialogs. It contains 652 real issues across 23 programming languages with structured annotations, and tests whether systems actually benefit from images or just rely on text.