Multimodal models show systematic biases in how they weight information across modalities—decision models are more robust to presentation changes than base models, but accuracy metrics alone miss important shifts in confidence that could affect real-world reliability.
This paper introduces MM-SystemOneBias, a benchmark for evaluating how multimodal AI models make decisions when presented with variations like different option orders, irrelevant information, or conflicting descriptions.