VLMs encode spatial relationships at the token level but lose this information when aggregating into global representations used for downstream tasks—a failure invisible to standard output-based evaluation.
This paper reveals that Vision-Language Models fail to preserve spatial relationships in their internal representations, even though they perform well on standard benchmarks. The authors introduce EICS, a metric to test whether models consistently encode concepts, and CASP-VL, a benchmark with controlled spatial perturbations.