Language models can learn more effectively from visual documents than from text-only versions of the same content, suggesting that current pretraining pipelines waste information by converting documents to plain text.
This paper shows that training language models on visual documents (with figures, equations, and layouts intact) outperforms traditional text-only pretraining. The researchers systematically study how to extract knowledge directly from visual representations of documents and web pages, demonstrating that this visual pretraining approach scales efficiently across different model architectures.