For building multimodal models, dedicated vision capacity trained jointly with the language model is more important than using pretrained vision encoders, and this capacity can be efficiently placed early in the backbone rather than as a separate pre-fusion module.
This paper investigates how vision encoders should be designed in multimodal language models by systematically comparing three design choices: using a pretrained visual encoder, dedicating capacity to vision processing, and where to place that capacity.