The transformer architecture transfers directly to vision — splitting images into patch tokens works as well as convolutions, unifying the architecture across modalities.
Applies a standard transformer directly to sequences of image patches, with no convolutions. When pre-trained on large datasets, the Vision Transformer (ViT) matches or exceeds CNNs on image classification while being more scalable.