Features in Vision Transformers don't stay in one layer—they migrate between layers during training, especially early on, revealing a previously invisible dimension of how these models learn.
This paper tracks how individual features move and change across Vision Transformer layers during training using Sparse Autoencoders. By visualizing features over both network depth and training time, researchers discovered that features migrate between layers early in training, with movement favoring earlier layers, and that deeper layers stabilize faster than shallow ones.