In hybrid architectures mixing recurrence and attention, recurrence dominates learning of retrieval functions. Staged training that temporarily freezes recurrence allows attention to develop, significantly improving model performance.
This paper identifies a 'Competing Learning' problem in hybrid language models that combine recurrent and attention layers: recurrence learns retrieval functions faster, blocking attention from learning. The authors propose staged training—freezing recurrence updates mid-training—to let attention catch up, improving performance on language modeling and retrieval tasks.