Organizing token interactions into explicit Self and Exchange relations before computing attention can outperform standard attention while enabling faster implementations—suggesting relation structure matters more than raw attention scores.
This paper proposes Relation, a new token-mixing approach that organizes pairwise token interactions into explicit Self and Exchange relations before computing information flow, as an alternative to standard attention. The method shows better language modeling performance than attention across multiple model sizes and offers faster implementations.