Attention mechanisms aren't arbitrary design choices—they're constrained by symmetry principles from physics.
This paper reveals that attention mechanisms in neural networks follow the mathematical structure of gauge theory when entities can rotate independently. The authors show that relations between entities can only interact through closed loops (Wilson loops), leading to a unique form of attention called holonomy attention.