Relaxing weight-norm constraints to allow controlled growth (proportional to √t) improves optimizer efficiency and model convergence speed compared to strict norm-fixing approaches, with practical gains of ~15% on large-scale pretraining.
This paper analyzes how weight-norm control affects optimizer performance during language model pretraining and proposes MuonGS, a new optimizer variant. The key insight is that controlling how fast weight norms grow—rather than fixing them—allows better exploration during training.