Pion's orthogonal update mechanism preserves weight matrix spectral properties during training, providing a geometrically principled alternative to gradient-based optimizers like Adam with competitive performance.
Pion is a new optimizer for training large language models that updates weights using orthogonal transformations instead of adding gradients like Adam does. By preserving the singular values of weight matrices, it keeps the spectral properties stable while still allowing the model to learn, offering a more geometrically-grounded approach to optimization.