Muon can now be used effectively with LoRA-style low-rank fine-tuning by approximating its solution mathematically, offering a practical way to combine two previously incompatible techniques.
This paper solves a mathematical incompatibility between Muon (an optimizer that works well for pretraining) and LoRA (a popular parameter-efficient fine-tuning method). The authors propose sMuon, which approximates Muon's benefits in the low-rank setting through linearization and least-squares, using only standard matrix operations.