You can make LoRA fine-tuning significantly faster and cheaper by only updating weight matrices with high condition numbers—the ones that actually need rebalancing—rather than updating all matrices uniformly.
This paper shows that not all LoRA weight matrices contribute equally to fine-tuning, and proposes κ-LoRA, which selectively updates only the matrices with high condition numbers (those with imbalanced singular values). By updating just the top 50% of matrices ranked by condition number, the method cuts fine-tuning time by 16% while maintaining accuracy and reducing memory use by 4.5%.