LoRA's optimization dynamics differ fundamentally between SFT and RL due to baseline subtraction in RL; adapting LoRA's spectrum management to each stage—spreading updates in SFT, rotating subspaces in RL—improves fine-tuning efficiency.
This paper analyzes how LoRA (a parameter-efficient fine-tuning method) behaves differently during supervised fine-tuning versus reinforcement learning. By tracking the mathematical properties of LoRA updates, the authors discover that RL concentrates weight changes along fewer directions while SFT spreads them out, and they trace this difference to how baseline rewards are subtracted in RL.