Restricting fine-tuning updates to the bottom singular subspace of weight matrices significantly reduces catastrophic forgetting while maintaining strong performance on new tasks—a practical technique that works with any optimizer and existing fine-tuning methods.
This paper tackles catastrophic forgetting in large language models during fine-tuning by introducing Subspace Tuning (SST), a method that restricts weight updates to less important singular directions. By protecting the principal directions of weight matrices, SST maintains existing capabilities while learning new tasks, achieving up to 99% subspace protection with minimal performance trade-offs.