By constraining fine-tuning to low-importance singular directions of pre-trained weights, TailLoR achieves better continual learning with fewer parameters than standard approaches.
TailLoR is a parameter-efficient method for continual learning that protects important features in pre-trained models by learning updates only in the 'long-tail' singular directions while keeping dominant directions fixed. This reduces catastrophic forgetting when learning new tasks sequentially.