Task vectors can be compressed to 1-5% of their original size while maintaining model performance, making it practical to store and dynamically merge multiple task-specific models without prohibitive storage costs.
This paper tackles the storage overhead problem in dynamic model merging by compressing task vectors (fine-tuned weight changes) using learnable compression techniques.