Organizing tasks into separate optimization paths with independent adapters outperforms using a single shared adapter—even with the same total parameters—because it reduces task interference and preserves positive transfer between compatible tasks.
This paper addresses a key problem in fine-tuning large language models: when you try to adapt a single shared adapter to many different tasks, they interfere with each other, hurting performance. The authors propose automatically organizing tasks into groups and sequences, then training separate adapters for compatible tasks instead of forcing everything through one shared adapter.