Dynamic, state-aware adapters that track information across tokens and retrieve relevant context segments significantly outperform static low-rank adaptation for reasoning tasks, without retraining the base model.
This paper proposes two new adapter methods that improve how language models adapt to reasoning tasks. MaLoRA adds dynamic, input-dependent scaling to low-rank adapters using recurrent state at the token level, while MaRA selects relevant context segments before answering. Together, these methods boost reasoning accuracy by up to 18% over standard LoRA across multiple models and benchmarks.