Linear attention models can be made significantly more efficient by allocating parameters based on sensitivity analysis: use low-rank gates for state erasure (which matters less) and invest savings into wider networks and better readout mechanisms.
Muir improves linear attention models by analyzing how state updates work differently across channels. The key insight is that erasing old information requires less capacity than writing new information, so the model uses low-rank gates for erasure and reinvests those parameters into a wider network.