By incorporating variance information into how segments are aggregated in linear attention models, MERLIN enables better long-context understanding without the quadratic complexity of standard attention—useful for scaling LLMs to longer sequences.
This paper improves Multi-State Linear Attention (MSLA) for long-context language models by tracking both the mean and variance of key alignments across segments, rather than just the mean.