The Kronecker Delta Rule offers a theoretically grounded way to update associative memories in attention models that's both stable and efficient for long-context tasks, outperforming existing linear attention methods.
KronLSTM introduces a new memory update mechanism for linear attention models that modifies memories along a single direction, reducing interference with stored information. This approach provides mathematically stable memory dynamics without extra normalization, making it effective for processing long documents and sequences.