Continuum Attention replaces discrete token lookup with continuous integration over memory, improving language modeling performance without increasing computational cost—suggesting continuous functions may be a better foundation for attention than discrete token operations.
This paper introduces Continuum Attention, a new attention mechanism for Transformers that treats memory as a continuous curve instead of discrete tokens. Rather than looking up individual key-value pairs, the mechanism integrates over a smooth memory representation weighted by exponential relevance.