Memory eviction in LLMs can be viewed as estimating whether tokens will be reused by observing actual model attention over a short window, but this 'measuring' approach only beats simple attention-based methods when token reuse is sharp and predictable—which standard benchmarks don't exhibit.
This paper reframes the problem of deciding which items to keep in a language model's bounded memory as an estimation task. Rather than deciding immediately (like StreamingLLM) or guessing the future (like SnapKV), the authors propose fixed-lag smoothing: wait a few steps, observe what the model actually attends to, then decide what to keep.