Pretrained models already know which tokens matter for the next prediction—you can train a lightweight module to detect this and skip expensive full-context reads, cutting inference cost without retraining the base model.
This paper shows that language models can predict when they need to read their full context history during generation. The authors introduce On-Demand Attention (ODA), which uses a small trained module to decide when to use expensive global attention versus cheaper local attention, reducing computation while maintaining quality on long documents.