Gradually expanding memory capacity during sequence processing—rather than using static memory—is a simple, cost-free improvement that works across different memory-based architectures and shows larger gains on longer contexts.
This paper introduces Proteus, a technique that improves how neural networks handle long text by gradually expanding memory capacity as more context arrives.