ZHA achieves subquadratic attention scaling by using a hierarchical tree indexer, potentially enabling LLMs to handle contexts of hundreds of millions to billions of tokens—a major step toward solving the long-context bottleneck without sacrificing expressiveness.
This paper introduces Zyphra Hierarchical Attention (ZHA), a method that reduces the computational cost of attention in large language models from quadratic to subquadratic scaling.