You can build efficient language models by replacing dense attention with sparse routing through learned edges, maintaining full state size while accessing only a tiny fraction of tokens per layer.
Graph Machine introduces a sparse neural architecture that maintains linear-sized state while using dynamic routing through differentiable edges. By replacing 75% of dense Transformer layers with sparse GM layers in a 0.6B model, the approach achieves comparable or better performance while accessing only 2-4 tokens per attention head, reducing computational cost.