Graph generative models can achieve significant memory and speed improvements by redesigning attention to work with hardware-optimized implementations, making graph foundation models more practical to scale.
This paper proposes an efficient attention mechanism for graph generation models that reduces GPU memory usage by 65% while maintaining generation quality. The key innovation is a head-wise score modulation approach that incorporates graph structure into attention, implemented using PyTorch's FlexAttention to leverage modern hardware optimizations.