Transformer architecture design matters as much as training objectives for diffusion-based language models—using architectures optimized for CDLMs rather than borrowed from LLMs or image diffusion models substantially improves generation quality.
This paper redesigns Transformer architectures specifically for continuous diffusion language models (CDLMs), which generate text by gradually refining noise into coherent sequences.