Self-conditioning and careful embedding geometry are critical design choices for continuous diffusion language models; simple architectural improvements like DoubleCat can match or exceed discrete diffusion performance without additional computational cost.
This paper investigates how to design better continuous diffusion language models by studying two key factors: self-conditioning (exposing models to realistic inference states during training) and embedding geometry (how word representations are shaped).