MDLMs make two independent decisions per step (token choice and mask position), and optimizing both separately via policy gradients outperforms methods that only optimize token predictions.
This paper shows how to apply reinforcement learning to Masked Diffusion Language Models (MDLMs) by properly accounting for both token selection and position unmasking decisions. The authors decompose the policy gradient into two components—what to generate and where to generate it—achieving strong results on math and coding tasks without needing intractable log-likelihood calculations.