Reinforcement learning has become a standard way to sharpen a language model's reasoning, but it doesn't port easily to diffusion language models, which write text by revealing masked positions bit by bit rather than left to right. The snag is that the quantity RL relies on, the probability of a given output, is hard to compute here, so earlier attempts approximated it by looking only at which tokens got placed.

The authors note that generation really involves two choices at every step: what token to drop into a slot, and which slots to reveal or re-hide next. Picture filling a crossword where you pick both the letter and which square to open, while prior methods graded only the letters. By writing the objective to account for both choices, they improve results on math and coding, reporting 87.1% on GSM8K and 53.4% on MBPP.

These numbers come from the abstract, so consult the paper for the derivation and setup.