Overview
Imagine drafting a proof by filling blanks across the page. You could start with the key equation, a clue from the question, or the last line. In a masked diffusion language model, the first place to write can steer the entire answer. Moontae and his colleagues find that confidence-based decoding often starts with an end marker near the far edge of the output window. Nearby positions then become easier to fill with more end markers, shrinking the space left for an explanation.
Key Findings
A blank page fills from the edges
A left-to-right language model begins a calculation at the first word and keeps moving. A masked diffusion language model starts with a row of blanks. It can fill some near the end first, go back for a missing phrase, or fill several at a time. Every round gives it two choices. It has to decide which blanks to reveal and which tokens to put there. Those are different decisions, and the paper asks which one can shape the rest of the answer.
The authors found that selecting blanks by model confidence often starts at an edge and then favors positions close to tokens already revealed. In the paper's Figure 1, early high-confidence end-of-sequence tokens occupy slots that could have carried content. A revealed token stays in place in the masked diffusion setup studied here. The model can keep working through its denoising rounds while much of the answer canvas is already taken up by endings. Figure 3 tracks both effects. End markers account for more than half of the revealed tokens during most steps in the illustrated setting.
One early choice gives the experiment two different futures
The authors isolate where randomness enters generation. In one condition, they sample the token identity throughout denoising using a temperature of 0.9. In another, they sample the position uniformly on the first step and choose the token greedily. The two policies use the same inference budget. Figure 4 compares pass-at-, the chance that at least one of attempts solves a problem. On GSM8K, randomizing only the first position beats sampling token identities at all steps across the attempt budgets shown.
The next test asks whether a good opening stays useful even when later decisions change. The researchers reuse the first four denoising steps from trajectories that ended correctly or incorrectly and introduce randomness in the continuations. Figure 5 shows higher accuracy from the prefixes taken from successful trajectories, under either continuation scheme. The difference was 50.2% versus 34.4% with position sampling, and 69.1% versus 36.1% with temperature sampling. These are conditional comparisons for this experiment. They show the importance of an opening, while leaving room for what happens next.
A small planner makes the first move and an end marker waits
The first intervention scores candidate sets of positions using the frozen language model's final-layer hidden representations at those positions. The authors generate training examples by varying the initial reveal and completing each resulting trajectory greedily. A lightweight, five-million-parameter planner learns from task-level outcomes. At inference, it chooses among 32 candidate sets for the first step. Later positions go back to confidence-based selection. A progressive schedule reveals fewer slots early so the first move commits less of the answer at once.
The second intervention adjusts the end marker for position selection. Appendix D.4 defines the temperature schedule below, with denoting the denoising step and the number of steps.
The algorithm divides the end-token logit by this temperature before calculating confidence for the reveal decision. After choosing a position, it uses the original logits to predict the token. The schedule starts near three and returns to one at the final step. This changes where the model writes under early uncertainty while retaining the original token scores for filling the chosen positions.
More room to reason helps some tasks more than others
Table 1 compares LLaDA 8B Instruct at 32 denoising steps. GSM8K and MATH reward a correct final answer. Countdown asks the model to build an arithmetic expression, and Sudoku is measured by cell accuracy under the paper's protocol.
| Position and token strategy, Table 1 | GSM8K | MATH | Countdown | Sudoku | Average |
|---|---|---|---|---|---|
| Top-1 probability | 46.6 | 19.2 | 42.2 | 71.2 | 44.8 |
| Top-1 + learned planner | 55.0 | 22.4 | 44.1 | 65.2 | 46.7 |
| Top-1 + EOS annealing | 50.9 | 22.4 | 46.1 | 63.6 | 45.7 |
| Top-1 + both | 56.8 | 22.8 | 43.8 | 67.0 | 47.6 |
| Probability margin + both | 58.6 | 23.0 | 45.3 | 69.5 | 49.1 |
| Random initial position | 52.6 | 17.6 | 30.1 | 48.3 | 37.1 |
A uniformly random first choice helps the grade-school arithmetic benchmark, yet it does badly on Sudoku. The authors suggest that Sudoku's worked example teaches a rigid order that random reveals can disrupt. The planner recovers much of that loss, and the combined interventions deliver the highest average in the table. They still give up some Sudoku accuracy relative to probability-margin decoding on its own. On GSM8K, the top-1 comparison reports 157.2 non-end tokens for the greedy baseline and 188.6 when both interventions are used.
Table 2 tests more denoising steps. The mean for the combined approach rises to 50.2 at 64 steps and 52.8 at 128. The ordinary top-1 baseline falls from 39.7 to 37.7 over those same budgets. The paper also evaluates Dream 7B Instruct. Taken together, the experiments suggest that a diffusion model can spend its parallelism badly. Additional rounds are useful only if the decoder has left itself a productive problem to finish.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Why does writing the end marker matter if there are still blanks?
A: An end marker can become a strong local clue for the remaining predictions. In our experiments, nearby positions often acquire confidence next. Repeated end tokens can occupy a large share of the fixed window before substantive content appears.
Q: Do the two interventions change the same decision?
A: The planner chooses which positions to reveal at the first step. Annealing changes how strongly the end marker contributes to position ranking over time. Once a position is selected, the original model logits still determine its token.
Q: Will a longer denoising run always help?
A: In the fixed-window experiments, plain confidence decoding often lost accuracy as it took more, smaller steps. The guided approach improved the reported averages at 64 and 128 steps, too. The effect depends on how the policy allocates those steps and how strongly the task favors a particular order.
Q: Would the first-step planner be worth training for a new format?
A: The planner is trained on task-specific outcomes. The Sudoku regression shows why testing a fixed positional rule is worthwhile when the answer has strict structural dependencies. A new format calls for its own validity checks, and a deployment comparison should count the offline rollouts needed to collect labels as well as inference latency.
Future Ideas
Let a constraint checker choose where a structured answer begins
Try the planner on structured runbooks with required fields and dependency rules. Create offline labels by completing multiple initial position choices and checking each final output against the schema and a set of executable dependency rules. Let the planner prefer initial positions that lead to valid runbooks, such as prerequisites or units that later fields rely on.
Compare the learned planner with ordinary confidence decoding, a fixed prerequisite-first order, and end-token annealing alone. Hold the backbone and generation budget fixed. Split the evaluation by runbook template, track exact constraint satisfaction and invalid truncation, and measure end-to-end latency on realistic batches. Keep the learned component only if it improves held-out validity over the fixed rule at comparable cost.
Count the cost of building and refreshing the labeled trajectory set. State how many uses would be needed to amortize that cost, and compare the stricter templates with a fixed structural order before considering a learned rule.
Sources
Original abstract
Diffusion-based language models (dLLMs) have emerged as a promising alternative to autoregressive language models, offering the potential for parallel token generation and bidirectional context modeling. However, harnessing this flexibility for fully non-autoregressive decoding remains an open question, particularly for reasoning and planning tasks. In this work, we investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis. Specifically, we uncover an inherent failure mode in confidence-based non-autoregressive generation stemming from a strong proximity bias-the tendency for the denoising order to concentrate on spatially adjacent tokens. This local dependency leads to spatial error propagation, rendering the entire trajectory critically contingent on the initial unmasking position. Leveraging this insight, we present a minimal-intervention approach that guides early token selection, employing a lightweight planner and end-of-sequence temperature annealing. We thoroughly evaluate our method on various reasoning and planning tasks and observe substantial overall improvement over existing heuristic baselines without significant computational overhead.