Overview
Ask a language model to finish a movie review on a positive note, and a sentiment classifier can reward it for piling on praise. The model can score higher while producing unfamiliar or repetitive text. The early DfPO manuscript tests a way to use the original language model as a guide during reinforcement learning. It samples two groups of tokens and adjusts each group in a different direction.
Key Findings
The authors list DfPO at the ICLR 2024 workshop on Generative Models for Decision Making. The figures, equations, and results here come from the supplied ICLR 2024 submission manuscript, titled DfPO: Degeneration-free Policy Optimization via Action Masking in Natural Language Action Spaces. The later ICML proceedings paper revises the update and adds experiments. Author’s publication record
The fluency penalty creates a tuning problem
In the IMDb task, the model continues a partial movie review. A DistilBERT sentiment classifier supplies the reward. Perplexity tracks how predictable the generated text is and serves as a proxy for naturalness. In Figure 1, the PPO run with no KL penalty quickly drives up sentiment, and its perplexity rises beyond the chart’s range. Several penalty settings keep perplexity close to the baseline while producing smaller sentiment gains. The balance depends on both the KL coefficient and the target KL.
Two rollout policies divide the work
For a token after prefix , the manuscript compares its probabilities under the reference model and the current policy .
The positive mask samples among tokens the current policy ranks lower than the reference does (). It retains positive task advantages and makes those sampled tokens more likely. The negative mask samples among tokens ranked higher than by the reference (). It retains negative task advantages and makes those sampled tokens less likely. Equations 5–6 renormalize the current distribution separately within each mask. Equation 7 adds the two policy-gradient terms:
Here and are the task advantages estimated under the respective rollouts, and , . In Figure 2, “definitely” is eligible for promotion if its task advantage is positive and the current policy underweights it relative to the reference. “Not” is eligible for suppression if its task advantage is negative and the current policy overweights it. The colored example shows possible update directions for one prefix.
The IMDb trade-off depends on the starting model
The table selects the competing reinforcement learning runs from Table 2. Reported uncertainties are standard errors over five runs. Cold runs start from a pretrained model, while warm runs start from a supervised checkpoint. The corresponding initial-policy perplexities are 32.171 and 35.472.
| Starting model | Method | Sentiment ↑ | Perplexity ↓ |
|---|---|---|---|
| Cold | PPO | 0.602 ± 0.012 | 33.816 ± 0.233 |
| Cold | NLPO | 0.611 ± 0.023 | 33.832 ± 0.283 |
| Cold | DfPO | 0.621 ± 0.008 | 32.531 ± 0.099 |
| Warm | PPO | 0.626 ± 0.014 | 35.049 ± 0.347 |
| Warm | NLPO | 0.620 ± 0.014 | 34.816 ± 0.340 |
| Warm | DfPO | 0.662 ± 0.006 | 35.791 ± 0.071 |
Cold-start DfPO has the highest mean sentiment and lowest perplexity among its RL comparators. With the supervised start, DfPO still has the highest sentiment but also has the highest final perplexity of the three RL methods. Figure 3 shows that its perplexity stays close to its own starting point. Table 3 reports gains over both PPO and NLPO on all seven IMDb diversity measures in both initialization settings. Those diagnostics describe different properties of generated text.
The appendix repeats the IMDb experiment with the 6-billion-parameter GPT-J as both the initial and reference model. Sentiment rises while perplexity stays near its starting level. Figure 6 plots only DfPO, so it shows how one larger model responds to the update. A comparison at that scale would require PPO and NLPO runs on the same checkpoint.
The ablation separates training from decoding
Figure 5 compares the full warm-start method with either update term alone and with a naive policy gradient. The naive update raises sentiment as perplexity escapes the plotted range. Maximization alone gives a smaller sentiment gain than the full update, while minimization alone performs worse on sentiment. At inference, this manuscript decodes with the negative mask. Figure 9 holds the training procedure fixed and compares decoding policies. The negative mask gives the strongest sentiment improvement among the three decoding choices shown.
CommonGen gains depend on the metric
CommonGen asks a T5 model to describe a plausible scene using given concepts. DfPO optimizes METEOR and leads on that metric in Table 4. PPO has the highest CIDEr mean, and NLPO leads on SPICE. Table 5 also gives DfPO a lower mean segmental type-token ratio (MSTTR, a vocabulary-diversity measure) than the other supervised-start methods.
| Method | METEOR ↑ | CIDEr ↑ | SPICE ↑ | MSTTR |
|---|---|---|---|---|
| Supervised checkpoint | 0.309 | 1.381 | 0.443 | 0.509 |
| PPO + supervised | 0.329 | 1.474 | 0.447 | 0.514 |
| NLPO + supervised | 0.326 | 1.464 | 0.448 | 0.516 |
| DfPO + supervised | 0.335 | 1.455 | 0.445 | 0.489 |
The early manuscript documents gains on the metric used as a reward, alongside trade-offs on some of the independent measures. Figure 7 adds the learning curves for METEOR and perplexity, which stay on different scales.
Questions & Answers
The answers are simulated author responses grounded in the supplied manuscript.
Q: What does the reference model contribute?
A: It provides a probability for each token at the same prefix. The difference in log probabilities determines which rollout can sample that token. The task advantage determines whether its gradient contributes.
Q: Does a higher sentiment score guarantee better writing?
A: The classifier measures sentiment. Perplexity and diversity describe other aspects of the output, and the warm-start result shows why those measures matter. A human evaluation would be needed to compare readability and faithfulness directly.
Q: Can we decode with the ordinary trained policy?
A: The study compares that choice with both action masks at inference. The negative mask gives the strongest sentiment improvement in Figure 9. It also requires keeping the reference policy available during generation.
Future Ideas
Test the two masks on a style-transfer task with an explicit meaning-preservation check. Compare DfPO and PPO at the same rollout budget and reference checkpoint. At similar reward levels, ask blinded readers to score fidelity, naturalness, and repetition.
Run the same decoding options after training to separate improvements due to the policy update from improvements due to the inference mask.