Learning & Reasoning / ICLR Workshop 2024

Improving a Language Model’s Score Without Losing Its Voice

DfPO uses a reference language model to decide which tokens a reinforcement learning update can promote or suppress. The results show where this improves the trade-off between reward and fluency, and where the trade-off remains.

Reading note · The figures and results are from the supplied ICLR submission manuscript for the work presented at the workshop.

Degeneration-free Policy Optimization: RL Fine-Tuning for Language Models without Degeneration

Youngsoo Jang, Geon-Hyeong Kim, Byoungjip Kim, Yu Jin Kim, Honglak Lee, Moontae Lee

Overview

Ask a language model to finish a movie review on a positive note, and a sentiment classifier can reward it for piling on praise. The model can score higher while producing unfamiliar or repetitive text. The early DfPO manuscript tests a way to use the original language model as a guide during reinforcement learning. It samples two groups of tokens and adjusts each group in a different direction.

Key Findings

The authors list DfPO at the ICLR 2024 workshop on Generative Models for Decision Making. The figures, equations, and results here come from the supplied ICLR 2024 submission manuscript, titled DfPO: Degeneration-free Policy Optimization via Action Masking in Natural Language Action Spaces. The later ICML proceedings paper revises the update and adds experiments. Author’s publication record

The fluency penalty creates a tuning problem

In the IMDb task, the model continues a partial movie review. A DistilBERT sentiment classifier supplies the reward. Perplexity tracks how predictable the generated text is and serves as a proxy for naturalness. In Figure 1, the PPO run with no KL penalty quickly drives up sentiment, and its perplexity rises beyond the chart’s range. Several penalty settings keep perplexity close to the baseline while producing smaller sentiment gains. The balance depends on both the KL coefficient and the target KL.

Figure 1. PPO with different KL settings on IMDb review continuation. Higher sentiment is better for the task, and higher perplexity indicates less predictable text. Curves average five runs.
Figure 1. PPO with different KL settings on IMDb review continuation. Higher sentiment is better for the task, and higher perplexity indicates less predictable text. Curves average five runs.

Two rollout policies divide the work

For a token aa after prefix ss, the manuscript compares its probabilities under the reference model π0\pi_0 and the current policy πθ\pi_\theta.

uθ(s,a)=log⁡π0(a∣s)−log⁡πθ(a∣s).u_\theta(s,a)=\log\pi_0(a\mid s)-\log\pi_\theta(a\mid s).

The positive mask samples among tokens the current policy ranks lower than the reference does (uθ>0u_\theta>0). It retains positive task advantages and makes those sampled tokens more likely. The negative mask samples among tokens ranked higher than by the reference (uθ<0u_\theta<0). It retains negative task advantages and makes those sampled tokens less likely. Equations 5–6 renormalize the current distribution separately within each mask. Equation 7 adds the two policy-gradient terms:

∇θJ=Edπmask+ ⁣[(A+)+∇θlog⁡πθ(a∣s)]+Edπmask− ⁣[(A−)−∇θlog⁡πθ(a∣s)].\begin{aligned} \nabla_\theta J ={}&\mathbb E_{d^{\pi_{\mathrm{mask}}^+}}\! \left[(A^+)_{+}\nabla_\theta\log\pi_\theta(a\mid s)\right]\\ &+\mathbb E_{d^{\pi_{\mathrm{mask}}^-}}\! \left[(A^-)_{-}\nabla_\theta\log\pi_\theta(a\mid s)\right]. \end{aligned}

Here A+A^+ and A−A^- are the task advantages estimated under the respective rollouts, and (x)+=max⁡(x,0)(x)_+=\max(x,0), (x)−=min⁡(x,0)(x)_-=\min(x,0). In Figure 2, “definitely” is eligible for promotion if its task advantage is positive and the current policy underweights it relative to the reference. “Not” is eligible for suppression if its task advantage is negative and the current policy overweights it. The colored example shows possible update directions for one prefix.

Figure 2. The positive and negative action masks split the token candidates for the same review prefix. DfPO promotes tokens in the first group with positive task advantages and suppresses tokens in the second group with negative task advantages.
Figure 2. The positive and negative action masks split the token candidates for the same review prefix. DfPO promotes tokens in the first group with positive task advantages and suppresses tokens in the second group with negative task advantages.

The IMDb trade-off depends on the starting model

The table selects the competing reinforcement learning runs from Table 2. Reported uncertainties are standard errors over five runs. Cold runs start from a pretrained model, while warm runs start from a supervised checkpoint. The corresponding initial-policy perplexities are 32.171 and 35.472.

Starting modelMethodSentiment ↑Perplexity ↓
ColdPPO0.602 ± 0.01233.816 ± 0.233
ColdNLPO0.611 ± 0.02333.832 ± 0.283
ColdDfPO0.621 ± 0.00832.531 ± 0.099
WarmPPO0.626 ± 0.01435.049 ± 0.347
WarmNLPO0.620 ± 0.01434.816 ± 0.340
WarmDfPO0.662 ± 0.00635.791 ± 0.071

Cold-start DfPO has the highest mean sentiment and lowest perplexity among its RL comparators. With the supervised start, DfPO still has the highest sentiment but also has the highest final perplexity of the three RL methods. Figure 3 shows that its perplexity stays close to its own starting point. Table 3 reports gains over both PPO and NLPO on all seven IMDb diversity measures in both initialization settings. Those diagnostics describe different properties of generated text.

Figure 3. Sentiment improves in both cold and supervised starts, while each DfPO run stays close to its own initial perplexity. The curves average five runs.
Figure 3. Sentiment improves in both cold and supervised starts, while each DfPO run stays close to its own initial perplexity. The curves average five runs.

The appendix repeats the IMDb experiment with the 6-billion-parameter GPT-J as both the initial and reference model. Sentiment rises while perplexity stays near its starting level. Figure 6 plots only DfPO, so it shows how one larger model responds to the update. A comparison at that scale would require PPO and NLPO runs on the same checkpoint.

Figure 6. IMDb sentiment and perplexity during DfPO training on GPT-J. Shaded bands report standard error over five runs.
Figure 6. IMDb sentiment and perplexity during DfPO training on GPT-J. Shaded bands report standard error over five runs.

The ablation separates training from decoding

Figure 5 compares the full warm-start method with either update term alone and with a naive policy gradient. The naive update raises sentiment as perplexity escapes the plotted range. Maximization alone gives a smaller sentiment gain than the full update, while minimization alone performs worse on sentiment. At inference, this manuscript decodes with the negative mask. Figure 9 holds the training procedure fixed and compares decoding policies. The negative mask gives the strongest sentiment improvement among the three decoding choices shown.

Figure 5. Ablation comparing the full update with naive policy gradient, likelihood maximization alone, and likelihood minimization alone on IMDb.
Figure 5. Ablation comparing the full update with naive policy gradient, likelihood maximization alone, and likelihood minimization alone on IMDb.
Figure 9. The negative mask gives the highest sentiment score among the three inference policies tested after DfPO training.
Figure 9. The negative mask gives the highest sentiment score among the three inference policies tested after DfPO training.

CommonGen gains depend on the metric

CommonGen asks a T5 model to describe a plausible scene using given concepts. DfPO optimizes METEOR and leads on that metric in Table 4. PPO has the highest CIDEr mean, and NLPO leads on SPICE. Table 5 also gives DfPO a lower mean segmental type-token ratio (MSTTR, a vocabulary-diversity measure) than the other supervised-start methods.

MethodMETEOR ↑CIDEr ↑SPICE ↑MSTTR
Supervised checkpoint0.3091.3810.4430.509
PPO + supervised0.3291.4740.4470.514
NLPO + supervised0.3261.4640.4480.516
DfPO + supervised0.3351.4550.4450.489

The early manuscript documents gains on the metric used as a reward, alongside trade-offs on some of the independent measures. Figure 7 adds the learning curves for METEOR and perplexity, which stay on different scales.

Figure 7. Supervised-start DfPO improves METEOR during CommonGen training, while perplexity remains close to its starting level.
Figure 7. Supervised-start DfPO improves METEOR during CommonGen training, while perplexity remains close to its starting level.

Questions & Answers

The answers are simulated author responses grounded in the supplied manuscript.

Q: What does the reference model contribute?

A: It provides a probability for each token at the same prefix. The difference in log probabilities determines which rollout can sample that token. The task advantage determines whether its gradient contributes.

Q: Does a higher sentiment score guarantee better writing?

A: The classifier measures sentiment. Perplexity and diversity describe other aspects of the output, and the warm-start result shows why those measures matter. A human evaluation would be needed to compare readability and faithfulness directly.

Q: Can we decode with the ordinary trained policy?

A: The study compares that choice with both action masks at inference. The negative mask gives the strongest sentiment improvement in Figure 9. It also requires keeping the reference policy available during generation.

Future Ideas

Test the two masks on a style-transfer task with an explicit meaning-preservation check. Compare DfPO and PPO at the same rollout budget and reference checkpoint. At similar reward levels, ask blinded readers to score fidelity, naturalness, and repetition.

Run the same decoding options after training to separate improvements due to the policy update from improvements due to the inference mask.

Sources