Learning & Reasoning / ICML 2024

A Better Reward Score Can Make a Worse Sentence

Degeneration-free Policy Optimization screens token-level updates using a fluent reference model. The experiments track what the language model gains and what it retains.

Degeneration-free Policy Optimization: RL Fine-Tuning for Language Models without Degeneration

Youngsoo Jang, Geon-Hyeong Kim, Byoungjip Kim, Yu Jin Kim, Honglak Lee, Moontae Lee

Overview

A language model asked to write a positive movie review can raise a sentiment score by finding a shortcut. The text may repeat flattering words while losing the rhythm and detail of a review. Moontae and his team study this tension between a task reward and the fluency the model learned before fine-tuning. Their method, Degeneration-free Policy Optimization, changes which updates count so the model can pursue a better task score while limiting moves away from a fluent reference.

Key Findings

A movie-review model has two responsibilities during fine-tuning. It should write a positive continuation when it is asked to, and the continuation should still read like part of the review. Figure 1 shows how changing the strength of PPO’s KL penalty can tilt a learning run toward one measurement or the other. DfPO shifts that choice to the probability of each selected token.

Figure 1. IMDb learning curves across five runs. PPO reacts strongly to the penalty settings on the left. On the right, DfPO raises sentiment scores with little change in perplexity from its own starting point.
Figure 1. IMDb learning curves across five runs. PPO reacts strongly to the penalty settings on the left. On the right, DfPO raises sentiment scores with little change in perplexity from its own starting point.

Two signals for one token

In reinforcement learning for text, an action is a token and the state is the prompt plus the tokens already generated. An advantage estimates whether a sampled action did better or worse than expected under the current policy. A positive advantage calls for raising its probability, and a negative advantage calls for lowering it. A separate signal measures how the token’s probability differs from a reference model. Standard KL-regularized optimization trades off reward and this difference through a penalty coefficient.

DfPO instead uses the sign of the log-probability gap, Rθ(s,a)R_\theta(s,a), to mask individual updates. The gap is the reference log probability minus the current one, as in Section 2.2 and the mask definition in Section 3.2.

Rθ(s,a)=log⁡π0(a∣s)−log⁡πθ(a∣s).R_\theta(s,a)=\log\pi_0(a\mid s)-\log\pi_\theta(a\mid s).

Let AA be the action advantage and omit the (s,a)(s,a) arguments for readability. The two branches in Equations 5–7 weight the log-probability gradient by the following scalar:

1{Rθ>0}max⁡(A,0)⏟increase+1{Rθ<0}min⁡(A,0)⏟decrease.\underbrace{\mathbf{1}\{R_\theta>0\}\max(A,0)}_{\text{increase}} +\underbrace{\mathbf{1}\{R_\theta<0\}\min(A,0)}_{\text{decrease}} .

A positive advantage calls for an increase. The increase counts only when the reference already assigns that token a higher probability. A decrease counts when the advantage is negative and the current model assigns the token a higher probability than the reference. For example, if the current model gives a repeated adjective more weight than the reference and the reward signal favors it, that attempted increase contributes zero. Shared parameters can still change probabilities for other tokens.

Figure 2. A movie review continuation shows the current and reference token probabilities, the two masks, and the token updates each branch retains or drops.
Figure 2. A movie review continuation shows the current and reference token probabilities, the two masks, and the token updates each branch retains or drops.

Measuring reward and fluency together

The following values come from Table 4, with GPT-2 continuing IMDb review text from the pretrained checkpoint. Perplexity measures how predictable the output is under the evaluation model. The authors use it as a proxy for naturalness.

MethodSentiment score ↑Perplexity ↓
Starting model0.489 ± 0.00632.171 ± 0.137
PPO0.602 ± 0.01233.816 ± 0.233
NLPO0.611 ± 0.02333.832 ± 0.283
DfPO0.620 ± 0.00932.563 ± 0.082

Table 4 reports standard errors across five independent runs. The PPO and NLPO rows come from Ramamurthy and colleagues’ RL4LMs paper. In this comparison DfPO improves the sentiment score while keeping perplexity closer to the starting model.

Figure 3. The figure compares the sentiment–perplexity frontier for GPT-2 on IMDb. Higher sentiment and lower perplexity put a point closer to the upper-left corner.
Figure 3. The figure compares the sentiment–perplexity frontier for GPT-2 on IMDb. Higher sentiment and lower perplexity put a point closer to the upper-left corner.

Figure 3 uses the supervised-start setting, so its plotted values differ from the pretrained-start rows above. Its DfPO point is near the perplexity of the supervised model and in the range of the best sentiment scores shown for DPO. Across the plotted PPO and DPO settings, moving toward higher sentiment usually cost more perplexity. Figure 3 and Table 4

A second test trades toxicity against fluency

For RealToxicityPrompts, the study starts with GPT-J and asks it to continue prompts in language that scores as less toxic. PPO and DPO appear with settings selected to match different aspects of DfPO’s result. At the same reported non-toxicity score, DPO reaches similar perplexity. The experiment demonstrates one way to get a competitive combination without searching a separate KL penalty coefficient for DfPO. Its results depend on how the study scores toxicity and chooses checkpoints. Table 6

Method with GPT-JNon-toxicity score ↑Perplexity ↓
Starting model4.066 ± 0.03639.463 ± 0.000
PPO selected for non-toxicity4.347 ± 0.19452.610 ± 2.342
DPO selected for non-toxicity4.343 ± 0.02639.266 ± 0.059
DfPO4.343 ± 0.01839.091 ± 0.071

Means and standard errors are reported across three runs.

Can the same rule work outside a movie review?

The CommonGen experiment asks a T5 model to generate a plausible sentence using given concepts. Training uses METEOR as the reward. Table 2 reports these means and standard errors across five runs for methods that start from the supervised model:

MethodMETEORROUGE-LBERTScore
Supervised0.309 ± 0.0010.411 ± 0.0010.929 ± 0.000
PPO + supervised0.329 ± 0.0030.436 ± 0.0040.930 ± 0.001
NLPO + supervised0.326 ± 0.0020.431 ± 0.0020.930 ± 0.000
DfPO + supervised0.336 ± 0.0010.443 ± 0.0010.929 ± 0.001

The highest METEOR and ROUGE-L in this subset accompany nearly flat BERTScore. Perplexity, diversity, and reward each measure something different. Even an increase in multiple automatic metrics leaves semantic errors and readers' judgments to check. Tables 2 and 7

Pull the objective apart

Appendix C.4 follows four versions of the IMDb training run. With an unmasked policy gradient, sentiment rises and perplexity eventually diverges. Keeping only the probability-increase term curbs the increase in perplexity and yields a smaller task gain than DfPO. The probability-decrease term alone fails to improve the task in this test. The ablation concerns one training setup, and any new application needs its own tests of reward and writing quality. Appendix C.4

Figure 11. The IMDb ablation compares an unmasked policy gradient, each individual likelihood branch, and DfPO with both branches. The bands show standard errors across five runs.
Figure 11. The IMDb ablation compares an unmasked policy gradient, each individual likelihood branch, and DfPO with both branches. The bands show standard errors across five runs.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why can the sentiment score improve when the writing gets worse?

A: A reward model measures selected properties. It can give a high score to repeated positive language even when a reader would find the review awkward or uninformative. Measuring a separate property of the generated text helps detect that failure.

Q: Does the mask guarantee that the whole model stays close to its reference?

A: The eligibility test is local to a sampled state and action before an update. Shared network parameters couple many actions and states. The benchmark tests whether a useful trade-off survives that coupling.

Q: Does DfPO eliminate hyperparameter tuning?

A: The proposed mask avoids a separate KL penalty coefficient search in these experiments. Choices such as the learning rate, reference model, and task reward still matter. The reported comparisons use common task and decoding settings.

Q: Could the reference mask block a useful behavior?

A: Yes. A desirable response may use a rare recovery instruction or an unfamiliar domain term that the reference model handles poorly. The mask makes eligibility depend on how the current model's token probability compares with the reference. Inspect excluded updates alongside task success. Perplexity and diversity scores alone can miss that failure, along with repetition and semantic errors that human readers notice.

Future Ideas

Fine-tune troubleshooting replies without erasing rare instructions

Use a versioned support knowledge base to create a reward for providing a correct troubleshooting sequence, including uncommon recovery steps. Start from a model that already writes readable support replies. Compare DfPO with PPO at equal training and tuning budgets and include the original model as an anchor. On product families held out from fine-tuning, have blinded reviewers score factual accuracy, readability, and omitted safety-critical steps. Inspect which tokens the mask rejected in rare cases. Expand the experiment only if task completion improves and both omissions and language-quality failures stay within predeclared limits.

Sources

Original abstract

As the pre-training objectives (e.g., next token pre- diction) of language models (LMs) are inherently not aligned with task scores, optimizing LMs to achieve higher downstream task scores is essen- tial. One of the promising approaches is to fine- tune LMs through reinforcement learning (RL). However, conventional RL methods based on PPO and a penalty of KL divergence are vulnerable to text degeneration where LMs do not generate natural texts anymore after RL fine-tuning. To address this problem, we provide Degeneration- free Policy Optimization (DfPO) that can fine- tune LMs to generate texts that achieve improved downstream task scores, while preserving the abil- ity to generate natural texts. To achieve this, we introduce KL-masking which masks out the ac- tions that potentially cause deviation from the reference policy when its likelihood is increased or decreased. Then, we devise truncated advan- tage functions for separately performing likeli- hood maximization and minimization to improve the task performance. In the experiments, we pro- vide the results of DfPO and baseline algorithms on various generative NLP tasks including text continuation, text detoxification, and common- sense generation. Our experiments demonstrate that DfPO successfully improves the downstream task scores while preserving the ability to gen- erate natural texts, without requiring additional hyperparameter search.