Learning & Reasoning / arXiv 2024

Train on an Answer After It Has Been Reconsidered

A language model uses detailed feedback to explore revisions before turning the strongest candidates into training examples.

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

Kyungjae Lee, Dasol Hwang, Sunghyun Park, Youngsoo Jang, Moontae Lee

Overview

A pairwise preference says which answer won. The authors find their preference reward model separates more and less factual biographies, but does not reliably distinguish correct and incorrect math answers. Reinforcement Learning from Reflective Feedback gives a model more specific criticism and lets it revise responses before learning from them.

Key Findings

An answer can lose for several different reasons

An answer can sound fluent, make a faulty inference, and miss a fact. A single reward score can rank it, but the score may not say what needs to change. Reinforcement Learning from Reflective Feedback gives that explanation a job. A separate feedback model evaluates multiple aspects, and the policy model uses the feedback to produce revisions. An answer can enter training after it has had a chance to change.

Figure 1. Feedback guides the search for revised answers before selection and another round of training.
Figure 1. Feedback guides the search for revised answers before selection and another round of training.

Search among drafts, then train on the comparison

The reported exploration generates ten initial answers and twenty revisions per prompt. A response rated successful on every evaluated aspect can become a positive example, with reward used for ranking. A moderate or failed rating on any aspect can put it in the negative set. The reward-only comparison uses initial answers without revisions or feedback ratings, taking a highest-reward positive and a random negative. That setup tests the full package of revision plus feedback-based selection. The main comparison cannot isolate either change by itself. Method and data

The team compares two ways to absorb the selected examples. Rejection-sampling fine-tuning trains on accepted answers. Direct Preference Optimization trains on contrasts between preferred and rejected ones. After one round, the updated policy can generate a new batch of candidates for a second round. The experiments report up to two rounds.

The stronger recipe depends on the target

Table 2 conditionJust-Eval totalFactScoreMath accuracy
Initial model4.6070.7941.77
First DPO round, reward only4.6270.7941.09
First DPO round, reflective feedback4.6678.5047.92
Rejection sampling then DPO4.6476.3051.02
Two DPO rounds4.6379.3049.66

Just-Eval uses GPT-4 judgments. FactScore estimates the supported share of factual claims. The first reflective DPO round improves all three displayed metrics over the reward-only DPO row. The second-round combinations have different strengths. Rejection sampling followed by DPO yields the highest math accuracy among the rows shown, while two DPO rounds produce the highest FactScore shown. Table 2

Find where the extra samples are worth buying

The authors test what happens when they spend their candidate budget on initial answers, feedback messages, or revised answers. In that Just-Eval study, varying the number of initial answers matters most, while sampling different feedback messages changes little. They also compare how reward and feedback separate stronger and weaker answers on math and factuality. On factuality, the reference material supplied to the feedback model matters. These tests put a practical limit on the story: written criticism is valuable when it helps find a better candidate and the evidence is good enough to judge the revision. Figures 2–4

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why generate several revisions?

A: A critique suggests where to improve, but a single revision can still miss the issue. Sampling gives the selection step alternatives.

Q: Does the best training schedule depend on the outcome?

A: The paper’s table shows different conditions leading on factuality and mathematics. A target application should define the most important outcome first.

Q: When is the reward model a weak judge of a revision?

A: The paper finds that its preference scores separate factual biographies better than they separate correct and incorrect solutions in the math setting. Feedback can also be unreliable for a poor biography when the system sees no reference. When an external answer check or a source record exists, use it to measure whether a persuasive revision fixed the targeted problem.

Future Ideas

A revision curriculum for software release notes

Collect release notes with factual mistakes that can be checked against linked issues and commits. Compare reward-only selection with feedback on missing sources and incorrect version numbers, at the same candidate budget, and record the cost of generating and checking candidates. Score retained factual corrections after training on a held-out set, and check that models avoid inventing details when the evidence is incomplete.

Before training, randomize whether the same reviewer sees a generic comment or targeted evidence-backed advice on a draft. Compare factual corrections and style changes separately at equal candidate budgets.

Sources

Original abstract

Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs. RLRF employs a self-reflection mechanism to systematically explore and refine LLM responses, then fine-tuning the models via a RL algorithm along with promising responses. Our experiments across Just-Eval, Factuality, and Mathematical Reasoning demonstrate the efficacy and transformative potential of RLRF beyond superficial surface-level adjustment.