Overview
A person reviewing an unfinished answer can see trouble before the final sentence. A proposed troubleshooting step might work only if the caller has administrator access, and the reviewer may know a simpler route that avoids the problem. That judgment depends on plausible continuations of both choices. This study asks whether language-model training can make better use of comparisons by modeling the likely continuation, as well as the choice that begins it.
Key Findings
A choice can be good even when the outcome is bad
Consider two routes out of a stalled troubleshooting session. One depends on a permission the caller may lack. The other keeps several workable options open. A person reading the first two steps can already distinguish them, even before learning whether the call ends well. Moontae and his colleagues propose that a preference label can contain information about those unrealized alternatives. They use regret to formalize that judgment as the gap between what an optimal policy can attain and what a particular action followed by a particular policy can attain.
This matters when the same opening is followed by different generators. A behavior policy is the model that actually produced a training response. A reference policy is the baseline the learner is measured against. In Direct Preference Optimization, or DPO, a learner compares preferred and rejected responses using their relative likelihood under the learner and reference. RePO gives the behavior policy an explicit place in that calculation. The paper frames this as an account of how people might judge partial responses. Its experiments test the training consequences of that account.
Following the value gap into the rest of the answer
The paper models a response as a sequence of choices. The state is the current context, and the action is an output step. It also charges a policy for moving too far from a reference policy. Lemma 5.3 shows that different reward functions can induce the same optimal policy. An additive state-dependent term can still alter the value assigned to individual choices. Lemma 5.5 shows that regret cancels that term, so the learning objective does not require choosing one convention from that class of equivalent rewards.
Theorem 5.4 makes the future term explicit. Below, is the context, is the current choice, is the behavior policy, and is the optimal policy. The subscript on each action value denotes the same underlying reward that makes optimal. The expectation follows trajectories generated by the behavior policy from that context and action, and discounts later differences.
Kullback–Leibler divergence measures the discrepancy between the two distributions over the next output. The right-hand side adds that discrepancy along the contexts actually visited by . The implication for a two-answer comparison is that a local difference in likelihood can carry a different meaning depending on what each choice sets in motion.
What to do when the model that wrote the data is gone
Computing the future term exactly would require generating fresh continuations from each intermediate point. RePO instead reuses the observed continuation and averages its future token log-ratios over a finite horizon. When the original generator's probabilities are logged, those values go into the estimate. In the deterministic variant, RePO_det, the observed next token gets probability one under the approximate behavior policy. This lets a practitioner train on text from a different model family and tokenizer. It also removes the original generator's uncertainty from the comparison.
The authors test that setting with mathematical reasoning data generated by Qwen-family models. With Llama3.1-8B-Instruct, RePO_det records 47.40 on MATH500 versus 35.00 for DPO and 38.60 for the base instruction model in Table 3. On GSM8K, the unchanged model reaches 81.96 and RePO_det 80.44. Figure 4 follows the same question across Llama3.2-1B and Gemma-2-2B using five random seeds. The improvement does not appear on every benchmark, which is useful information if the data and target model come from different families.
A complete solution and an unfinished one become a test of the theory
Lemma 6.1 identifies a condition under which a verified complete answer has lower estimated regret than the expected regret of an intermediate choice. The authors test a related prediction by cutting the last 16 to 80 tokens, including the answer, from successful solutions. They keep the complete solution as the preferred answer and label its truncated version as the rejected one. This creates explicit supervision about what can be seen when a trajectory stops early.
| Qwen3-1.7B, Table 4 | GSM8K accuracy | MATH accuracy |
|---|---|---|
| DPO | 72.55 | 52.14 |
| DPO with 30K additional pairs | 77.71 | 53.52 |
| RePO | 80.52 | 54.50 |
| RePO with 30K additional pairs | 80.29 | 55.26 |
The result is consistent with the training objective already expressing some of the signal supplied by the added pairs. It comes from verifier-accepted math answers under the paper's augmentation protocol. It does not measure how human judgments change when someone sees a partial answer.
The choice of test changes the leading method
Tables 1 and 2 compare Qwen3-4B-Base checkpoints under preference and mathematical reasoning evaluations. AlpacaEval 2 adjusts the win rate for answer length. The math tests check solutions for correctness.
| Training objective | AlpacaEval 2 length-controlled win rate | Arena-Hard win rate | GSM8K accuracy | MATH500 accuracy |
|---|---|---|---|---|
| DPO | 32.89% | 44.5% | 87.87% | 57.80% |
| KTO | 52.31% | 63.9% | 90.83% | 67.60% |
| RePO | 55.08% | 60.1% | 90.60% | 66.20% |
| RePO_det | 51.66% | 59.9% | 91.05% | 65.40% |
RePO leads this AlpacaEval 2 comparison. KTO leads on Arena-Hard and MATH500, and RePO_det leads on GSM8K. The differences sharpen the paper's practical point. The information a training set preserves about its generator, the labels it uses, and the tasks on which a model will be judged all affect whether estimating future consequences helps.
Questions & Answers
Q: What does the paper’s theorem actually attribute to the continuation?
A: Theorem 5.4 holds the starting context and first action fixed. It compares the value of continuing optimally from there with the value of following the behavior policy that generated the response. The difference is a discounted sum of forward KL divergences along contexts visited by the behavior policy. Once you fix that first choice, the comparison asks what it costs to keep going under one policy rather than the other. It does not say that each later token caused the original preference, or that the difference between any two answers can be read off from this theorem alone.
Q: What do we lose when we only have a single logged answer?
A: Under the behavior policy, one sampled token’s log ratio has an expectation equal to forward KL at that context, provided the candidate model has support wherever the generator does. A single continuation visits just one sequence of contexts, so its estimate can be noisy. Section 6 also replaces the theorem’s discounted sum with a finite, undiscounted mean and uses the trainable model in place of the unknown optimal policy. That changes the quantity being estimated. RePO_det takes a further step by treating the recorded token as certain under a constructed point-mass policy. It can use previously collected text but cannot recover the original generator’s uncertainty.
Q: Does the paper show that people judge an unfinished answer by its possible futures?
A: The paper gives a theory for how that judgment could work. Its main alignment experiments use synthetic preference data, and the math experiments use a correctness verifier. The masking experiment asks whether adding complete-versus-truncated examples changes training outcomes. It does not observe people choosing between the same opening steps while their plausible continuations vary. A study that holds the openings fixed and randomizes what readers learn about their consequences could test that interpretation directly.
Q: What should a team save when it creates its own preference data?
A: For logged-policy RePO, record which generator produced each response, the tokenizer and sampling settings, and token log probabilities under the actual sampling distribution. Keep the original continuation and the point at which it was assessed. Temperature and top-p filtering can change the distribution that produced the tokens. If generator records are available only for some datasets, compare the logged and deterministic objectives separately on matched examples. Cross-tokenizer results in the paper show where a text-only variant can help, while the table also contains cases where the original model wins.
Q: Does reward-shaping invariance make the benchmark comparison independent of training choices?
A: Lemma 5.5 establishes invariance to a state-dependent potential within the paper’s specified reward-equivalence class. That is an algebraic property of the regret expression. It does not control which practical estimator, learning rate, data, or model a run uses. A comparison designed to isolate the objective should use the same candidate pairs, backbone, and training budget, with the same tuning protocol.
Future Ideas
Give reviewers the same choice with different knowledge of its consequences
Build a sandbox for troubleshooting equipment with a simulator that knows which permissions, tools, and recovery paths actually exist. Create pairs of plausible opening actions for the same support ticket. Some actions should have several recoverable downstream routes, while others should have a similar first-step appearance and fewer ways out. Use a frozen behavior generator to populate a small set of feasible continuations for each action, and verify the consequences against the simulator.
Randomly assign reviewers to see just the opening decisions, one sampled outcome for each decision, or a matched summary of several possible outcomes with their frequencies. Keep the opening actions fixed and counterbalance their positions. Preregister how the generator’s branches are sampled and screened for feasibility. Give each reader one version of a pair to avoid carryover. Where a common continuation meets predefined requirements for both openings, use it as a separate control and report which pairs qualify. A short manipulation check can establish whether reviewers understood the differences in recovery paths.
Compare two regret scores, one from the displayed branch and one averaged over an independent set of generator draws. Fix the learned policy that stands in for the unknown optimum before running the study. Compare both scores with a prefix-only scorer and a success-probability model on held-out product families. Show the same sampled outcome in the single-branch and multiple-branch arms. Include cases where the distributions have similar expected success and different recovery options. This gives the comparison a chance to distinguish the effect of seeing more text from the effect of learning about the alternatives. Account for repeated ratings by reviewer and action pair, and track length and fluency as potential explanations. The first result would test sensitivity to what happened next. The second would probe whether a reader also cares what else could have happened. An adequately powered result with stable preferences across those conditions would weaken the case for this interpretation in the support domain. A shift explained equally well by the success-only model would leave the experiment without evidence for the richer regret account.
Spend the next rollout where it can change the training signal
When the original generator is replayable, use separate sampling budgets for easy and ambiguous comparisons. Start with a few pilot pairs of continuations. Let be the signed difference between their two finite-horizon future corrections, using fixed candidate and behavior policies. Use pilot disagreement, distance from the decision boundary, and variance to choose a budget . Draw a fresh batch to estimate the correction. The second batch is independent of the pilot, conditional on the example and frozen policies, so
Here is the expected finite-horizon surrogate correction for this pair. The identity does not cover the full discounted theorem or the final supervised loss. Freezing the sampling setup also matters. Recompute or recalibrate when the policies change. Test interval coverage on held-out prompts and document how very large log ratios affect it. Any clipping changes the target, so report the clipped version separately. If a calibrated uncertainty band around the combined local score and future correction still crosses the decision boundary, fall back to the local objective. That fallback can favor easy examples or particular generators, so report the coverage and error for each regime.
Compare with a uniform rollout budget, a one-suffix RePO implementation, and routing based only on length or entropy. Match charged generation and scoring costs, including pilot draws. Also compare against spending the same compute on more training pairs. First measure sign agreement against a larger independent reference sample, then measure held-out training outcomes. Keep examples drawn from a different behavior generator separate to see whether the calibration transfers. For old data with no replayable generator, treat this as a different deployment setting and evaluate the text-only objective independently.
Sources
Original abstract
Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization , which reframes RLHF through rather than reward maximization. Human preferences are often shaped by anticipation of outcomes and comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Experiments on mathematical reasoning benchmarks and human preference datasets demonstrate consistent performance gains, indicating that is an effective and human-aligned approach for training large language models.