Overview
A model solves a word problem and gets it right. Ask it to reconsider and it might introduce a mistake. On another problem, a wrong first answer could survive because the model sees nothing to fix. SCORE trains a smaller model to revise its reasoning and tests different ways of deciding when to call that refiner. The same trained refiner produced much better results when a stronger verifier chose which answers to revisit.
Key Findings
There are two jobs in correcting a solution. Someone has to notice the error and someone has to work out how to fix it. The researchers set up a study in which they can improve either of those jobs independently. That makes it possible to see why a stronger reviser may barely move the final score.
Learn to repair a first attempt
The researchers sampled ten solutions from each base model for every training question. They checked the final answers and paired incorrect attempts with attempts that reached the correct answer. The small model used the successful attempt as a hint while writing step-by-step feedback on the failed one. Rules removed badly formed feedback and the researchers also tested whether the feedback led to a correct revised answer. Then they fine-tuned the model to critique the first erroneous step and produce a new solution in one pass. The successful attempt is available during training-data construction. The refiner receives the question and its initial solution during fine-tuning and testing.
When to revise is a separate decision
A verifier reviews the initial solution and decides whether to call the refiner. The experiment runs one refinement pass and compares three gates. A classifier from the same base model is trained on final-answer correctness, GPT-4 is prompted to judge solutions, and an oracle knows whether the first answer is correct. The oracle illustrates what the same refiner can do when that decision is supplied from the labels.
| Gemma-7B-it setup | GSM8K accuracy | MATH subset accuracy | CommonsenseQA accuracy |
|---|---|---|---|
| Initial answers | 36.3 | 27.1 | 67.2 |
| Same-model verifier and SCORE | 36.7 | 27.1 | 67.5 |
| GPT-4 verifier and SCORE | 42.5 | 39.2 | 75.0 |
| Oracle verifier and SCORE | 47.4 | 44.2 | 85.4 |
Selected final-answer accuracies from Table 6 of the ACL Findings paper. The MATH subset tests transfer from a refiner trained on GSM8K. The authors also evaluate QASC and RiddleSense for transfer from CommonsenseQA. The GPT-4 verifier improved Gemma’s final accuracy across all five reported datasets. The learned verifier produced small gains on some and a small decline on RiddleSense. The oracle still leaves many wrong answers unrepaired. Figure 3 displays the gate’s two failure modes separately. A false alarm sends an already correct solution through a revision that can spoil it. A missed error keeps an incorrect first answer. That is how a stronger repair model can fail to move the final score when paired with a weak gate. Figure 3
The first wrong step is enough to start a repair
Table 3 compares feedback formats. Asking the reviser to identify the first error worked better in this ablation than asking it to judge every step. The authors also found that generating a critique and making the correction in one pass was sufficient in their setup. The practical point is that the critique needs to identify a useful place to intervene. A longer critique can add opportunities to misdiagnose steps that were already correct.
More repair practice leaves the same gate in place
The data-size ablation uses subsets of 14,499 examples. When the system has a strong verifier, the reviser benefits from more examples up to roughly ten thousand in the reported GSM8K test. A weak self-verifier did not show the same benefit. That result explains why the oracle column is useful. It isolates the revisions the model could make if the decision to intervene were already correct.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Is the smaller model given the right answer at test time?
A: We use a successful attempt as a hint when making training feedback. At test time the refiner receives the question and the initial attempt. GPT-4 can judge that attempt in one experiment. Only the oracle comparison uses the ground-truth correctness label to choose which attempts to revise.
Q: What can a weaker verifier get wrong?
A: It can approve a flawed solution and give the refiner no chance to repair it. It can also flag a correct solution and expose it to an unnecessary rewrite. Figure 3 follows both paths for one model on GSM8K.
Q: Do the transfer results cover all of MATH?
A: The reported MATH result uses a subset of 181 Level 1 problems under the paper’s selection criteria. It shows transfer to that evaluation slice. Broader transfer requires separate measurements across other problem levels and formats.
Q: Is the oracle’s final score 100?
A: The oracle perfectly decides whether the initial final answer is correct in this experiment. It gives the refiner an opportunity to repair incorrect answers. Some of those repairs still fail, so the final accuracy can stay well below 100.
Q: How can we tell whether the gate or the refiner improved the score?
A: Hold the refiner fixed and compare which attempts different verifiers send to it. That design isolates the decision to revise. The paper's one-pass results also use final-answer correctness as a training or scoring signal. A right answer can still have a bad intermediate step, so a separate process audit is needed to tell whether a revision really repaired the reasoning.
Future Ideas
Let arithmetic checks decide which explanations get a second pass
An assistant that explains basic quantitative tasks could emit a compact record of its calculations alongside each answer. A tool could check arithmetic, substitutions, units, and constraints on that record. A failed check would identify a concrete step for the smaller refiner. If no check applies, the system could hand the case to a stronger reviewer instead of rewriting confidently on its own.
Start on tasks with executable ground-truth checks and a hand-reviewed sample of reasoning steps. Compare always revising, a learned verifier alone, and the tool-assisted verifier at matched cost. Measure how often wrong answers are repaired, correct answers are spoiled, and cases go unresolved. A pilot would earn wider use by preserving correct answers and reducing errors on held-out problem families at a cost lower than sending every answer to the stronger reviewer.
Add a baseline that spends the same inference cost on a fresh independent attempt. Record arithmetic errors and mistakes in problem setup separately so the test shows what an executable calculation check can catch.
Sources
Original abstract
Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs), where LLMs refine their solutions using self-generated critiques that pinpoint the errors. This work explores whether small (≤ 13B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs. We propose a novel pipeline that prompts smaller LMs to collect self-correction data that supports the training of self-refinement abilities. First, we leverage correct solutions to guide the model in critiquing their incorrect responses. Second, the generated critiques, after filtering, are used for supervised fine-tuning of the self-correcting reasoner through solution refinement. Our experimental results show improved self-correction abilities of two models on five datasets spanning math and commonsense reasoning, with notable performance gains when paired with a strong GPT-4-based verifier, though limitations are identified when using a weak self-verifier for determining when to correct.