Evaluation & Judgment / COLM Workshop 2025 (SCALR)

A Verifier Can Work Through the Steps Too

ThinkPRM labels each step as it generates a verification trace, then derives a score for the complete solution from the probability of a final yes or no. The study tests whether long traces can make a small amount of process supervision go further.

Process Reward Models That Think

Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, Lu Wang

Overview

A correct final answer can follow from a wrong intermediate step. A process reward model evaluates the steps along the way, but collecting enough step labels is expensive. ThinkPRM trains a reasoning model to write out a verification and give step-level judgments from a smaller labeled training set.

Key Findings

A proof can fail on one line and look convincing for the rest of the page. An ordinary verifier might supply a score. ThinkPRM generates a reason for its assessment, giving the authors a way to test whether longer verification traces make the judgment better.

ThinkPRM learns to produce step-by-step verification from synthetic traces checked against human step labels.
ThinkPRM learns to produce step-by-step verification from synthetic traces checked against human step labels.

Train on worked verifications

The data pipeline compares a sampled verification trace with human process labels and keeps the trace if all labels match.
The data pipeline compares a sampled verification trace with human process labels and keeps the trace if all labels match.

The authors ask QwQ-32B-Preview to critique PRM800K solutions step by step, then keep chains whose labels all match the human labels. This produces 1,000 verification chains covering about 8,000 step labels, roughly 1% of PRM800K. The verifier receives the proposed solution and generates reasoning about its steps. Its labels can then be used to select a candidate answer or guide a search over candidates.

Table 1 compares long training traces with shortened versions of the same 1,000 traces. The authors used gpt-4o-mini to produce concise rewrites intended to preserve essential reasoning. Both conditions are evaluated by verification F1 on subsets of ProcessBench.

VerifierTraining traceOlympiadBench F1OmniMath F1
R1-Qwen-1.5BLong87.375.7
R1-Qwen-1.5BShort64.866.7
R1-Qwen-14BLong87.385.7
R1-Qwen-14BShort55.360.8

The long-trace versions outperform these compressed-trace variants. The experiment changes both trace length and the level of detail preserved by the rewrite, so it cannot isolate length alone.

Curate verification reasoning from a small share of the labels

The training pipeline keeps synthetic traces only when their step judgments agree with the available human labels. That makes the selected subset small and helps control the obvious failure where the verifier learns from an incorrect label sequence. Agreement does not guarantee every explanation is sound. The resulting benchmark must still check the trained verifier on new steps and solutions.

Spend inference on judging, then use the judgment

The paper distinguishes identifying errors from using a verifier to choose among candidate solutions. A better step-level score can affect best-of-several selection or search, both of which are ways to spend test-time computation. The long-versus-short comparison uses the same underlying training examples with shortened traces. That makes it possible to ask about the role of the verification text rather than attributing the change to receiving more labeled solutions.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: How is this different from grading the final answer?

A: The verifier checks intermediate steps, which can reveal a faulty argument even if it ends at the expected number.

Q: Does every problem need a long verification?

A: The study shows a benefit in these training and evaluation settings. A routing policy could try cheap checks first and reserve longer verification for cases where they are inconclusive.

Q: Why can a second verifier help after an early mistake?

A: An incorrect early judgment can influence the verifier’s later labels. The paper also reports scores concentrated near zero or one, which limits how much a raw score can tell us about uncertainty. An independent check of the first disputed step can help detect that failure and gives a routing rule something to calibrate against.

Future Ideas

Verify a short proof at two levels of effort

Collect proofs with independently labeled first errors. Use a cheap check for arithmetic and formal rule violations, then send unresolved arguments to a reasoning verifier. Calibrate the routing rule on a separate set of labeled proofs before evaluating it. Compare with always using the longer verifier at the same total token budget. Track first-error accuracy and valid proofs incorrectly rejected.

Hold test-time tokens fixed and compare traces with similar content at different lengths when feasible. Count false approvals and errors in identifying the first consequential step.

Sources

Original abstract

Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models are released at https://github.com/mukhal/thinkprm.