Overview
A wrong solution can have a convincing last line. To find the actual problem, a checker may need to revisit the algebra that produced it. ThinkPRM turns this checking process into a generation task. A language model explains its verification, gives step-level judgments, and can use its judgment to rank candidate solutions.
Key Findings
The checker gets a page of its own
When several candidate solutions all look convincing, their final answers are a poor guide to which argument holds up. A discriminative process reward model assigns scores to steps. ThinkPRM gives the checker room to write its own verification trace. It can follow a calculation, explain why a step fails, and issue judgments along the way. The paper first compares this trained verifier with direct language-model judging and with classifiers trained to rate steps.
Filter the checker’s reasoning one step at a time
The team asks QwQ-32B-Preview to produce traces for solutions from PRM800K. A trace is kept when its step judgments match the human labels and it meets the paper’s format and length filters. About 1,000 retained traces contain about 8,000 step labels. The authors use them to fine-tune small and larger reasoning models. In a separate comparison, filtering for agreement at intermediate steps produces better downstream verification than retaining traces based only on the final outcome, even though outcome filtering yields more examples. Figure 3 and Section 3.2 · Figure 9
The authors also ask GPT-4o mini to compress the same 1,000 verification traces and train on the shorter versions. Verification F1 drops in these selected ProcessBench comparisons. The experiment changes both how much the checker can say and which details survive compression. Table 1
| Verifier base | OlympiadBench long | OlympiadBench short | OmniMath long | OmniMath short |
|---|---|---|---|---|
| R1-Qwen-1.5B | 87.3 | 64.8 | 75.7 | 66.7 |
| R1-Qwen-14B | 87.3 | 55.3 | 85.7 | 60.8 |
Spend extra compute on the answer or on checking it
The verifier can rerank completed solutions or score partial solutions during beam search. At inference, the system can average judgments from several sampled verification traces or ask the checker to double-check its own judgment. In compute-matched tests on MATH-500 and GPQA-Physics, verifier-guided selection is comparable to sampling more answers and voting at low budgets. The reported advantage grows as the budget increases. The broader evaluation also examines AIME and code generation, testing whether the approach helps beyond the training format. Figures 7–8 · Figures 10–11
A written judgment can still lead the checker astray
The authors observe that an early incorrect judgment can make the checker more likely to mark subsequent steps incorrect. The model’s score can also concentrate near zero or one even when a reader might want uncertainty. Automatic labels from Math-Shepherd give a promising alternative to human labels in one small experiment, while the extra cost of generating a trace remains. The paper’s contribution is a way to train a checker to do more work when it has reason to, with measurable limits on how reliably it spends that effort. Sections 5.5 and 5.8
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Is a written check necessarily a correct check?
A: It gives a trace that can be inspected. The verifier can still repeat the solution’s mistake or produce a plausible but invalid critique. We evaluate its judgments against labels and candidate solutions.
Q: Why generate multiple verification traces?
A: Several checks may explore different possible errors. More checks also cost time and can share the same blind spot. The paper studies trade-offs between generation and verification compute.
Q: Can a verifier be confident for the wrong reason?
A: The paper reports solution scores concentrated near zero or one and finds that an early mistaken step judgment can bias judgments of later steps. A confident score can therefore conceal a weak check. The trace-shortening ablation also changes what information remains in the training examples, so it does not isolate the effect of length by itself.
Future Ideas
Spend a long check on the step that could change the answer
In a math tutoring benchmark, start with a cheap check and route only unresolved steps to a longer verifier. Use verified counterexamples to test whether the route catches reasoning errors that a symbolic check misses. Compare first-error detection and correct solution selection at equal compute, separately measuring mistaken rejection of sound solutions and the router’s calibration.
Calibrate the decision to request more verification on held-out proofs, and compare against a verifier with the same test-time token budget. Count false approvals of convincing invalid solutions separately.
Sources
Original abstract
Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models are released at https://github.com/mukhal/thinkprm.