Overview
A long chain of reasoning can help solve a difficult problem, but it makes every quick question expensive if the model uses it by default. This study looks at the moment the model chooses between a direct answer and an explicit reasoning trace. The researchers use answer correctness and a small fixed bonus for the short mode to train that choice.
Key Findings
A routing choice hidden in one newline
Imagine a reasoning model that receives "Solve 3x + 7 = 22" and, by habit, writes out a long internal deliberation before answering. Large reasoning models often work this way, and the paper (Section 1) calls the resulting waste overthinking. Proprietary services route queries behind the scenes, but open models lack a documented recipe that is cheap to train. The authors propose Root-token Policy Optimization, or RPO, which teaches a model to decide whether to think by training a single token.
That token is the first newline token after the <think> tag. In the authors' template (Section 3.2), a single newline \n opens a long reasoning trace, which they call LONG. A double newline \n\n selects SHORT, whose template immediately closes the thinking block. Because these are distinct tokens, the choice is one categorical action. Previous adaptive-reasoning methods such as AdaptThink and Thinkless apply reinforcement learning across whole reasoning trajectories (Section 2.2). The authors argue that this makes training expensive in memory and time and that it often depends on curated data or a hard-to-train starting model.
Two stages, one trained decision
The pipeline starts with supervised fine-tuning.
The model generates answers under both templates on OpenR1-Math queries, and only correct, template-valid answers survive. The authors sample 20K examples per mode, 40K in total (Section 4.2). Algorithm 1 keeps training only while default-template accuracy stays within a tolerance of the original model. Appendix B.5 shows why. On R1-Distill-1.5B, one epoch of SFT dropped MATH500 accuracy from 83.2 to 75.1, and three epochs recovered only part of it.
| Model | Stage | MATH500 P@1 | AIME24 P@1 |
|---|---|---|---|
| R1-Distill-1.5B | Baseline | 83.2 | 29.4 |
| R1-Distill-1.5B | SFT 1 ep | 75.1 | 20.8 |
| R1-Distill-1.5B | SFT 3 ep | 77.6 | 22.3 |
| R1-Distill-7B | Baseline | 93.5 | 54.2 |
| R1-Distill-7B | SFT 1 ep | 90.4 | 43.5 |
| R1-Distill-7B | SFT 2 ep | 88.0 | 44.5 |
Published paper Table 10 (forced-LONG evaluation, pass@1 in %).
The second stage is RPO itself (Algorithm 2). For each prompt the model produces 16 rollouts, 8 forced to SHORT and 8 to LONG. Each rollout receives a reward equal to the number of correct answers in its own mode group, scaled by a weight. LONG uses and SHORT uses , a mild nudge toward saving compute (Section 3.3). Advantages are then standardized separately within each mode across the prompts in a batch. Within one prompt, rewards in a mode are identical, so the variance across prompts is what carries the learning signal. The loss touches only the root token's log-probability, with a KL penalty toward the SFT checkpoint.
This is Equation 1 in Section 3.3. Here is the root token of rollout , is its standardized advantage, is the model's root-token distribution, is the post-SFT reference, and sets the penalty weight. Training uses about 1K queries from s1k and a compression dataset. Items where LONG is rarely correct are removed, and easy and hard queries are sampled 1:1 to prevent mode collapse.
Why updating one token saves so much
Both methods must store the key-value cache while generating. The difference comes from backpropagation, whose activation memory the paper models as proportional to (Section 3.3), with the number of updated tokens. Setting removes the dependence on reasoning length.
| Model | Method | Act(bp) per rollout (GiB) | Peak at B=8 (GiB) |
|---|---|---|---|
| 1.5B | Original GRPO (T=12K) | 13.18 | 146.86 (1.00×) |
| 1.5B | RPO (T=1) | 1.10 × 10⁻³ | 41.40 (0.28×) |
| 1.5B | RPO with LoRA | 1.10 × 10⁻³ | 19.12 (0.13×) |
| 7B | Original GRPO (T=12K) | 35.16 | 436.34 (1.00×) |
| 7B | RPO (T=1) | 2.93 × 10⁻³ | 155.11 (0.36×) |
| 7B | RPO with LoRA | 2.93 × 10⁻³ | 51.01 (0.17×) |
Published paper Table 2 (1K prompt plus 12K generation, 8 concurrent rollouts).
Table 5 extends the accounting to compute for an 8B dense model. RPO totals 10,241 PFLOPs against 522,240 for AdaptThink and 393,984 for ThinkLess, about 0.02× AdaptThink. A budget-matched test in Appendix C.3 (Table 11) asks whether the gap reflects the method or just the budget. At roughly 10K PFLOPs on R1-Distill-1.5B, full-trajectory GRPO reached 80.8 on MATH500 but still routed 90.2% of queries to LONG. Two RPO checkpoints reached 77.6 with 20.5% Think and 80.4 with 43.4% Think.
Some models use the saved tokens differently
Table 3 tracks three stages on math benchmarks. Think% is the fraction of queries routed to LONG, and RAR is the paper's reasoning-per-accuracy ratio, where lower is better.
| Model | Stage | Avg P@1 (%) | Avg tokens | Think% | RAR |
|---|---|---|---|---|---|
| Qwen3-8B | Baseline (LONG) | 78.6 | 11,970 | 100.0 | 1.27 |
| Qwen3-8B | + SFT | 63.8 | 9,467 | 57.3 | 0.90 |
| Qwen3-8B | + RPO | 62.0 | 9,202 | 44.8 | 0.72 |
| Exaone4-1.2B | + SFT | 51.1 | 7,700 | 43.4 | 0.85 |
| Exaone4-1.2B | + RPO | 48.2 | 5,237 | 23.6 | 0.49 |
| Qwen3-1.7B | + SFT | 50.1 | 9,658 | 72.8 | 1.45 |
| Qwen3-1.7B | + RPO | 51.7 | 10,100 | 74.0 | 1.43 |
Published paper Table 3 (average over MATH500, AIME24, AIME25; pass@1 averaged over 16 runs).
The effect depends on the backbone. On Qwen3-8B and Exaone4-1.2B, RPO cuts Think% at a small accuracy cost. On Qwen3-1.7B, it recovers accuracy at similar Think%. The paper reads this as RPO refining a router that SFT already created. On Qwen3-32B with LoRA, RPO gave an RAR of 1.55 against 1.52 after SFT, which the authors call comparable but not better (Section 5).
Against other methods on R1-1.5B (Table 4), RPO reached an average pass@1 of 51.3 with RAR 0.96 and 20.5% Think on MATH500. AdaptThink reported 56.5 and RAR 0.74 but at a far larger training budget, so the authors treat the two as different points on a budget-accuracy frontier. Beyond math (Table 6), Qwen3-8B improved RAR from 1.85 after SFT to 1.34 after RPO, with average pass@1 rising from 30.7 to 45.4.
The newline becomes a dial
Because the first token's probability is readable, it works as a dial. Routing to SHORT when this probability reaches a threshold changes the balance between the modes. Figure 3 shows the shift on MATH500.
On R1-1.5B (Table 1), sent 62 of 500 queries to SHORT at 93.6% accuracy and kept overall accuracy at 83.8%. Lowering to 0.70 sent 155 to SHORT and dropped overall accuracy to 82.2%. On the final Qwen3-8B checkpoint (Table 12), sweeping from 0.90 to 0.60 moved Think% from 60.0 to 38.1 and math average pass@1 from 63.0 to 60.8.
Figure 5 plots pass@1 against Think% for intermediate Qwen3-8B checkpoints, and shows points with similar accuracy at different thinking ratios. Counterfactual analysis of errors (Table 15) found that at there were 4 false-SHORT errors and 77 false-LONG errors. At those became 16 and 73. The router therefore errs toward wasteful thinking, and lowering the threshold trades 12 more harmful skips for only 4 fewer wasteful ones.
Calibration is the weak spot. Figure 6 shows reliability diagrams for Qwen3-1.7B, with ECE of 0.2717 and correlation of 0.200 on MATH500, and ECE of 0.2234 and correlation of 0.420 on AIME25. The authors call the signal informative but weak and dataset dependent.
A switch still needs both modes to work
Forced-mode tests (Table 9) show the post-trained models still work when the router is overridden. For Qwen3-8B, forced thinking scored 77.3 on math average after RPO against 78.6 for the original, and forced no-think scored 42.6. This suggests training reshapes when the model thinks without breaking either mode. Appendix A.2 reports that prefix sentences can replace the newline marker, with Qwen3-8B math averages of 62.0 for the newline fork and 61.8 and 61.5 for two prefix sentences.
The limits are clear in the paper. The binary decision does not control how long a mode thinks. Gains depend on the configuration, benchmarks lack agentic tool use, and the authors did not run a full ablation of RL dynamics or a controlled comparison against other GRPO variants from public checkpoints. The experiment makes the opening decision both trainable and measurable. The resulting policy still depends on having learned two useful ways to answer in the first place.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Does a short route mean the model performs no reasoning?
A: In the experiment, short means it omits the explicit long reasoning mode. The model still processes the question and generates the answer. The route measures visible generation behavior.
Q: Does the method know which questions are hard in general?
A: It learns a model-specific signal from how the short and long routes perform on training questions. A problem that needs long reasoning for one backbone might be answered reliably in short mode by another.
Q: Why can the cost fall even though training generates whole answers?
A: The rollout supplies the reward. The policy loss only updates the root decision, so the backward pass for that objective avoids retaining the full answer’s activations. Generating the answer still has a forward-pass cost.
Q: Can the root score be used as a universal threshold for costly decisions?
A: The usefulness of a threshold depends on the workload and the relative costs of the two routing errors. Sending a simple request through a long answer costs time. Sending a risky request through a short answer can cost accuracy. Calibration varies across the paper’s datasets, so use held-out examples from the intended workload and audit routing with the short and long generation modes held fixed. Savings on the training update do not imply fewer inference tokens.
Future Ideas
An adjustable reasoning budget for a spreadsheet assistant
Collect verified spreadsheet questions where answers can be checked against saved workbooks. Include simple formula explanations and questions that require inspecting several cells and verifying units. Let the model choose a direct answer or a longer analysis, and calibrate the routing threshold on held-out workbook families.
Compare fixed short, fixed long, a static difficulty rule, and a learned root-token router at matched token budgets. Track median and tail latency, accuracy, and both types of routing error for questions involving expensive downstream changes. Send uncertain cases through a separate checker before exposing the final answer.
Freeze the generator and its two modes before calibrating the router. Track threshold drift across workbook versions, and explicitly count cases where the short route misses a costly error that the frozen long route catches.
Sources
Original abstract
Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the think tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just 2% of the training compute of prior adaptive-reasoning methods.