Learning & Reasoning / ACL 2025

A Small Model Gets a Head Start on Hard Problems

The Long CoT Collection trains models on step-by-step solutions built from reasoning outlines. In the paper’s 0.5-billion-parameter experiment, that preparation led to larger gains from reinforcement learning.

One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL

Hyungjoo Chae, Dongjin Kang, Jihyuk Kim, Beong-woo Kwak, Sunghyun Park, Haeju Park, Jinyoung Yeo, Moontae Lee, Kyungjae Lee

Overview

When a model learns from a reward for getting a math answer right, it needs to get some answers right before that reward can teach it much. A weak starting model may fail so often that good examples are scarce. The Long CoT Collection gives it a head start with worked solutions. The researchers began with roughly a thousand outlines from a reasoning model and used them to guide the generation of a collection of about 100,000 longer rationales.

Key Findings

A small model can imitate the last line of a correct proof without learning what to do when its first attempt goes nowhere. The authors build examples that include the exploration along the way, then test whether those examples can give a short-reasoning model a better starting point for reinforcement learning.

Teach the shape of a solution

A reasoning outline can guide the next move, whether that means setting up the problem, trying a different approach, or verifying a calculation. The authors collected summaries of o1’s reasoning and estimated how many thought tokens it used. For a new question, they retrieved examples with related domains and budgets. GPT-4o generated a new outline and expanded it step by step, receiving the previous reasoning and the current and next outline steps. It then wrote the final solution. A separate GPT-4o check compared the answer with a reference and filtered out rationales judged incorrect. Checking the final answer leaves room for flawed intermediate reasoning.

The resulting rationales are used in supervised fine-tuning. This first stage teaches the model to produce the kinds of solutions in the collection. The second stage uses reinforcement learning with verifiable rewards, or RLVR. In the reported experiment, the training problems had integer answers, and the reward setup checked answer correctness, whether the answer could be parsed, and similarity between the sampled thought length and o1-mini’s thought length on the same problem. The small-model comparison uses the same Qwen2.5-0.5B base with and without the preliminary fine-tuning stage.

From a thousand outlines to a hundred thousand traces

Figure 2 distinguishes the seed collection from the generated training collection. The authors recorded 1,000 reasoning flows and thought lengths for the seed stage. They estimated thought length by subtracting the tokens in the returned answer from the total completion token count. This gives a rough effort signal for the outline writer. The resulting collection contains 100,000 rationales. See the data construction in Figure 2.

See what the larger models learned

The authors also fine-tuned larger models on the collection. Table 1 reports separate evaluations before the 0.5B reinforcement-learning comparison.

ModelGPQA DiamondMMLU-Pro
Llama 3.1 8B Instruct22.743.7
Llama 3.1 8B trained on the collection36.444.5
Qwen 2.5 7B Instruct37.649.9
Qwen 2.5 7B trained on the collection39.951.4

Selected scores from Table 1 of the ACL paper. The improvement is larger for Llama on GPQA Diamond than on MMLU-Pro. These are fine-tuning results for the 7B and 8B models. The reinforcement-learning comparison was run at 0.5B.

Give reinforcement learning a head start

The two sets of bars compare how much reinforcement learning improved Qwen-2.5-0.5B from its base and long-trace-trained starting points.
The two sets of bars compare how much reinforcement learning improved Qwen-2.5-0.5B from its base and long-trace-trained starting points.

The authors also compare the 0.5B model before and after reinforcement learning on the two starting points. Figure 1 reports that reinforcement learning adds 5.4 percentage points on MATH500 and 5.6 on GPQA after training on the collection. Those gains are about 2.5 and 2.1 times the corresponding gains from the base initialization. The result concerns this model and its training setup. It gives the paper a test of cold start beyond the earlier 7B and 8B fine-tuning comparison.

Ask for more thinking on purpose

The pipeline lets the authors shorten the outline budget before generating training data. Table 2 reports MATH500 scores of 66.6, 60.7, and 57.6 for models trained on data generated at 100%, 50%, and 25% of the original budget. At the smallest budget, the authors found that some outlines squeezed too much information into too few steps. Figure 8 shows that changing the budget also shifts the training collection’s length distribution. A live system that decides when to stop reasoning would need a separate evaluation. See the training-budget comparison in Table 2 and Figure 8.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Where did the examples come from?

A: We gathered a small seed of reasoning summaries and budget estimates from o1 on questions drawn from Magpie. GPT-4o expanded that guidance to new questions by making outlines and then filling in the steps. The seed still depends on a reasoning model. The approach reduces the need for it to generate every complete rationale.

Q: Why can a head start help reinforcement learning?

A: If the model can already produce a successful answer sometimes, reinforcement learning has examples of trajectories that earn a correctness reward. The useful question is whether the initialized model finds more successful trajectories and keeps improving under the same recipe. Our best-of-many and RL experiments examine different parts of that question.

Q: Does the answer filter check each step?

A: The filter compares the generated answer with a reference answer. A correct result can coexist with an invalid step. The paper compares aspects of rationale quality on a separate sample using a model judge. A deployment that relies on the steps themselves would need checks suited to the domain.

Q: Can a larger model expect the same multiplier?

A: We tested supervised fine-tuning at 7B and 8B and used a 0.5B model for the RL comparison because of compute constraints. Testing the same intervention across larger models and different reward setups would be needed to estimate how the multiplier changes.

Q: How would we know whether retrieved outlines mattered more than extra correct practice?

A: Compare long rationales generated with and without retrieved outline guidance while holding the questions, outcome filter, data volume, and training tokens fixed. Then vary the length-based reward separately. The published supervised and reinforcement-learning results show the effects under their own settings, while those controls would separate the outline's contribution from the extra practice and reward.

Future Ideas

Spend extra reasoning on the problems that earn it

A math tutor could first try a short outline, then check whether the answer satisfies the problem’s constraints. A cheap symbolic or numeric check could catch a failed substitution or inconsistent equation. The tutor could then allocate a second attempt with a larger outline budget. If it reaches the budget without a verified answer, it could show where verification stopped and ask a teacher or learner to inspect that step.

Start with arithmetic and algebra tasks that allow reliable checks. Train with short and long outlines for the same problem families, then compare fixed-short, fixed-long, and adaptive systems on unseen families at matched total inference cost. Track correctness, invalid steps in a hand-checked sample, and performance by problem difficulty. The adaptive policy should earn deployment by matching or improving accuracy at lower cost, including the difficult cases that a cheap verifier might misclassify.

Stress-test the cheap verifier on problems where the short-budget condition was weak. Report its false pass and false escalation rates and count the full token cost of each attempt.

Sources

Original abstract

With the release of R1, a publicly available large reasoning model (LRM), researchers commonly train new LRMs by training language models on R1’s long chain-of-thought (CoT) inferences. While prior works show that LRMs’ capabilities can be reproduced through direct distillation, the continued reliance on the existing models (e.g., R1) remains a critical limitation in advancing the field.As a first step toward independent LRM development, this paper explores the possibility of constructing a long CoT dataset with LLMs that are not trained for inference-time scaling.To this end, we present the Long CoT Collection, a dataset of 100K CoT rationales annotated using existing short CoT LLMs. We develop a pipeline that induces o1’s novel reasoning strategies into short CoT LLMs, enabling them to think longer and introducing controllability over the thought budget to better manage the overthinking problem.Our extensive analyses validate that our dataset achieves quality comparable to—or slightly below—R1. Furthermore, our experiments demonstrate that training on our dataset not only strengthens general reasoning skills, but also provides a strong foundation for reinforcement learning—models initialized on our data achieve 2-3x larger gains with RLVR. We make the codes, datasets, and models publicly available at LINK.