Learning & Reasoning / ICLR 2026

A Field Guide to the Way a Model Works Through a Problem

The CoT Encyclopedia groups patterns in written reasoning traces and tests whether a prompt can steer a model toward patterns associated with better outcomes.

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

Seongyun Lee, Seungone Kim, Minju Seo, Yongrae Jo, Dongyoung Go, Hyeonbin Hwang, Jinho Park, Xiang Yue, Sean Welleck, Graham Neubig, Moontae Lee, Minjoon Seo

Overview

Two models can give the same correct answer and show very different work. One narrows the question immediately. Another explores several alternatives before committing. A handful of familiar labels may miss differences like these. This paper builds a vocabulary from the traces themselves, then uses it to compare and guide the models that produced them.

Key Findings

Two models can get the same question right after taking very different routes. One tests several possibilities. Another carries a single plan through to the end. The researchers let examples suggest a vocabulary for those routes, then try to change which route a model takes.

Let the examples suggest the categories

The comparison shows a fixed list of reasoning behaviors on one side and a vocabulary learned from the traces themselves on the other.
The comparison shows a fixed list of reasoning behaviors on one side and a vocabulary learned from the traces themselves on the other.

A chain-of-thought trace is the written sequence a model generates before its answer. One rubric can distinguish a trace that follows a single sequence from one that explores several ideas in parallel.

The method analyzes model outputs. GPT-4o contributes to the criteria, rubrics, and trace labels, which can carry its evaluation biases into the taxonomy. It can show that traces differ under a rubric, even when a short list of cognitive behavior counts would miss the difference. A trace can still omit or misrepresent what caused the answer, so the categories are best read as descriptions of observable behavior.

The taxonomy has five stages. The paper proposes pairs of opposing strategy descriptions, embeds the proposed criteria, groups semantically similar ones, refines the groups into evaluation criteria, and applies the criteria to compare traces. For example, a top-down approach can begin by identifying an overarching structure, while a bottom-up approach can assemble the answer from local observations. The comparison describes behavior visible in the generated text. See the full taxonomy workflow in Figure 2.

Prompt the model with a strategy suggested by prior outcomes

The study estimates which patterns are associated with correct or safe responses, then tries prompting models to use those patterns. Section 4.1 and Figure 4 give these selected before-and-after results for dataset-level strategies. The intervention targets initially incorrect or unsafe cases, and the aggregate includes cases the model already handled correctly.

BenchmarkNo strategy instructionDataset-level selected strategy
GPQA-Diamond accuracy72.7%79.1%
XSTest safety91.1%94.0%
WildGuard safety89.1%92.9%

The same section compares selected strategies with random or lower-performing choices. The comparison checks whether the particular instruction matters beyond simply asking the model to try again.

The pattern can depend on the question

A strategy that works well across a dataset is a useful starting point. It can still be a poor fit for a particular problem. The question-level experiment trains a predictor on successful cases and asks whether the recommended pattern helps on a retry. The researchers test both prediction within a benchmark and transfer from other benchmarks. They use known first-attempt failures for the retry experiments. A system choosing when to retry before it sees the correct answer faces an additional prediction problem.

Change the training questions and watch the traces

The controlled training comparison uses two versions of NuminaMath, its original free-form questions and a synthetically generated multiple-choice version. Changing the format produces Cohen’s d effect sizes as large as 1.5. In the separate domain comparison with a fixed format, the reported effects are below 0.2. The paper also interpolates the weights of two trained models and observes changes in their traces.

Figure 6 reports the larger format effect across the six reasoning criteria in these experiments. That distinction matters when training a model to explore. Changing what the model sees as an answer choice can also change the kind of trace the training rewards. See the effect sizes in Figure 6.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Do the categories describe what the model internally thinks?

A: They describe patterns in the text it produces. The method can test whether those patterns predict outcomes or respond to prompting. It does not inspect the hidden computations that produced an answer.

Q: Why learn a strategy for individual questions?

A: Different questions may benefit from different approaches. A category that works well on average can be a poor fit for a specific question. Our experiments test whether similar questions provide evidence about useful patterns for another question.

Q: Could a second attempt alone explain the gain?

A: A retry is an alternative explanation. The study compares different kinds of strategy instruction, including randomly selected patterns. A product test should also include a plain retry with the same generation budget.

Q: What changes when the system has to choose a retry before it knows the answer key?

A: The reported steering comparisons use cases where failure can be identified with the outcome. At deployment, a selector must spend its retry budget without that information. Measure which actual failures a pre-answer-key gate catches and how often it retries a correct answer. The steering strategy and the decision to use it are separate sources of error.

Future Ideas

Match a verification strategy to a technical claim

Build a small assistant that reviews draft technical documentation. It first decides whether a claim calls for reproducing an example, looking for a contradictory source, or checking a calculation. Select the strategy using rubric labels developed from previously verified examples. Return the relevant evidence and the result of the check.

Compare the selector with a plain retry, a generic checklist, and a random strategy at matched compute. Keep some libraries entirely out of training. Count material errors found, false corrections, and cases that appropriately require a human. A useful selector should help on unfamiliar documentation without increasing unjustified corrections.

Have blinded readers label strategies on outputs from a second backbone. Compare their judgments with the labels from the original pipeline before transferring the selector.

Sources

Original abstract

Long chain-of-thought (CoT) is an essential ingredient in effective usage of modern large language models, but our understanding of the reasoning strategies underlying these capabilities remains limited. While some prior works have attempted to categorize CoTs using predefined strategy types, such approaches are constrained by human intuition and fail to capture the full diversity of model behaviors. In this work, we introduce the COT E NCYCLOPEDIA , a bottom-up framework for analyzing and steering model reasoning. Our method automatically extracts diverse reasoning criteria from model-generated CoTs, embeds them into a semantic space, clusters them into representative categories, and derives contrastive rubrics to interpret reasoning behavior. Human evaluations show that this framework produces more interpretable and comprehensive analyses than existing methods. Moreover, we demonstrate that this understanding enables performance gains: we can predict which strategy a model is likely to use and guide it toward more effective alternatives. Finally, we provide practical insights, such as that training data format (e.g., free-form vs. multiple-choice) has a far greater impact on reasoning behavior than data domain, underscoring the importance of format-aware model design.