Retrieval & Search / NAACL 2025 Findings

Before an AI Searches, Give It an Outline to Follow

QPlanner builds a tree of possible subtopics, then selects branches that fit what the reader asked to include or leave out.

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

Takyoung Kim, Kyungjae Lee, Young Rok Jang, Ji Yong Cho, Gangwoo Kim, Minseok Cho, Moontae Lee

Overview

Ask for the history of a technology and a search engine may hand back the latest news. Add that you already know the recent story, and a good answer now has to decide what to leave out. QPlanner puts that decision before the search and drafting stages. It turns a broad question into related subquestions, then selects an outline that respects the reader’s requested coverage.

Key Findings

Two readers may ask for an account of the same film and want different things. One wants the ending left out. Another only wants to understand the ending. A search system that treats both as the same topic will gather plenty of related material and can still miss the brief.

A question tree lets a planner select branches according to what a reader asks to cover.
A question tree lets a planner select branches according to what a reader asks to cover.

Separate the topic from the brief

The paper calls a base question plus its coverage instruction a C2 query. A reader might ask for a film description while excluding reviews and reception. For that topic, a candidate outline can start with the plot, then ask about main events, the opening conflict, and the climax.

The authors build training examples by expanding the base question into a question tree. In their setup, three branches at each of three levels yield 39 subquestions. Candidate outlines select a small connected part of that tree. The same GPT-4 model used to build and parse outlines judges how well they follow the request. QPlanner first learns from about 31,000 parsed outlines, then pairs the highest- and lowest-rated outlines for each query in preference training. At inference it generates a question tree and an outline from the incoming query.

The outline can change what the system reads

Table 4 reports automated outline scores on a five-point scale.

MethodMean outline scoreStandard deviation
Random baseline2.571.44
Supervised QPlanner2.791.40
Preference training with synthetic negatives2.981.39
Preference training with combined negatives3.011.36
DPO-QPlanner3.161.33

The authors also test the outlines in a retrieval-augmented pipeline. In the search-only comparison in Figure 4, the final response prompt stays fixed and the outline changes which documents the system retrieves. The study reports 761 of 1,000 satisfactory judgments for DPO-QPlanner and 215 for vanilla retrieval. These are pooled judgments on 100 sampled queries, with ten ratings per query. A satisfactory judgment could include partial fulfillment of the coverage request.

Build a map with room to choose

QTREE contains 10,000 hierarchical subquery sets. The branches make it possible to compare several plans for the same query while changing how deep the response goes. A good plan for a request to avoid reviews might retain plot branches and exclude reception. A good plan for a broader request can cover both. The dataset makes those choices visible and gives a planner explicit examples to learn from.

Let people judge the shape of the plan

Section 5.2 compares human and automated outline ratings on 100 sampled test requests. The preference-trained planner’s mean human rating is 3.29, versus 3.03 for the supervised planner. The human and model scores are positively correlated, but the correlation for the preference-trained planner is 0.39. That makes the human comparison valuable even when the automated means point in the same direction. Section 5.2 The paper separately uses the outlines to guide downstream writing and retrieval. An outline can make the final response easier to target, while the passages actually retrieved still determine what it can support.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why use a tree before writing an outline?

A: The tree gives us a way to explore alternate subtopics and levels of detail. The outline makes a narrower choice about what belongs in this response. Keeping both steps visible makes it easier to inspect an omission.

Q: Can an outline prevent factual mistakes?

A: It can improve the choice of evidence and the organization of a response. Factual correctness still depends on the retrieved material and how faithfully the generator uses it.

Q: Does a high outline score mean the finished answer is good?

A: An outline score measures whether the plan follows the coverage instruction. We also ask people to evaluate finished responses, because search and generation can introduce errors after a sound plan is written.

Q: How can we tell whether an outline actually improves search?

A: The search-only comparison holds the final response prompt fixed, so it helps isolate retrieval guided by subqueries. The outline evaluator and training data share a judge, and human outline ratings show limited agreement. A stronger test checks whether the system found evidence for required points and kept excluded material out of the response, rather than relying on one scope score.

Future Ideas

Let readers check a brief before the search

For an internal documentation assistant, show a compact proposed outline with included and excluded subjects before an expensive search. Let the reader correct a single branch. Run it first on queries with clear coverage boundaries and reference documents.

Compare a visible editable outline with a hidden outline and a direct retrieval baseline at the same document budget. Measure required-topic coverage, exclusion violations, evidence support, and time spent editing. Report the benefit separately for queries with explicit constraints and those where the reader’s intent remains unclear.

Add a baseline that extracts requirements and exclusions directly from the request. Log the number of searches and amount of retrieved material as well as the time readers spend correcting the outline.

Sources

Original abstract

Interactions with large language models (LLMs) often yield long and detailed responses, leveraging both parametric knowledge and retrieval-augmented generation (RAG). While these responses can provide rich insights, they often include redundant or less engaging content not aligned with user interests. This issue becomes apparent when users specify particular subtopics to include or exclude – termed coverage-conditioned (C2) queries – as LLMs often struggle to provide tailored responses. To address this challenge, we investigate the role of query outlines, sequences of subqueries designed to guide LLMs in generating responses that meet specific user requirements. To systematically create and evaluate these outlines, we introduce QTree, a dataset of 10K hierarchical sets of information-seeking subqueries that define structured boundaries for outline creation and evaluation in C2 scenarios. Additionally, we develop QPlanner, a 7B language model trained to generate customized outlines within boundaries of QTree. We evaluate the effectiveness of the generated outlines through automatic and human judgements, focusing on their impact within retrieval-augmented generation (RAG) systems. Experimental results demonstrate that QPlanner, especially when trained with alignment techniques like DPO, generates higher-quality outlines that better fulfill diverse user needs.