Retrieval & Search / ACL 2025

A Search System Needs a Set of Clues, Not a Longer Ranking

SetR chooses passages for the different pieces of a question. In the reported tests, it often answers better with fewer passages in context.

Shifting from Ranking to Set Selection for Retrieval Augmented Generation

Dahyun Lee, Yongrae Jo, Haeju Park, Moontae Lee

Overview

Ask which river runs through the birthplace of a writer and a search engine may return five biographies that all repeat the same fact. The answer requires a second kind of evidence. This paper asks a retrieval system to select a collection of passages that works together. The selector identifies the question’s requirements and maps the candidates to those requirements before choosing a subset in one step.

Key Findings

A researcher who needs the dates of two events may get five passages that all explain the first one. A ranking system can call each passage highly relevant and still leave half the question unanswered. This study makes the unit of selection the bundle of passages.

Set-wise selection assembles complementary passages around the information requirements of the question.
Set-wise selection assembles complementary passages around the information requirements of the question.

Change what the selector is asked to produce

A conventional retrieval-augmented generation system retrieves candidate passages, reranks them, and passes a fixed number to an answer generator. SetR treats the selection as a set. It can return an unordered subset of any size. The full variant used fewer than three passages on average with either retriever in Table 1. Its full variant identifies the separate information requirements of a question before choosing passages.

The authors used GPT-4o to label selections from the top 20 candidate passages for 40,000 training questions and fine-tuned Llama-3.1-8B-Instruct on these examples. They trained three versions. The first emits only passage selections. The second includes general step-by-step reasoning. The full version explicitly identifies information requirements. The selector still depends on the first retrieval stage to surface useful candidates. Training and selection procedure

Less context can be more useful

Model with bge-large-en-v1.5 retrievalMean passagesHotpotQA F1MuSiQue F1MultiHopRAG accuracy
Retrieval only5.0030.9710.7841.82
RankGPT with GPT-4o5.0034.4513.2545.69
SetR, selection only3.4137.8415.2846.20
SetR, reasoning plus information requirements2.9138.1115.4347.14

These are selected end-to-end results using the dense retriever. The MultiHopRAG generator is GPT-4o. The other two benchmarks shown use Llama-3.1-8B-Instruct. The full variant improves MuSiQue F1 by 2.18 points over RankGPT in this setup while supplying about two fewer passages on average. Results depend on the retriever. With BM25 on MuSiQue, the selection-only variant scores 11.07 F1, ahead of the full variant's 10.57. The paper also examines token use in the selection and answering stages. Table 1 · Token analysis

What the second passage adds

A useful way to read the task is to give the question requirements R={r1,…,rm}R=\{r_1,\ldots,r_m\} and ask which ones a selected set covers. If C(p)C(p) denotes the requirements supported by passage pp, the collective coverage in this illustration is

C(S)=⋃p∈SC(p).C(S)=\bigcup_{p\in S}C(p).

This expresses the motivation in simple notation. The learned selector in the paper generates a selection from the candidate list and can use explicit requirement descriptions. It does not solve the illustrative union formula as an exact optimization problem.

On MultiHopRAG, the difference between order and coverage is visible in Table 2. RankGPT’s mean reciprocal rank at ten is 0.6358 and the full SetR variant’s is 0.5742. At five passages, the same SetR variant has higher presence-based precision, 0.2268 versus 0.1799. A ranking metric favors putting the first relevant passage near the top. The presence measure looks at what is in the selected evidence. Figure 3 examines a further distinction, how many separate requirements that evidence can cover. See the coverage comparison in Figure 3.

A compact set changes the generator’s job

The ablations separate selecting passages, generating reasoning for that selection, and explicitly identifying requirements. The variants behave differently with different candidate retrievers. The strongest configuration for a particular metric need not be the one that uses the fewest passages. Read the table as evidence about both the answer and the amount of material passed downstream. A selector can only recover evidence that made it into the candidate pool.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why might several high-ranking documents be unhelpful?

A: They can all describe the same part of the question. If a lower-ranked passage provides a necessary second fact, it may add more value to the final answer than one more passage repeating the first fact.

Q: Does explicitly listing requirements always help?

A: The results vary with the retriever and benchmark. Our selection-only model is competitive and sometimes scores higher. The main question is whether the selected context contains the needed pieces at a useful cost.

Q: Can the selector recover an important fact that never appears in the candidate pool?

A: Our selector works over the retrieved candidates. If they omit a required fact, a separate search step would have to bring new evidence into the pool.

Q: Is a smaller evidence set necessarily cheaper or sufficient?

A: Passage count is an easy measure, but passages can differ in length, and one omitted passage may contain the only support for a required fact. Keep the candidate pool and downstream reader fixed. Record both input tokens and the share of requirements with at least one supporting passage to separate savings from lost evidence.

Future Ideas

Build a policy reader that notices the missing exception

Give the reader questions about a rule and its exceptions. Ask it to identify the rule, the qualifying condition, and any effective date as separate evidence requirements. Use a fixed retrieval pool for the first comparison. Mark unsupported requirements and compare whether to abstain or launch a targeted follow-up search in a separate second experiment.

Measure answer correctness, coverage of the necessary policy clauses, and input tokens at matched retrieval budgets. Include dated or superseded policies and questions whose evidence is deliberately absent from the pool. Check that a smaller context does not remove the exception that changes the answer.

Sources

Original abstract

Retrieval in Retrieval-Augmented Generation (RAG) must ensure that retrieved passages are not only individually relevant but also collectively form a comprehensive set.Existing approaches primarily rerank top-k passages based on their individual relevance, often failing to meet the information needs of complex queries in multi-hop question answering.In this work, we propose a set-wise passage selection approach and introduce SetR, which explicitly identifies the information requirements of a query through Chain-of-Thought reasoning and selects an optimal set of passages that collectively satisfy those requirements.Experiments on multi-hop RAG benchmarks show that SetR outperforms both proprietary LLM-based rerankers and open-source baselines in terms of answer correctness and retrieval quality, providing an effective and efficient alternative to traditional rerankers in RAG systems.The code is available at https://github.com/LGAI-Research/SetR