Retrieval & Search / ACL 2023

Find the Evidence That Would Make Someone Ask the Question

PromptRank gives a language model candidate evidence and measures how likely it is to generate the original question.

Few-shot Reranking for Multi-hop QA via Language Model Prompting

Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Lu Wang

Overview

Some questions need two documents. One may name the person, object, or event that connects the question to a second source. A keyword search can find a plausible first page and still miss the chain. PromptRank reranks pairs of documents by working backward from the evidence to the question.

Key Findings

Search takes one step, then follows a link

A fact-checker may find the article that names a company and still need a second document to identify its founder. The retrieval problem in PromptRank is to find both articles and keep the relationship between them. The system first retrieves documents with TF-IDF, which favors informative terms shared with the question. It reranks those candidates, follows hyperlinks from the promising ones, and reranks the resulting paths. A relevant page still needs a route into the initial pool or a subsequent expansion. Figure 2

Figure 1. In the HotpotQA comparison, PromptRank improves retrieval without using the large task-specific training sets of supervised retrievers.
Figure 1. In the HotpotQA comparison, PromptRank improves retrieval without using the large task-specific training sets of supervised retrievers.

See which pair would have prompted the question

The central move is to put a path in a prompt and score the question it would make the model expect. If cc is a candidate path and τc\tau_c is the prompt containing it, Equation 1 is

Score⁡q(c)=PLM(q∣τc).\operatorname{Score}_q(c)=P_{\mathrm{LM}}(q\mid\tau_c).

The question remains fixed across candidates. This avoids directly comparing generation probabilities for paths with different lengths and wording. The authors test instruction search, temperature scaling, ensembling across instructions, and in-context demonstrations. The prompt without an instruction uses no task labels. Most other PromptRank rows in the main table tune the prompt and temperature using a set of 128 labeled examples.

A good document can be part of a bad pair

Scoring the whole path makes a large difference on 4,000 HotpotQA questions. Document recall at two rises from 22.8 when the model reranks individual passages to 46.9 when it reranks whole paths. For the larger main comparison, a document receives the score of its best path. Recall checks that both reference documents appear in the retrieved set, and answer recall checks for the answer string on questions with span answers. Table 4 · Table 1

RerankerRecall at 2Recall at 10Answer recall at 2
TF-IDF and BM2519.154.749.5
PromptRank, T5-XL without instruction42.868.969.3
PromptRank, T5-XL with eight demonstrations54.573.679.1
PathRetriever66.477.882.2

The supervised PathRetriever comparison uses about 90,000 examples. The better no-instruction score over the lexical baseline shows how much a pretrained language model can contribute before any task labels are supplied. The remaining gap shows why demonstrations and task-specific training are different evidence regimes.

Hand the articles to a reader

The team also asks whether the paths help answer the questions. With the same reader, development-set exact match rises from 39.6 with TF-IDF to 55.7 with PromptRank without an instruction and 58.3 with ten demonstrations. The supervised MDR retriever reaches 62.3 in that comparison. PromptRank therefore improves both document retrieval and the downstream reader. It buys that improvement by scoring candidate paths through a language model, so the number of paths and the number of prompts matter to deployment cost. Tables 2–3

Questions & Answers

Q: Why score a question instead of generating one and comparing the wording?

A: The reranker measures the probability of the actual question under each path. That gives candidates a shared target and avoids relying on a similarity score between two independently worded questions.

Q: Does the method require thousands of examples?

A: Several reported settings use no demonstrations in the prompt. The tuned settings still use labeled examples for instruction and temperature selection. The paper separates that supervision from the much larger training sets used by fully supervised systems.

Q: Is the top-ranked pair enough to answer?

A: It can provide evidence for a downstream reader. A complete system still needs to check whether both passages support the answer and whether the answer is correct.

Q: When is the added reranking work worth the latency?

A: The available evidence sets an upper limit on what reranking can recover. On the reported setup the 3B reranker takes 1.38 seconds per query, rising to 5.22 with ensembling. That cost is easier to justify when first-stage recall is high and selecting the right combination improves a downstream decision. A comparison needs the same candidates and an explicit time budget.

Future Ideas

Rerank two links in a change-impact question

For an engineering team, pair a proposed configuration change with candidate pairs containing a dependency document and a runbook. Score how well each pair explains the team’s original impact question. Evaluate on past changes whose dependencies and actions were confirmed by maintainers.

Compare backward scoring with semantic reranking and a graph-aware baseline on the same candidate pairs. Measure how often the top two paths contain both necessary links, analyst time, and the rate of plausible paths with a missing link. Keep the dependency decision with the reviewer.

Measure candidate-pool recall before reranking and ask maintainers to accept alternative evidence pairs that substantiate the same dependency. Compare incremental quality against latency at a fixed candidate budget.

Sources

Original abstract

We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on language model prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples — 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval.