Overview
Some questions need two documents. One may name the person, object, or event that connects the question to a second source. A keyword search can find a plausible first page and still miss the chain. PromptRank reranks pairs of documents by working backward from the evidence to the question.
Key Findings
Search takes one step, then follows a link
A fact-checker may find the article that names a company and still need a second document to identify its founder. The retrieval problem in PromptRank is to find both articles and keep the relationship between them. The system first retrieves documents with TF-IDF, which favors informative terms shared with the question. It reranks those candidates, follows hyperlinks from the promising ones, and reranks the resulting paths. A relevant page still needs a route into the initial pool or a subsequent expansion. Figure 2
See which pair would have prompted the question
The central move is to put a path in a prompt and score the question it would make the model expect. If is a candidate path and is the prompt containing it, Equation 1 is
The question remains fixed across candidates. This avoids directly comparing generation probabilities for paths with different lengths and wording. The authors test instruction search, temperature scaling, ensembling across instructions, and in-context demonstrations. The prompt without an instruction uses no task labels. Most other PromptRank rows in the main table tune the prompt and temperature using a set of 128 labeled examples.
A good document can be part of a bad pair
Scoring the whole path makes a large difference on 4,000 HotpotQA questions. Document recall at two rises from 22.8 when the model reranks individual passages to 46.9 when it reranks whole paths. For the larger main comparison, a document receives the score of its best path. Recall checks that both reference documents appear in the retrieved set, and answer recall checks for the answer string on questions with span answers. Table 4 · Table 1
| Reranker | Recall at 2 | Recall at 10 | Answer recall at 2 |
|---|---|---|---|
| TF-IDF and BM25 | 19.1 | 54.7 | 49.5 |
| PromptRank, T5-XL without instruction | 42.8 | 68.9 | 69.3 |
| PromptRank, T5-XL with eight demonstrations | 54.5 | 73.6 | 79.1 |
| PathRetriever | 66.4 | 77.8 | 82.2 |
The supervised PathRetriever comparison uses about 90,000 examples. The better no-instruction score over the lexical baseline shows how much a pretrained language model can contribute before any task labels are supplied. The remaining gap shows why demonstrations and task-specific training are different evidence regimes.
Hand the articles to a reader
The team also asks whether the paths help answer the questions. With the same reader, development-set exact match rises from 39.6 with TF-IDF to 55.7 with PromptRank without an instruction and 58.3 with ten demonstrations. The supervised MDR retriever reaches 62.3 in that comparison. PromptRank therefore improves both document retrieval and the downstream reader. It buys that improvement by scoring candidate paths through a language model, so the number of paths and the number of prompts matter to deployment cost. Tables 2–3
Questions & Answers
Q: Why score a question instead of generating one and comparing the wording?
A: The reranker measures the probability of the actual question under each path. That gives candidates a shared target and avoids relying on a similarity score between two independently worded questions.
Q: Does the method require thousands of examples?
A: Several reported settings use no demonstrations in the prompt. The tuned settings still use labeled examples for instruction and temperature selection. The paper separates that supervision from the much larger training sets used by fully supervised systems.
Q: Is the top-ranked pair enough to answer?
A: It can provide evidence for a downstream reader. A complete system still needs to check whether both passages support the answer and whether the answer is correct.
Q: When is the added reranking work worth the latency?
A: The available evidence sets an upper limit on what reranking can recover. On the reported setup the 3B reranker takes 1.38 seconds per query, rising to 5.22 with ensembling. That cost is easier to justify when first-stage recall is high and selecting the right combination improves a downstream decision. A comparison needs the same candidates and an explicit time budget.
Future Ideas
Rerank two links in a change-impact question
For an engineering team, pair a proposed configuration change with candidate pairs containing a dependency document and a runbook. Score how well each pair explains the team’s original impact question. Evaluate on past changes whose dependencies and actions were confirmed by maintainers.
Compare backward scoring with semantic reranking and a graph-aware baseline on the same candidate pairs. Measure how often the top two paths contain both necessary links, analyst time, and the rate of plausible paths with a missing link. Keep the dependency decision with the reviewer.
Measure candidate-pool recall before reranking and ask maintainers to accept alternative evidence pairs that substantiate the same dependency. Compare incremental quality against latency at a fixed candidate budget.
Sources
Original abstract
We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on language model prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples — 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval.