Retrieval & Search / ACL 2023 Findings

When Should an Answer Come From a Document or a Stored Question?

A unified reader generates an answer from each source and compares its confidence. The selective version modestly improves both reported open-domain benchmarks.

When to Read Documents or QA History: On Unified and Selective Open-domain QA

Kyungjae Lee, Sang-eun Han, Seung-won Hwang, Moontae Lee

Overview

A search result can contain the answer in a long article. A stored question-answer pair can supply the same fact in one sentence. A reader given both may still choose poorly when one source is wrong or out of date. This paper trains a reader to use documents and stored question-answer pairs, then lets it compare two candidate answers rather than commit to one fixed source.

Key Findings

Two archives can have different kinds of answers

A document collection offers passages that explain a subject. A question archive can supply a compact answer to a question that someone has effectively asked before. The study tests both routes. Its question archive is PAQ, a collection of about 65 million question-answer pairs generated from Wikipedia. A unified reader can consume documents or package the retrieved pairs as a short passage of examples. The team first shows that adding that history passage to documents can improve retrieval and answering. The next question is how to keep a weak source from drowning out the stronger one.

Figure 1. The same reader generates an answer from each source and compares their calibrated confidence.
Figure 1. The same reader generates an answer from each source and compares their calibrated confidence.

The most fluent answer can still be the wrong one

The selective reader generates two answers and chooses the one with higher confidence. Its first signal is sequence likelihood, normalized so a longer answer need not pay an extra penalty just for containing more tokens. In Equation 6, its length-normalized version is a geometric mean over the answer tokens:

PLM(a^∣q,c)1/∣a^∣=[∏t=1∣a^∣PLM(a^t∣a^<t,q,c)]1/∣a^∣.P_{\mathrm{LM}}(\hat a\mid q,c)^{1/|\hat a|} =\left[\prod_{t=1}^{|\hat a|} P_{\mathrm{LM}}(\hat a_t\mid\hat a_{<t},q,c)\right]^{1/|\hat a|}.

A model can become more certain about incorrect answers during training. The paper adds two signals to counter this pattern. Answerability asks whether the retrieved material can answer the question. Consistency learns from how often sampled outputs match the reference during training. The reader verbalizes these auxiliary judgments through the same decoder, using labels such as True and High. The hybrid confidence combines all three signals before the two candidates compete. Sections 3.3.1–3.3.4 and Figure 3

A close contest favors choosing a source

XL reader configurationNatural Questions exact matchTriviaQA exact match
Documents only56.073.5
Question history only47.756.8
Concatenate the two sources56.774.2
Selective QA58.274.5

Exact match compares predicted answers with accepted reference strings. The selective XL configuration uses 50 retrieved documents and a history passage. Its gain over concatenation is 1.5 points on Natural Questions and 0.3 on TriviaQA. The difference is small on TriviaQA, where the document reader already performs strongly. Table 2

When confidence decides whether to answer at all

The authors also let the reader leave low-confidence questions unanswered. In the Natural Questions experiment with the Large reader, the hybrid confidence keeps the highest exact-match accuracy across the plotted coverage levels for both evidence sources. The cross-dataset test offers another check. Trained on TriviaQA and evaluated on Natural Questions, selection reaches 39.0 exact match, against 34.1 for documents and 35.2 for question history alone. The improvement still leaves a wide gap to the hypothetical selector that always knows which candidate is correct. Figure 4 and Tables 4–5

Questions & Answers

Q: Why not put all the retrieved material in the same prompt?

A: The paper evaluates that option as a baseline. In the tested setup, generating separate candidates lets the model compare how answerable the question appears under each source. Extra retrieved context can introduce noise. On Natural Questions, selection using likelihood alone also falls below the document-only reader, which supports evaluating the confidence rule alongside the input format.

Q: What does consistency mean here?

A: During training, the authors sample answers and use the fraction matching the reference as supervision for a prompted consistency estimate. That gives the reader another clue beyond the score of its single greedy answer.

Q: Could the reader choose a fluent but unsupported answer?

A: Yes. Confidence estimates can be wrong and the retrieved sources may both lack evidence. The risk-coverage analysis illustrates how often the reader is wrong when it retains only a subset of predictions.

Q: What would improve if a system picked the right source more often?

A: An oracle choosing between the same candidates reaches 64.6 exact match on Natural Questions, compared with 58.1 for the learned XL selector in the same ablation. That estimates the available room in those candidates under the reported evaluation. If both source answers are unsupported or old, a better choice between them leaves the core evidence problem unresolved.

Future Ideas

Let a help center resolve conflicts between its archives and current policy

Create an evaluation set of questions for which old answers and current documentation sometimes disagree. Generate candidates separately from the archived Q&A and the policy collection. Compare a confidence selector with concatenation, a document-only baseline, and a selector that also verifies policy dates and quotes supporting passages.

Use chronologically separated tests and include questions that neither source can answer. Measure exact answer correctness, support from the current policy, and the rate of confident stale answers. Evaluate abstention at the same coverage so a system cannot appear safer simply by answering fewer questions.

Sources

Original abstract

This paper studies the problem of open-domain question answering, with the aim of answering a diverse range of questions leveraging knowledge resources. Two types of sources, QA-pair and document corpora, have been actively leveraged with the following complementary strength. The former is highly precise when the paraphrase of given question q was seen and answered during training, often posed as a retrieval problem, while the latter generalizes better for unseen questions. A natural follow-up is thus leveraging both models, while a naive pipelining or integration approaches have failed to bring additional gains over either model alone. Our distinction is interpreting the problem as calibration, which estimates the confidence of predicted answers as an indicator to decide when to use a document or QA-pair corpus. The effectiveness of our method was validated on widely adopted benchmarks such as Natural Questions and TriviaQA.