Overview
A language model can write a convincing paragraph about a question before checking a source. A search system can find a genuine passage that barely touches the question. Putting both passages in front of a reader gives it more material and more ways to be misled. Moontae and his colleagues test a method that first asks whether the retrieved text carries evidence and then whether the generated text agrees with it.
Key Findings
The reader gets two sets of notes
One set of passages comes from searching a document corpus. The other comes from asking a language model to write what it knows about the question. Either can supply a missing fact. The generated text can also be confidently wrong, and a real document can be irrelevant. COMBO brings both sets together by asking which pairs can actually help the reader.
Learn from what the reader can afford to lose
The team needs a way to label whether a retrieved passage carries evidence and whether a generated one agrees with it. The datasets do not provide these labels. The authors rerun a trained reader after removing a retrieved passage, adding a generated one, or doing both. When removal turns a correct answer into an incorrect one, that retrieved passage earns a positive evidence signal. Checking whether a generated passage can fill the same gap helps construct consistency labels. These are tests of what this reader uses. A redundant source can be useful even when removing it does not change the answer. Figure 2
Seat compatible passages next to one another
Let be the evidence score of retrieved passage and the predicted consistency between it and generated passage . Equation 2 uses the evidence threshold to gate the pair score:
A one-to-one matching maximizes the sum of scores. The matched pairs are sorted and passed to a Fusion-in-Decoder reader with source types identified and generated text preceding retrieved text in each pair. The reader can compare passages while encoding them. A pair with zero weight can still appear because the matching covers all passages. In ablations, removing the pair structure drops NaturalQuestions development exact match from 52.3 to 51.5, and removing the evidence discriminator drops it to 51.8. Table 2 and Figure 4
The gains show up when the sources conflict
| Method | NaturalQuestions test | TriviaQA test | WebQuestion test | HotpotQA dev |
|---|---|---|---|---|
| Retrieved passages only | 46.7 | 61.9 | 48.1 | 59.9 |
| Generated passages only | 40.3 | 67.8 | 51.5 | 42.6 |
| Direct merging | 52.7 | 74.2 | 51.1 | 61.6 |
| Random matching | 53.3 | 74.2 | 51.6 | 61.5 |
| COMBO | 54.2 | 74.6 | 53.0 | 61.6 |
Exact match counts predictions that agree with accepted answer strings under the benchmark normalization. The two-source rows are averages over three runs. In the conflict analysis, the gains over direct merging tend to be larger in groups with a higher estimated conflict rate. In the highest-rate bin, COMBO reaches 61.0 versus 56.6 for direct merging, while random matching reaches 61.8. That bin contains just 1.6% of NaturalQuestions development examples. Table 3
On HotpotQA overall, the matching approach ties direct merging. A bridge question can require two supporting facts from different passages, so testing agreement in one pair cannot capture the whole chain. The published walkthrough lets a reader inspect how the matching scores order individual passage pairs. Figure 6
Questions & Answers
Q: Why include generated text if it can be wrong?
A: Generated passages can introduce relevant concepts or names that a retriever missed. Pairing them with retrieved passages helps the reader compare both sources. The generated text can be useful context while the retrieved passage supplies independently checkable evidence.
Q: Is agreement proof that a retrieved passage is correct?
A: The compatibility score is conditional on the retrieved passage being evidential according to the classifier. Two passages can agree on an error. The method assumes the retrieval corpus supplies factual evidence when the right passage is present.
Q: Why keep weak pairs?
A: The matching covers all passages and allocates the best available partners under that constraint. The paper leaves filtering incompatible pairs for future work. Filtering would add another trade-off between removing distracting text and discarding useful evidence.
Q: Do the single-hop gains justify using pairwise matching for multi-step evidence?
A: The reported single-hop gains suggest matching can help the tested reader, while the overall HotpotQA score is unchanged in the cited comparison. A pair of passages can agree yet fail jointly to support the steps needed for a correct answer. Multi-step tests should evaluate the full evidence chain and the final answer separately.
Future Ideas
A version-aware support reader
Use generated passages to suggest the terminology for an error, and retrieve manuals carrying product versions and publication dates. Mark the version explicitly in each pair. Compare direct merging, COMBO-style matching, and a variant that withholds a pair when its only official source applies to a different version. Build a held-out set containing old and current procedures, conflicting unofficial advice, and questions with no documented answer. Score answer correctness, citation support, abstention, and latency separately. Run the pilot on a small product family with versioned manuals before testing it on a broader catalogue.
Sources
Original abstract
Open-domain question answering (QA) systems are often built with retrieval modules. However, retrieving passages from a given source is known to suffer from insufficient knowledge coverage. Alternatively, prompting large language models (LLMs) to generate contextual passages based on their parametric knowledge has been shown to improve QA performance. Yet, LLMs tend to “hallucinate” content that conflicts with the retrieved knowledge. Based on the intuition that answers supported by both sources are more likely to be correct, we propose COMBO, a Compatibility-Oriented knowledge Merging for Better Open-domain QA framework, to effectively leverage the two sources of information. Concretely, we match LLM-generated passages with retrieved counterparts into compatible pairs, based on discriminators trained with silver compatibility labels. Then a Fusion-in-Decoder-based reader model handles passage pairs to arrive at the final answer. Experiments show that COMBO outperforms competitive baselines on three out of four tested open-domain QA benchmarks. Further analysis reveals that our proposed framework demonstrates greater efficacy in scenarios with a higher degree of knowledge conflicts.