Overview
When a reader asks why a paper used a particular experimental setup, the answer may draw on a paragraph about the method, a table of results, and a caveat in the discussion. Finding a relevant paragraph is a start. The reader also needs to know which parts support the answer and how they fit together. QASA was built to measure those steps on machine learning papers.
Key Findings
The question a sentence leaves behind
A reader stops at a claim in a machine-learning paper. The method is said to work when labels are scarce. Does that mean the researchers used fewer labeled examples overall, or that they kept some labels out of the training run? The sentence that prompted the question may be one paragraph, while the useful evidence sits in an experimental setup and a table. QASA makes the work between the question and the answer visible. The authors want a benchmark where a system can find the evidence, explain what each piece contributes, and put the pieces together.
Listening to the reader before building the benchmark
The team observed ten people reading scientific papers and thinking aloud. Their questions fell into three broad types. A surface question asks for information already given. A testing question checks an interpretation or compares concepts. A deep question asks for a connection or explanation that takes more work. Expert readers later generated questions from papers they chose. The authors also recruited 17 paper authors to supply testing and deep questions about their own work, with notes on the passages that might prompt readers to ask them. The finished benchmark has 1,798 question-answer pairs.
Answer annotators first judge which paragraphs supply evidence. They can add passages beyond the ten automatically retrieved candidates. They then explain the relevant contribution of each selected paragraph and compose an answer. If the evidence settles only part of the question, they identify what is still missing. About 12% of the benchmark questions were labeled unanswerable from the paper. The answerable questions had an average of 1.67 evidential rationales. Expert review of 100 sampled examples found 90% of answers correct and 87% well grounded. The reference answers therefore give a concrete target while leaving some annotation uncertainty. Figure 1 shows a complete example of the annotation.
Give the system three separate jobs
Associative selection identifies candidate paragraphs that address the question. Evidential rationale generation explains the useful part of each selected paragraph. Systematic composition joins the rationales into an answer. That split makes the failure more legible. A system might find the correct page but draw the wrong inference. A different system might write a clear answer from evidence that leaves out an experimental condition.
The researchers retrieve candidate paragraphs by comparing vector representations of questions and passages. A single language model is trained with separate instructions for the three steps, using public data and synthetic examples. The pipeline passes each stage’s output to the next. When selection returns no paragraph, the authors fall back on the top three retrieved candidates. That choice keeps the pipeline moving and leaves a question for systems that need to abstain when the available evidence is thin. See the three-stage workflow in Figure 2.
Take the rationale out and see what changes
The paper measures the three stages with their own metrics and also runs the complete pipeline. Removing the rationale step in the fine-tuned FLAN-T5 system reduces all three overlap scores in the full-stack comparison.
| Full-stack system | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Pretrained InstructGPT | 27.11 | 11.90 | 19.75 |
| Fine-tuned FLAN-T5 | 32.22 | 14.62 | 24.53 |
| Fine-tuned FLAN-T5 without rationales | 27.73 | 11.31 | 19.32 |
These are the reported values from Table 4. ROUGE-1 and ROUGE-2 count overlap with the reference in single words and adjacent pairs. ROUGE-L uses the longest shared subsequence. The comparison is evidence for the intermediate step in this system. It also raises a more important question for a reader. Does a higher score indicate an answer whose claims are better supported?
Read the answers side by side
Nine experts judged answers to 100 questions, comparing groundedness, completeness, specificity, and fluency. They tended to prefer the fine-tuned system on the first two dimensions and InstructGPT on the latter two. A second audit examined 50 answerable examples with low ROUGE-L. In that set, 48% of InstructGPT’s outputs were judged to refuse an answer despite relevant evidence, while 44% of outputs from the fine-tuned system were judged irrelevant. The authors also found cases where low overlap came from different wording or an incorrect reference. Figure 3 and Table 5 report the human comparisons and errors.
The separate results show why a scientific reading tool needs several measures. A smooth sentence can skip the condition that matters, and a defensible answer can use words its reference never used. Giving the reader the supporting passage and the intermediate explanation makes both cases easier to inspect.
Questions & Answers
Q: Why bother writing an explanation before the answer?
A: Questions about research often ask how two pieces of evidence fit together. A rationale gives the reader an intermediate step to inspect. It can reveal where the model reached beyond what the source actually supports.
Q: What does the rationale ablation show?
A: In the full QASA pipeline, removing rationale generation reduced the reported ROUGE scores. The comparison measures the pipeline as implemented. It does not establish that any written rationale will be faithful or that a different system needs the same stages.
Q: What would you test on newer models?
A: A controlled follow-up would hold the source articles, candidate passages, and review criteria fixed and test whether the answers cite the right evidence, preserve important experimental conditions, and distinguish an unanswered question from one the paper resolves.
Q: Why can a plausible research answer still fail a claim-by-claim check?
A: An answer can overlap a reference in words while changing the condition under which a result holds. Human assessment in QASA gives groundedness and completeness their own criteria, with trade-offs in fluency and specificity. Checking the claim against the passage lets a reader distinguish an ambiguous source from a fluent but unsupported inference.
Future Ideas
A reading desk that checks claims one at a time
Build a paper reader that presents each substantive answer claim alongside its supporting passage. Let researchers flag a passage as supporting, contradicting, or insufficient for the claim. Start with questions about methods and results from held-out machine learning papers. Store the passage and the reason it supports the claim so readers can inspect evidence without hunting through the document.
Compare it with a standard citation-enabled answer using the same retrieved passages and a fixed review budget. Measure unsupported claims caught, missed conditions, time to verify, and how often the tool abstains on questions the paper cannot settle. The feature earns its place if readers can catch more consequential errors at the same effort.
Include a separate label for genuinely ambiguous or incomplete source evidence. Check agreement on that label before interpreting it as a model failure.
Sources
Original abstract
Reasoning is the crux of intellectual thinking. While question answering (QA) tasks are prolific with various computational models and bench- mark datasets, they mostly tackle factoid or shal- low QA without asking deeper understanding. Dual process theory asserts that human reason- ing consists of associative thinking to collect rele- vant pieces of knowledge and logical reasoning to consciously conclude grounding on evidential ra- tionale. Based on our intensive think-aloud study that revealed the three types of questions: sur- face, testing, and deep questions, we first pro- pose the QASA benchmark that consists of 1798 novel question answering pairs that require full- stack reasoning on scientific articles in AI and ML fields. Then we propose the QASA approach that tackles the full-stack reasoning with large lan- guage models via associative selection, evidential rationale-generation, and systematic composition. Our experimental results show that QASA’s full- stack inference outperforms the state-of-the-art INSTRUCT GPT by a big margin. We also find that rationale-generation is critical for the per- formance gain, claiming how we should rethink advanced question answering. The dataset is avail- able at https://github.com/lgresearch/QASA.