Overview
A researcher needs to train a medical-imaging model in-house because patient scans cannot leave the hospital. A paper may demonstrate that property without using those words. After reading it, the researcher adds a second requirement about the evaluation setting. A search agent has to revisit its earlier answer and may eventually need to report that it found no match. CompLit turns these parts of scientific search into separately scored tests.
Key Findings
The question changes after the first paper comes back
A researcher asks for a medical imaging method trained with in-house reinforcement learning. An abstract may describe the method without saying who ran the training. The clue can be in the backbone, the experimental setup or the paper it compares itself against. Once the researcher sees a likely result, another requirement can emerge. Perhaps the method must cover a particular imaging modality. Perhaps it must generate an explanation in free text. Each new condition can disqualify an answer that seemed sound a moment earlier.
CompLit makes these separate stages of scholarly search measurable. The standard test gives the full request at once, with a known target. The other tests ask for implicit evidence, reveal requirements over multiple turns, or ask the agent to recognize that the candidate fails a requirement.
The evidence lives in the paper, the figure, or the comparison
The authors begin with comparison tables from surveys. The surveyed papers share a scope, yet the table distinguishes them by features. A condition can come from a table cell, a comparison with another paper, a figure in the target paper, or its bibliographic record. The query combines conditions that serve a research purpose. Automatic verifiers tie those conditions to evidence, and the authors check the claims and the intent themselves. A uniqueness check retrieves and screens nearby candidates for another known match.
Section 3.1 gives the logic of the task. A paper supports a request with conditions only if it supports each condition. This is the paper's satisfaction relation written as a conjunction.
The Standard setting combines 350 queries with known target papers. The 220 Implicit queries require evidence to be inferred throughout. Cumulative takes those 220 requests and spreads their conditions across three to five turns. Unmet starts from 300 requests with an easy-to-find candidate and adds a condition that the candidate fails. The search should then return no paper. A corpus check screens those combinations for known matches. As with any benchmark based on a bounded construction, that procedure cannot rule out a match in every document on the open web.
A search result, an opened paper, and a verified answer are different things
Table 2 compares systems that retrieve from the supplied corpus with agents that can use a search harness. Accuracy on Cumulative refers to the final turn. On Unmet, accuracy is the proportion of cases where the agent correctly abstains. A dash indicates that the setting is unsupported for that baseline.
| System, Table 2 | Standard | Implicit | Cumulative | Unmet |
|---|---|---|---|---|
| Qwen3 embedding plus reranker | 66.0% | 25.9% | — | — |
| Harness-1 | 69.7% | 42.3% | — | 9.0% |
| DeepSeek-V4-Pro | 74.3% | 37.3% | 35.5% | 17.3% |
| Kimi-K3 | 78.0% | 51.8% | 37.3% | 18.7% |
| Claude Code, Opus 4.8 | 86.0% | 65.9% | 50.5% | 26.3% |
| Codex CLI, GPT-5.6-Sol | 87.1% | 67.7% | 61.8% | 52.7% |
The ranking is only part of the evidence. Figure 3 follows the target through retrieval, opening and selection. On Implicit, agents lose opportunities at all three stages. The benchmark separates a paper the agent never found from one it opened and failed to verify. Figure 4 follows the answers over successive turns. Claude Code recovers from many early wrong choices as additional requirements arrive. Kimi and DeepSeek recover from fewer, and their wrong choices accumulate. That distinction only appears when the user can add to the request. The measured performance includes the search harness, tools and budgets alongside the underlying models.
Sometimes the nearest paper is the wrong answer
The Unmet setting is deliberately full of plausible near misses. An agent may locate the exact paper that satisfied the original request and still need to say that the revised request has no known match. Figure 5 on the same source page traces those cases through tool calls. Opening a near miss is common. Rejecting it after seeing the failed condition is much less reliable. The most successful system in the table abstains on only 52.7% of the cases. For a researcher, that gap can mean following a citation that seems relevant until the experiment that mattered turns out to be absent.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Why not evaluate only whether the target appears in a list?
A: The task asks the system to commit to a paper or abstain. Returning a long list can move verification back to the user. A ranked list is useful for exploration, but it measures a different behavior from identifying a paper that meets every requirement.
Q: What changes when requirements arrive over several turns?
A: The system has to retain old conditions and assess new ones against the same candidate. A paper may satisfy the first request and become invalid when a new condition appears. We evaluate the answer after all conditions have been revealed.
Q: Does a correct abstention prove that no paper exists anywhere?
A: The benchmark establishes a label through its construction and verification procedure. An actual search is limited by its corpus, time window, and coverage. A result can say that the available search found no match while making those limits clear.
Q: How should an evaluator handle a valid paper outside the designated target?
A: The benchmark’s target label gives a consistent controlled comparison. A live service can encounter a different paper that meets every condition, or a paper published after the benchmark was made. Have experts verify those alternatives in a separate live evaluation, and record the time window and databases searched when judging abstention.
Future Ideas
Keep a live shortlist that survives changing requirements
Give each candidate paper a table of requirements and supporting evidence. When the user adds a criterion, update the table before replacing the paper. Keep strong near matches with a visible reason they fail. If every candidate lacks evidence for at least one requirement, return a bounded abstention with a description of the search coverage.
Compare with a fresh query at every turn and with a standard conversational search agent. Keep CompLit’s single-target labels for comparison with the paper. Let experts adjudicate alternative papers on a separate live-search set. Track final relevance, satisfied requirements, correct abstention, tool calls, and the time needed to verify an answer. Include tasks where later turns overturn an earlier candidate.
Log which corpus and date range were searched for each abstention. On the live set, allow reviewers to add newly published matches and record whether that changes the outcome.
Sources
Original abstract
How much can we trust an agent to find the paper we need? As researchers, we search the literature from what we have read toward what we have not. We thus describe what we want in our own terms, which a paper may show only implicitly; our requirements grow as we read; and what we seek may lie in an empty region no paper yet covers. We introduce CompLit, a literature search benchmark spanning all eight arXiv domains, with one diagnostic setting for each of these needs. The strongest agents, Codex (GPT-5.6-Sol) and Claude Code (Opus 4.8), answer 86-87% of Standard queries, yet fall to 66-68% when requirements are implicit (implicit), lose another 6 and 15pp, respectively, as they accumulate (cumulative), and abstain on at most 53% of the queries no paper satisfies (unmet). No agent leads in every domain, and all are less accurate on papers from before 2019 or after 2023. Tracing their search, we find that agents often reach the right paper, or the closest one, yet misjudge it. Doing well on CompLit is a step toward agents that we can trust to find the paper we need, or to tell us that none exists.