Overview
A research assistant can bring back a stack of useful sources and still leave you without a defensible answer. Maybe the report skipped a constraint. Maybe a citation supports a nearby sentence and says little about the conclusion that follows. DEER, a benchmark for deep research systems, makes those failures visible by grading the report as a whole and checking the evidence behind individual claims.
Key Findings
A report that looks right can still fail the expert
Imagine asking an AI research system for a report on a hard technical question. The answer arrives with tidy headings, a confident introduction, and a long reference list. A domain expert might still find that it never stated what it excluded, skipped a decisive step in the argument, or cited a paper that does not say what the sentence claims. The DEER paper (Section 1) argues that existing report benchmarks struggle with exactly this situation for three reasons. Their criteria are often coarse or generated by LLMs, LLM judges can miss errors that need domain expertise, and source checking usually covers only claims that carry an explicit citation marker.
DEER responds with 50 report-generation tasks across 13 domains. The seed questions come from Humanity's Last Exam, which the authors map onto the topic mix they found in 5,842 in-house user queries (Section 3). A domain expert rewrites each question as a report prompt, removes answer-revealing content, and has another expert in the same field cross-review the result.
A fixed rubric plus task-specific expert guidance
Figure 2 shows the full design. The authors drew on 80 established reporting standards across 20 domains and consolidated them into a taxonomy of 7 dimensions and 25 subdimensions (Section 4.2, Table 10). Ten independent experts reviewed it. Experts then turned the 25 subdimensions into 46 criteria and 101 scorable rubric items, each tagged as Coverage (are required components present) or Quality (how well they are executed). The same rubric applies to every report in every task, so scores can be compared item by item.
A fixed rubric alone leaves a gap, because a judge without domain knowledge may not know what a correct answer must contain. DEER therefore adds Expert Evaluation Guidance for each task. Experts list the substantive content an adequate report should cover as concrete, checkable statements (Section 4.3). The judge scores each item from 1 to 10, and the guidance takes priority over the rubric in the evaluator prompt (Figure 10). Figure 7 shows how the output can be read as a drill-down from dimensions to subdimensions to individual rubric items.
Checking claims that carry no citation marker
The second half of DEER handles external evidence. A claim extractor splits the report into atomic claims and sorts them into six types (Section 4.4, Table 14). Types A through C are verifiable. Type A is an inline-cited claim, B is uncited but supported earlier in the same section or paragraph, and C is uncited but supported in an earlier section. Types D through F are structural recaps, claims needing no citation, and claims that need a source but have none.
For an uncited sentence, an LLM identifies the earlier sentences it depends on. The candidate citation set is defined by Equation 1 of the paper:
Here is the sentence being checked, is its explicit citations, and is the set of earlier sentences it depends on. A sentence therefore inherits citations from the sentences it relies on. The inherited citations only serve as candidates. A verifier then reads the cited documents and marks the claim supported only if the evidence actually backs it (Figure 8). Table 1 of the paper walks through an example in which a sentence about solar cells inherits a citation, while an assertion about durability in hurricanes is marked as needing a source.
Search helps the evidence score before it helps the argument
The authors ran 16 systems across four families, from plain language models to full research agents. The judge was GPT-5.2 for report quality and GPT-5-mini for information verification, at roughly $0.5 to $1.0 per report (Section 5.1).
| System (Table 2 of the paper) | Request fulfillment | Analytical soundness | Info. integrity | Info. sufficiency | Mean |
|---|---|---|---|---|---|
| GPT-5.2 (think) | 5.54 | 6.21 | 2.11 | 4.16 | 5.89 |
| GPT-5.2 (think+search) | 5.50 | 6.05 | 5.57 | 6.17 | 6.62 |
| OpenAI (deep) | 4.79 | 5.33 | 6.97 | 6.89 | 6.50 |
| Qwen3-235B (deep) | 4.22 | 4.80 | 6.44 | 7.90 | 6.09 |
Search lifts the evidence dimensions sharply. For GPT-5.2, Information Integrity rises from 2.11 to 5.57. Request fulfillment and analytical soundness drift slightly lower. The authors note that reasoning models without search beat their search-enabled counterparts on report-quality metrics other than the information scores, and suggest that extra external material can blur the problem definition and argument structure (Section 6.1). Across systems, structure, format, and ethics scores are comparatively high while request fulfillment and analytical soundness lag.
Figure 3 breaks this down further. The scope subdimension under request fulfillment is particularly low, which means reports often fail to say what they cover and what they leave out. Reference amount is low under information sufficiency, which suggests systems lean on a small set of sources. By domain, agents score higher in AI, engineering, and psychology and lower in history and physics (Section 6.2).
An expert note can change how a judge reads the same report
For 45 reports, two domain experts each gave an independent rating, 90 ratings in all. The authors added evaluation components step by step and compared five LLM judges against the humans (Section 6.3).
| Evaluation method (Table 3) | Pearson r | Spearman ρ | Pairwise agreement |
|---|---|---|---|
| Vanilla | 0.63 | 0.59 | 0.68 |
| + Dimensions | 0.66 | 0.62 | 0.77 |
| + Granular Rubrics | 0.64 | 0.59 | 0.76 |
| + Expert Guidance | 0.73 | 0.71 | 0.84 |
| Inter-Human | 0.81 | 0.74 | 0.79 |
Splitting the rubric into many fine-grained items slightly lowered correlation, and expert guidance gave the largest gain. Agreement among the judges themselves followed the same shape (Table 4). Krippendorff's α was 0.46 for Vanilla, 0.32 with dimensions, 0.33 with granular rubrics, and 0.55 with expert guidance. A leave-one-component-out test (Table 20) gave Pearson r of 0.631 without granular rubrics, 0.643 without expert guidance, and 0.734 for the full method. The authors also check whether the evaluator rewards longer reports and whether the score scale matters. Report length showed no meaningful Pearson correlation with the average score across the 800 reports in the baseline comparison (r = −0.02, Table 24), and the 5-band anchored 10-point scale beat a 5-point scale and a no-anchor variant (Table 23).
How much does it cost to follow the evidence?
Back-tracking was tested against expert-annotated dependencies on 131 correctly typed B/C claims (Table 15). It reached a Jaccard score of 0.7070, with precision 0.7109 and recall 0.7383. Sliding windows that simply inherit the previous k sentences gain recall as k grows, reaching 0.9341 at k=15, but precision falls to 0.5627.
Cost engineering also mattered (Tables 5 and 6). GPT-5-mini with batch size 20 extracted claims with Recall 90.66 and 6-class F1 of 67.52 at $0.10, versus $1.66 for GPT-5.2 at batch size 10. For verification, grouping claims by evidence URL and retrieving only relevant chunks cut cost from $12.84 to $0.95 per 1k, while matching the baseline F1 of 87.25 at group size 20. Rankings from the verification pipeline stayed stable across backbone choices (Kendall's W of 0.934 for Integrity and 0.936 for Sufficiency, Table 22). Ranking stability across judge models for report quality was lower and dipped with granular rubrics (W of 0.583) before reaching 0.742 with guidance (Table 21).
The scope is limited to text-only reports, and the authors acknowledge that LLM judges may carry biases relative to human experts. The distinction returns to the report on the desk. A benchmark can show which parts deserve a closer read. A domain expert still has to make the judgment when it matters.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: What is a claim a citation checker might miss?
A: A paragraph can include a checkable factual statement without a citation marker. We can follow its dependence on earlier cited statements and then test whether the resulting candidate sources support the claim. A retrieved source can be relevant and still fail that test.
Q: Why write guidance if there is already a rubric?
A: A rubric can ask whether the report acknowledges important assumptions. A domain expert can say which assumptions are decisive for this task. The rubric is shared across tasks, and the guidance supplies the context a judge needs to apply it.
Q: Does a higher benchmark score make a report ready to use?
A: The scores help locate weaknesses and compare reports under a defined evaluation setup. An operational decision also depends on the cost of a missed claim, the reliability of the source, and expert verification of conclusions that carry consequences.
Q: How would you test citation back-tracking separately?
A: Ask experts to label both the supporting passages and unsupported claims. Compare direct-citation checks with the dependency-based method on uncited claims that have and lack supporting evidence. Report how often each method recovers real support and how often it assigns support where experts find none.
Q: How much can we infer from agreement with expert reviewers?
A: Agreement on the labeled reports shows how the judge behaves on that sample. The agreement subset is small at the level of individual tasks, and the claim-verification set contains few naturally unsupported claims. It is therefore difficult to infer how often the judge will catch unsupported claims in a different discipline or resolve reports that score nearly the same. A separate test can deliberately sample natural citation failures and have experts verify them.
Future Ideas
Audit the change, not the whole report
Build a revision review tool for an analyst updating a weekly evidence brief. Between versions, identify changed claims and their supporting sources. Route new or weakened claims to an evidence reviewer and changed conclusions to a subject expert. Store the old and new claim beside the exact cited passage so the reviewer can check what moved.
Pilot on versioned briefs from one team. Preserve dated source snapshots when access allows. Seed edits that change a number, the scope of a claim, or a citation’s support. First test whether claim alignment identifies the changed material more reliably than a text diff under the same review-time budget. Test automated evidence judgments separately on supported and unsupported changes before making them part of routing.
Use real version histories to supplement seeded errors. Deliberately sample naturally weakened or unsupported claims for double expert labeling, and report whether the verification stage recovers the right supporting passage as well as whether it flags a problem.
Sources
Original abstract
Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and which criteria to use; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. In addition to rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that current systems produce structurally plausible, evidence-citing reports, but still struggle to fully satisfy expert-level user requests and achieve logical completeness. Beyond performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement.