Evaluation & Judgment / EMNLP 2025

AI Reviewers Spot Technical Flaws. What About Novelty?

An analysis of ICLR reviews finds that language models emphasize technical validity and often miss questions about what a paper adds to prior work. Training a reviewer to match expert attention improves aggregate focus similarity and can lower agreement on individual target and aspect labels.

Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim

Overview

Two reviewers can read the same experiment and raise different concerns. One might ask whether the comparison was fair. The other might ask whether the method had already been demonstrated in earlier work. A review that keeps returning to the first question can sound diligent while giving editors little help with the second. This study measures how language-model reviews allocate their attention and compares that pattern with expert reviews of AI submissions.

Key Findings

Two peer reviews can both sound serious and reach similar recommendations. One might notice that the main comparison has no control while the other spends its time on presentation. The researchers build a vocabulary for those differences and use it to ask where automatic reviews spend their attention.

The focus-level framework locates a review comment by the part of the paper it addresses and the quality it judges.
The focus-level framework locates a review comment by the part of the paper it addresses and the quality it judges.

Where a review spends its attention

The researchers assign two labels to each point in a review. The target identifies what the point discusses, such as the method, experiment, or prior research. The aspect identifies the quality it considers, such as validity or novelty. A concern about a missing experimental control could count as Experiment and Validity. A concern about an earlier method that already did the same thing could count as Method and Novelty. Separate counts for praise and criticism yield four distributions. A distribution describes the share of points assigned to each category.

To build the expert comparison, the team extracted strengths and weaknesses from OpenReview meta-reviews and supplemented them with details from individual reviews. Two authors jointly labeled a sample of 327 points, and the automatic annotator, o3-mini, achieved Cohen’s kappa of 0.81 for targets and 0.79 for aspects against those labels. Kappa measures agreement beyond what would be expected by chance. The main evaluation used rejected ICLR submissions because the available accepted versions could contain post-review revisions. The authors also assembled a separate, smaller sample of accepted submissions matched to their submitted versions.

The counting step exposed a second difference. Across the evaluation, models produced 7.88 points on average and the expert references produced 5.39. Model reviews also tended to include similar numbers of strengths and weaknesses across papers, while expert counts varied more with the submission. A longer or more balanced review can look thorough while spending its space on familiar concerns. See the count distributions in Figure 3.

The gap between aggregate balance and individual judgment

Two models can devote similar shares of their reviews to experiments and still point to different issues on a given paper. The authors use KL divergence to compare aggregate attention and F1 to compare the sets of target–aspect labels. Lower KL divergence and higher F1 indicate better agreement on their respective measures.

ModelAverage KL divergenceOverall F1Weakness F1
GPT-4o0.0820.3480.354
DeepSeek-R10.1200.3730.400
Fine-tuned GPT-4o0.0220.3060.322

Selected results from Table 3 of the EMNLP paper. The fine-tuned model’s overall distribution was closest to the experts’ among the reported systems. Its score comes from a 98-paper held-out split, while the other models were scored on the full rejected-paper sample. DeepSeek-R1 matched their target–aspect labels more often. Even matching both labels can leave a gap in substance. Two Experiment and Validity comments may refer to entirely different experiments.

Figure 4 and the results section show that human and AI reviewers both spent substantial attention on methods, experiments, and validity. The tested off-the-shelf models put less emphasis on novelty when identifying weaknesses. The paper reports F1 of 0.126 for novelty in weaknesses in the text. Per-model scores in Table 12 range from 0.013 to 0.182. This makes novelty a concrete place to look for an omission in an otherwise plausible review.

A blind spot has an address

The framework crosses the target of a comment with the property being evaluated. This makes it possible to locate a missing concern, such as weak validation of an experiment, even when a review has plenty of criticism overall. The dataset includes 676 papers. The researchers build and test an automatic annotator so the same scheme can be applied at scale, then compare the human and model distributions.

The distribution analysis pinpoints that effect. Method, experiment, and validity are frequent labels where agreement is stronger. Novelty receives little attention in the weaknesses of the off-the-shelf model reviews. The fine-tuned GPT-4o’s aggregate focus is closer to the experts’ on that label, even though its paper-level overlap is lower than some alternatives in the table above. See the labeled focus distributions in Figure 4.

Counting the topics leaves room for a second check

The study also measures overlap in more specific strengths and weaknesses. A similar distribution of topics does not guarantee the reviewers made the same observation about a given paper. The two comparisons answer different editorial questions. One says where an evaluator generally looks. The other asks whether it found a concern a person actually raised. The paper uses expert summaries as a reference and discusses biases in the dataset drawn from ICLR.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Could a model produce a useful review with a different focus from a person?

A: Yes. We measure alignment with expert review patterns along two axes. A model could offer a sound observation that a reviewer missed. Judging its usefulness calls for a separate check of what it says, which evidence it cites, and whether the issue matters to the paper’s claims.

Q: Why distinguish targets from aspects?

A: The target says where a concern lands. The aspect says what kind of concern it is. A reviewer can praise an experiment’s validity and question its novelty in the same review. Keeping the axes separate helps reveal both kinds of attention.

Q: Why did the main analysis use rejected submissions?

A: The authors needed paper and review versions to correspond. The accepted files available in OpenReview could include revisions after the reviews were written. For a smaller accepted-paper comparison, we manually sought versions close to the submission deadlines.

Q: Does the fine-tuned model have better novelty judgment?

A: Its attention to novelty weaknesses more closely resembled the expert distribution, and it had the lowest aggregate divergence. Validating novelty judgment itself would require checking whether each cited antecedent supports the comparison and whether the proposed distinction holds.

Q: If a model and an expert emphasize similar topics overall, do they find the same flaws?

A: An aggregate distribution can agree while the model misses a specific unsupported claim or experimental flaw on an individual paper. Per-paper target–aspect F1 gives a closer comparison, although the reference depends on which observations the meta-review kept and how comments were labeled. A separate blinded reread can check whether the different criticisms are correct and supported by the paper.

Future Ideas

Build a novelty evidence ledger for reviewers

For a batch of public preprints, ask a system to extract each explicitly claimed contribution. Retrieve prior papers available before the submission date and place the relevant passage next to the claimed difference. For each pair, offer one of three provisional labels: documented overlap, documented difference, or insufficient evidence. An expert can open both passages before deciding whether a criticism belongs in the review.

Start with a benchmark of previously published paper pairs annotated by domain experts. Compare the ledger with ordinary literature search and with a generic instruction to discuss novelty. Measure support for comparisons at a fixed review time, false assertions of overlap, and the share of ambiguous cases the system leaves open. A reviewer should be able to reject any ledger entry without affecting a publication decision. The pilot succeeds if reviewers find more material, source-backed comparisons with no increase in false overlap claims.

Sources

Original abstract

Peer review underpins scientific progress, but it is increasingly strained by reviewer shortages and growing workloads. Large Language Models (LLMs) can automatically draft reviews now, but determining whether LLM-generated reviews are trustworthy requires systematic evaluation. Researchers have evaluated LLM reviews at either surface-level (e.g., BLEU and ROUGE) or content-level (e.g., specificity and factual accuracy). Yet it remains uncertain whether LLM-generated reviews attend to the same critical facets that human experts weigh—the strengths and weaknesses that ultimately drive an accept-or-reject decision. We introduce a focus-level evaluation framework that operationalizes the focus as a normalized distribution of attention across predefined facets in paper reviews. Based on the framework, we developed an automatic focus-level evaluation pipeline based on two sets of facets: target (e.g., problem, method, and experiment) and aspect (e.g., validity, clarity, and novelty), leveraging 676 paper reviews from OpenReview that consists of 3,657 strengths and weaknesses identified from human experts. The comparison of focus distributions between LLMs and human experts showed that the off-the-shelf LLMs consistently have a more biased focus towards examining technical validity while significantly overlooking novelty assessment when criticizing papers.Dataset: https://figshare.com/s/d5adf26c802527dd0f62