Evaluation & Judgment / NeurIPS 2025

A Patent Benchmark That Shows Its Work

PANORAMA follows the documents behind an examiner's decision. Its three tasks reveal a gap between locating earlier inventions and judging what a new claim adds.

PANORAMA: A Dataset and Benchmark Tasks Capturing the Evaluation Trails and Rationales in Patent Examination

Hyunseung Lim, Sooyohn Nam, Sungmin Na, Ji Yong Cho, June Yong Yang, Hyungyu Shin, Yoonjoo Lee, Juho Kim, Moontae Lee, Hwajung Hong

Overview

An examiner's objection to a patent claim can turn on a single paragraph in an earlier document. Finding that document is one skill. Explaining which part matters and deciding what follows from it are others. Moontae and his colleagues assembled a dataset that makes those steps visible through the records of 8,143 U.S. patent applications examined from 2015 to 2024.

Key Findings

A patent claim rarely makes its way through examination in one piece. An examiner can point to a passage in an older patent, the applicant can revise the claim, and the next decision depends on what changed. PANORAMA brings those steps together so a model can be evaluated on pieces of the work an examiner actually does.

An examination trail links original claims, cited prior art, amendments, and an eventual notice of allowance.
An examination trail links original claims, cited prior art, amendments, and an eventual notice of allowance.

Follow the evidence from claim to decision

The authors link the initial claims, non-final rejection, cited patents, revised claims, and notice of allowance. The office action records which claim was examined, which prior art the examiner cited, and the reason for an objection. The dataset organizes those records so that researchers can examine how a claim and the supporting references relate. It uses parsed extracts from the underlying documents. The corpus was selected for applications that ultimately received a Notice of Allowance without a final rejection. Dataset description

The first benchmark task, PAR4PC, asks a model to select the examiner-cited prior art from eight candidates. PI4PC supplies a cited document and asks which of five paragraphs matters for a particular claim. NOC4PC supplies the prior art and relevant passages and asks the model to choose among a novelty rejection, a non-obviousness rejection, and allowance. Each benchmark provides the inputs for its own step. Success at one step does not automatically propagate to the next.

Check the paper trail against the source

The collection starts from 8,143 authentic office-action examples from 2015 to 2024. Parsing them into claim-level decisions makes the data searchable and gives the benchmarks specific targets. The authors asked seven domain experts to inspect parsed fields for 100 applications covering 1,874 claims. They judged 92.5% of claims to be extracted accurately. Of the 141 claims flagged as erroneous, 132 involved the legal basis code. The office actions can themselves be ambiguous about a citation or code, which helps explain why agreement on individual cases matters. See the collection and validation design in the original paper.

Decision-making is harder than finding a paragraph

Model and prompting settingPrior-art retrievalParagraph identificationNovelty and non-obviousness classification
Random response baseline5.6327.1032.33
GPT-4o, zero-shot47.3463.3334.69
Claude 3.7 Sonnet, chain-of-thought40.2960.5545.40
Gemma 3 12B Instruct, chain-of-thought77.3031.1117.67

The retrieval and paragraph tasks award partial credit for references cited against other claims in the same office action. The classification task reports macro-F1 on a 100-point scale. The random row averages 20 trials. The different tasks have different chance rates, so compare systems within each column. The same prompt strategy can help one task and hurt another. Table 2 · Table 3

Follow the claim through three decisions

The same model can excel at one link in the chain and struggle at another. In Table 2, Gemma 3 12B’s chain-of-thought setup ranks first among the three displayed language models for prior-art retrieval, but its final judgment score falls below the random baseline. The linked records make it possible to see that gap while testing each step on its own.

Training is evaluated one step at a time

The supervised fine-tuning results in Table 3 compare changes across the three tasks. Llama-3.1-8B-Instruct goes from 13.45 to 77.98 for prior-art retrieval, from 9.61 to 69.95 for paragraph identification, and from 15.71 to 49.39 on the three-way decision. The fine-tuned decision score still leaves substantial room for improvement. The authors use task-specific training splits and zero-shot prompts in this comparison. A benchmark prediction concerns the label in the recorded proceeding. A real patent judgment would involve additional evidence and review. The selection of applications that reached allowance shapes what counts as representative evidence in this dataset.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why split examination into three tasks?

A: We can distinguish a system that fails to find relevant material from one that finds it and then draws the wrong conclusion. Each task has an examiner-linked answer, which lets us compare those capabilities independently.

Q: Is the cited passage the only useful one?

A: A passage cited in the office action gives us a verifiable reference. Another paragraph may be relevant, particularly when it describes the same feature using different language. The benchmark's gold label reflects the cited evidence.

Q: Why is the final decision task difficult even with the reference in hand?

A: A claim can describe several features and relationships among them. The model has to compare that combination with the passages provided. The paper reports that some systems favor one outcome so heavily that they lose much of the benefit of having the evidence.

Q: How close is this benchmark to a full prior-art search?

A: The retrieval task works within a closed candidate set, and the final classification task supplies passages in the prompt. That helps identify which part of the pipeline fails. A real search must retrieve evidence from a much larger collection and can find relevant passages the examiner did not cite. The selected U.S. prosecution records also do not establish how well the benchmark transfers to other jurisdictions.

Future Ideas

Build a claim chart that exposes uncertain matches

Ask a system to break a claim into features and propose a quoted prior-art passage for each feature. Show exact paragraph identifiers and mark any feature for which the system lacks evidence. Let a patent researcher edit or reject every suggested match before using the chart as a search aid.

Use held-out application families with a date cutoff so related drafts and later citations cannot leak into the experiment. Compare against keyword search and a generic citation-generating model. Measure verified passage recall, unsupported quotations, and time to assemble an accurate chart. An expert panel should judge evidence independently of the examiner's citation when a plausible alternate passage is found.

Add an open-corpus test before trusting the closed-list results. For a commercial pilot, use independently cleared evaluation material rather than assuming the released dataset's noncommercial license covers deployment.

Sources

Original abstract

Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted claim meets the statutory standards of novelty and non-obviousness against previously granted claims -- prior art -- in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in office actions documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, Non-Final Rejections, and Notices of Allowance. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.