ClaimMark Inc. · Working paper · Version 1, 23 October 2026
Measuring AI prior-art search against the examiner: a public study of ClaimMark-A Prior Art Search
Abstract
We test a commercial AI prior-art search against the prior art the USPTO examiner actually cited, and actually rejected the claims with, on US applications drawn at random from the public record. Each search sees only the art published before the application’s priority date, runs on the production system, and is compared with keyword search on the same index. On 1,440 applications filed after the search model’s training data ends, the search placed about twice as much examiner-cited art in its top 20 results as keyword search (11.4% vs 5.5%; 95% interval for the gain 1.9x–2.3x). On 3,935 applications whose first office action rejected an independent claim over US prior art, it put at least one of the examiner’s main rejection references in its top 100 results for 34.8% of applications, and in its top 1,000 for 62.1%, against 20.3% and 40.7% for keyword search. A reading of the references it missed shows that most are art the examiner used for a single feature, often of a dependent claim, which whole-application similarity rarely ranks near the top. We release the sample, the method, and the answer key on request.
1. Introduction
Prior art missed before filing is paid for later: in rejections, responses, and continued examination, and after grant in front of a challenger. In a study of every inter partes review final written decision over a year, 89% of invalidations relied on at least one reference that was missing from the examination record, and art from US patents and published applications, with the art already of record, was enough to sustain 51% of all of them [1]. Practitioners put the cost of taking a review to a final written decision at about $324,000 [1].
Buyers of AI search tools are told the tools find more of that art. As the survey above shows, 12 of 20 tools publish no measured search accuracy, and none of the seven published tests meets all ten conditions a search test needs. This paper sets out a test that does, and its first results.
2. Data
The search runs over ClaimMark-A’s production index: 21.5 million US documents, grants from 1836 and published applications from 2001, with text, claims, and a semantic vector for each document. Two samples are drawn from the public record (Table 1).
| Sample | Population and draw | Answer key | Seed and hash |
|---|---|---|---|
| Held-out set | US grants issued 2023 to 2026 with earliest priority on or after 1 January 2023, after the search model’s training data [2] ends; 2,719 drawn at random. The 1,440 searched from their published application carry the published result. | Examiner citations, by patent family | seed 20261013 list SHA-256 fd959907…0931aaa |
| Rejection sample | 5,000 drawn at random from the 621,153 published US utility applications whose first office action, mailed 1 January 2015 to 30 June 2017, rejected claims under §102 or §103 [3]. | The references of that rejection | seed 20261020 sample SHA-256 8f887dfa…6aa130 |
3. Method
3.1 Search. Each application is searched from its published text, as an applicant would have it before filing, through the production search: keyword formulations from the abstract and independent claims fused with a semantic search over PaECTER vectors [2]. The baseline is the same system with the semantic stage off. Every search is limited to documents published before the application’s earliest priority date.
3.2 Answer keys. For the held-out set, the examiner’s citations. For the rejection sample, the references the examiner applied in the first action’s §102 and §103 rejections, read from the USPTO Office Action Research Dataset [3]. A third key, the main reference a rejection of an independent claim rests on, is read from the text of each office action; its rule was committed before any figure was computed.
3.3 Exclusions. References published on or after the priority date (secret prior art), references in the application’s own family, and documents not in the index are not targets. Their counts are reported.
3.4 Scoring. A target counts as found when a result belongs to its patent family. We report the share of targets found at every cut-off from the top 5 to the top 1,000, with 95% intervals from 5,000 bootstrap resamples of applications (what the interval means).
4. Results
4.1 Held-out set. On 1,440 US patent applications filed after our search model was trained, ClaimMark's search put about twice as much of the examiner-cited prior art in its top 20 results as keyword search alone (11.4% vs 5.5%; 95% interval for the gain 1.9x–2.3x). Examiners cite only part of the relevant prior art, so these figures measure agreement with the USPTO examiner and understate what a search finds.
4.2 Rejection sample, all rejection art. 4,778 of the 5,000 applications had rejection art that was public before their priority date and is in the index (Table 2). 9.5% of the examiners’ rejection references were still unpublished on the priority date and are not counted.
| Results read | ClaimMark-A | Keyword search |
|---|---|---|
| Top 100 | 21.9% (21.0–22.8) | 12.7% (12.0–13.4) |
| Top 1,000 | 45.3% (44.3–46.4) | 29.4% (28.4–30.4) |
Each share is of every rejection reference, so art the examiner used for a single feature counts as much as the reference the rejection rests on (Section 4.3). The top-1,000 figure is measured with the deeper search used for more than 100 results.
4.3 Main reference. 3,935 of the applications had at least one main reference, the reference a rejection of an independent claim rests on, that was US art, in the index, and public before the priority date (Table 3). On those applications, ClaimMark’s search put at least one of the examiner’s main rejection references in its top 100 results for 34.8% of applications, and in its top 1,000 for 62.1%, against 20.3% and 40.7% for keyword search alone.
| Results read | ClaimMark-A | Keyword search |
|---|---|---|
| Top 100 | 34.8% (33.3–36.3) | 20.3% (19.0–21.5) |
| Top 1,000 | 62.1% (60.6–63.7) | 40.7% (39.2–42.2) |
A search for more than 100 results reads deeper candidate pools, so the top-1,000 figure is measured with that deeper search, which is being released to Pro, included plans, and the API. Of the 4,779 applications with rejection art in the index, 844 were left out: their main reference was foreign, admitted prior art, not yet published on the priority date, or could not be read reliably from the action. Two audits of the reading, the rule, and the full counts are available on request.
4.4 Why the search misses. In a random sample of the rejection references the search did not find, each read against the application and the rejection, most were art the examiner used for a single feature, often of a dependent claim, rather than for the invention as a whole. Index defects and query-text defects were rare causes.
4.5 Non-US art. A second index of non-US documents, 7.5 million European Patent Office publications in full text and about 77 million records from other offices by their abstracts, was tested as an added channel in the same ranking. A gain on the tuning sample did not hold on the held-out sample. Searches now show non-US documents as their own list beside the US results, ranked separately and never merged into the US ranking; the figures in this study do not include that list.
5. Discussion
Whole-application similarity is the method AI search tools are converging on. The misses suggest its limit: the art that costs an applicant a rejection is often about one element of one claim. We expect the next gain in prior-art search to come from searching claim element by claim element, and from ranking a deeper pool of candidates. Non-US art, likewise, appears to earn its place as its own separately measured list rather than as more documents in one ranking, and that is how searches now show it.
6. Limitations
The ranking this study measures covers US documents only. Searches also show a separate list of non-US documents (European Patent Office publications and other offices' patent families), which the figures here do not include. The shares are modest in absolute terms, because much examiner art is cited for a single feature. Examiner citations are an incomplete answer key: a result the examiner never cited may still be the best art on the list, and receives no credit. The rejection sample was filed before the search model’s training data ends, so it cannot rule out that the model saw those cases; the held-out set can.
7. Data availability
The list of the 1,440 applications, with the published text searched, each priority-date limit, and the examiner-cited families scored against, is available on request, with the method. Request it here.
References
- Stephen Yelderman, Prior Art in Inter Partes Review, 104 Iowa Law Review 2705 (2019). Every final written decision, March 2017 to February 2018; the cost figure is the AIPLA 2017 economic survey as reported there.
- Mainak Ghosh et al., PaECTER: Patent-level Representation Learning using Citation-informed Transformers, arXiv:2402.19411 (2024).
- USPTO Office of the Chief Economist, Office Action Research Dataset for Patents (2017).
Survey sources
- Patsnap. PatentBench Novelty Search: 340 samples, examiner X/Y citations, Patsnap agent vs six general chat models with web search. Page headed "July 2026"; page metadata datePublished 2026-04-22. https://www.patsnap.com/benchmark/novelty-search/
- Questel. Paper by Questel AI Lab staff, arXiv 2604.22897 (24 Apr 2026): Sophia-Bench, 10,000 queries against a 75,050-document corpus, scored by citations. Tests the QaECTER embedding model used inside Sophia Search and Orbit, not the full product. Also reports results on the external DAPFAM benchmark. https://arxiv.org/abs/2604.22897
- Amplified. Blog post of 7 Apr 2026: 906 US patents challenged in PTAB IPR, ground truth = examiner citations (IDS excluded) plus PTAB invalidity references, about 185M-patent pool, compared with BM25, CPC, citation traversal and Claude-generated keyword queries. Measures Amplified's embedding component. https://www.amplified.ai/en/blog/07042026/benchmarking-components-llms/
- IPRally. Blog post by the CPO, 28 Apr 2026: a 50-case multi-claim evaluation of IPRally Agent against a regular single IPRally search, with examiner novelty citations as ground truth. Reports relative gains only. https://www.iprally.com/news/what-does-quality-actually-mean-for-automated-patent-search
- PQAI. Blog studies (22 Aug and 29 Aug 2026): claims from five Unified Patents PATROLL challenges entered into PQAI with a priority-date cutoff; reports where the 13 known winning patent references rank in the first 100 results (5 of 13 found). https://projectpq.ai/benchmarking-pqai-for-invalidity-searches-study/
- PatSeer. Four case studies (2023, updated 2025): the vendor writes an invention description, then manually scores the top 10 AI results against its key elements ("semantic performance score", e.g. 88.6%, 82.27%). Also a gated e-book (2024) comparing AI search with Boolean search. Its contents were not read, because it is behind a form. https://www.patseer.com/case-study-benchmarking-patseers-semantic-search-relevance/