Each event gets its own table. We also plot the frontier: each tool’s Replay result against its cost per case, per capability area, so a reader can see which tools no other tool beats on both. Any overall ranking shows its weights on the page, and readers can change them, because any fixed weighting is an opinion.

The method, harness, sampling code, and scoring prompts are published on GitHub, and the samples on HuggingFace. Every change to the method is dated and explained. We are inviting an outside patent attorney and an academic research group to review the method and the rulings alongside ClaimMark’s co-founder, patent attorney Jürgen Vollrath, and we will name them here when they accept.

Every benchmark we found measured something narrower than the question attorneys care about:

• Patsnap’s PatentBench measures only whether a search finds the references the examiner cited, on a test set nobody outside Patsnap can see.
• ABIGAIL’s PatentBench is open source and aims at drafting and Office Actions, but so far it scores only docketing arithmetic, and its dataset could not be downloaded.
• The academic benchmarks score similarity to the granted text, or a model’s opinion of the claims.

None of them asks the question an attorney is paid to answer: would these claims survive the examiner, the Board, or a court? Finding the right art is only the first step, and a claim that reads well can still be anticipated. A tool that cannot show what happens when its claims meet a real attack has not shown that it can be trusted.

So we built a benchmark that asks that question directly. The ClaimMark Gauntlet takes real applications and patents from the public record, lets a tool refine the claims, and replays the examiner’s and the Board’s actual attacks against the result. A win counts only when the claims survive and keep their scope. And because a benchmark is only as good as its honesty, we hold the Gauntlet to every rule we ask of others: random samples, outputs locked before scoring, and every result published.

None of these questions requires technical knowledge of AI. They are the questions an attorney would ask of any expert report: where did the data come from, who chose it, what was it compared against, and could someone else check it.

Our own column answers the same questions, and every answer in it can be checked against what we publish.

The single most useful test is the fifth one. A number means little until you know what it beat. A specialized search tool beating a general chatbot is expected. A drafting tool scoring well on arithmetic that general models already get right is expected. The informative comparison is against the best available alternative, run on the same cases, by rules fixed before anyone saw the results.

The second most useful is the third. An outcome that happened, such as an examiner’s rejection or a Board decision, cannot be argued with. A model’s opinion of a claim can be useful as a diagnostic, but as a score it measures agreement with the model, not quality.

85% is not the share of prior art found. It is the share of test cases in which the search returned at least one reference an examiner cited as an X reference. A search that finds one of five relevant references scores the same as one that finds all five.

37% is the share of prior art found. Of all the X references examiners cited across the test set, Patsnap’s top 100 results contained 37%. The rest, close to two in three, were not in the top 100. This is the more informative of the two figures, and it is the one Patsnap’s own tool scores on, in its own benchmark.

That is not a criticism of Patsnap’s engineering. Examiners search with classification codes, full-text databases, and years of art-unit experience, and matching them is hard. It is a reason to read the headline carefully.

• Which results count. The benchmark page defines the hit rate as a correct answer “among the top 1, 3, or 5 results”. The same page, and the benchmark hub, label the 85% headline “Top 100”. Those are very different tests, and the page does not say which one produced 85%.
• What the homepage says. Patsnap’s Eureka homepage presents the figure as “85% Prior art found in the top 100 results”. By the benchmark’s own definitions, the share of prior art found is the recall figure, 37%.
• Which competitor is quoted. The homepage compares Patsnap with ChatGPT 5.4 at 17.18%. The benchmark page reports ChatGPT 5.6 at 71.76%, within about 13 points of Patsnap.
• How the figures moved. In November 2025, Patsnap reported 81% and 36%, compared against ChatGPT o3 and DeepSeek-R1. The benchmark page now reports 85% and 37% against a different set of products, while Patsnap’s own novelty-search product page still shows 81% on the same day. No version history explains the change.

• Anything about drafting. It is a search benchmark. It says nothing about the quality of a drafted application or an Office Action response.
• How relevant each result is. A result is a hit or a miss. There is no graded relevance, and no check of which claim limitations a reference actually discloses.
• How it compares with other search tools. The comparators are general chat products with web search. No patent search tool, and no simple keyword search over the same collection, was run as a baseline.
• Whether the answer was in the question. The page does not say what text each query was built from. If it came from a granted patent, the examiner’s citations are printed on the patent’s face, and may be in a model’s training data.
• How much the number moves. There are no repeat runs and no confidence intervals. With 340 cases, sampling alone puts an 85% figure within about four points either way, before any run-to-run variation.

The answer key is an outcome, not an opinion: references that examiners actually cited, grouped by family so that a U.S. patent and its European counterpart count once. The test set spans four offices and two languages. And Patsnap published something, with a worked example, which most of the industry has not.

We cannot. The test set, the queries, and the per-case results are not published, so nobody outside Patsnap can run the benchmark, including Patsnap’s own customers. If Patsnap publishes the set, we will run ClaimMark-P on it and publish every output.

Instead we do two things. First, we run the same protocol on our own sample: applications drawn at random from the public record, scored against the references the examiner cited, with the search capped at each application’s priority date. That figure appears beside Patsnap’s with a plain caveat: a different sample, a different index, and a Patsnap figure nobody can verify. It is not a head-to-head, and we do not call it one.

Second, we measure what the hit rate leaves out. XRef, the search event of the ClaimMark Gauntlet, grades relevance using the X, Y, and A categories that European and international search reports assign to every citation, and checks whether a tool’s account of which claim limitations a reference discloses matches the examiner’s own mapping in the rejection.

Patsnap may still find more of the examiner’s references than we do. Its index is larger, and for most offices our worldwide search reads abstracts and bibliographic data rather than full text. That is why search recall is one row of our results, not the headline.

ABIGAIL is a self-funded patent AI company. Its product drafts applications, prepares Office Action responses, runs prior-art searches, and handles prosecution docketing. PatentBench is published under the Apache 2.0 license on GitHub, installable as a Python package, with a white paper signed by ABIGAIL’s founder, a registered patent attorney.

ABIGAIL uses PatentBench on its website to point out that far better-funded rivals publish no benchmarks at all. That criticism is fair as far as it goes. The same standard should apply to PatentBench itself, and that is what this page does.

• It is open source, with a license that lets anyone run and extend it.
• It is built from real Office Actions, not synthetic cases.
• It says plainly which layers are finished and which are not.
• It plans to calibrate the automated scores against registered attorneys, which every AI-judged benchmark needs.

When the drafting and prosecution layers are scored, they will be scored mainly by a model reading the output against a rubric. That is useful for catching form problems. It cannot tell you whether an argument would have persuaded the examiner, because the examiner is not in the loop. The strongest answer key for a response to a rejection is what the examiner did next, and for a drafted claim, whether it survived examination. The academic work page covers why rubric scores should be a diagnostic rather than a grade.

The scored layer of PatentBench is docketing. ABIGAIL’s product syncs deadlines from the USPTO with weekend and holiday rollover, prefills USPTO forms, and offers examiner analytics. ClaimMark-P does not do docketing. It shows the period for reply exactly as the examiner wrote it and tells the attorney to use the firm’s docket. We do not calculate due dates, because a wrong date is a malpractice problem and firms already run dedicated docketing systems with a second-check rule. ClaimMark-P does generate IDS forms from the references in a matter.

So on PatentBench we would submit only the drafting and prosecution domains, and say so.

When the dataset can be downloaded, we will run ClaimMark-P on the drafting and prosecution cases and publish every output, whatever the score. Separately, the ClaimMark Gauntlet scores against what examiners and the Board actually did, which no AI judge can stand in for. We would welcome ABIGAIL’s participation in both.

Patent-CR and PatRe move in the right direction. Both are built on real examination history, so the model has to deal with an examiner’s objections rather than an idealized task. PatRe separately tests a model given the examiner’s references and a model that must find them, which is the right way to tell a search failure from a reasoning failure. Dis2Pat is the first to take the drafting problem from the inventor’s side, and it asked patent attorneys to compare outputs directly. Vibe Patenting checked its AI judges against a patent attorney and reported, honestly, where they disagreed.

1. The granted text is treated as the one right answer. Many claim sets would have been allowable. The granted claims are the product of a negotiation with the examiner that the drafter could not have seen. Scoring similarity to them rewards predicting the end of prosecution, not drafting well at the start of it.
2. The answer leaks into the question. When an inventor’s disclosure is written by a model from the granted patent, as in Dis2Pat, it carries that patent’s structure, emphasis and vocabulary. The same risk arises for search when the query is built from a granted patent whose citations are printed on its face.
3. The test cases are in the training data. Patents from 2016 and 2017, as in PatentScore, and granted European texts, as in EPD, are in every large model’s training corpus. A model may be recalling rather than reasoning, and none of these papers reports a set held out past the model’s training cutoff.
4. An opinion stands in for an outcome. Where there is no granted text to compare against, a model grades the output. Vibe Patenting found that agreement between AI judges and a patent attorney depended strongly on which quality was being measured.

Underneath all four is one missing question. None of these benchmarks asks whether the claims would have held up: before the examiner, before the Board, or in court. That is the question an attorney is paid to answer.

PatentScore is the most complete published rubric for grading patent claims, and its authors published it in full. It deserves a close reading, because rubric scores like it are likely to become the default way vendors grade drafting.

How we use it. ClaimMark-P already runs its own rubric reviews, per statute, with every finding tied to a claim and a recommendation, and its antecedent-basis check is rule-based rather than a model’s opinion. We treat those as diagnostics that drive recommendations, not as a score. In the Gauntlet we run PatentScore’s published prompts, unchanged, on each claim set before and after refinement, and report the result as a form-conformance row, labeled as a model-judged rubric that says nothing about outcome.

We are writing up this method as a paper for peer review. The samples were registered, with their hashes, before any tool ran on them, so nobody, including us, could adjust the rules after seeing the numbers. We would welcome co-authors from the research groups whose work is reviewed here, and an outside patent attorney to review the scoring rulings.

A benchmark written by one vendor will always be read with suspicion, including ours. So we publish the Gauntlet’s method, code, sampling procedure, and scoring prompts on GitHub and HuggingFace, and we invite patent attorneys, researchers, and other vendors to challenge any part of it.

• Attorneys: tell us which outcomes matter in your practice, and review our rulings on a sample of cases.
• Researchers: co-author the method paper, or run the Gauntlet on your own system.
• Vendors: run the Gauntlet and submit your results with proof of work, or tell us where the method is unfair to your product. Every vendor we score gets a right of reply, published beside its row.