Fix PATENTS ground truths: regenerate query3 from released data, de-ambiguate query1/query2

[shreyashankar]

Summary

All three PATENTS queries have a 0% pass rate across the 270 recorded runs in submissions/ and leaderboard_submissions/, including the top leaderboard agents. Auditing the ground truths against the released data found two distinct problems, both fixed here:

• query3: the ground truth depends on multi-assignee data that is not present in the released database — regenerated from the released data.
• query1 / query2: the ground truths are internally consistent but encode arbitrary, unstated implementation conventions that no reasonable agent would guess — regenerated under principled conventions. The query texts are unchanged; the non-default conventions are documented in db_description_withhint.txt.

The PR also recomputes Pass@1 for every leaderboard row under the new validators — from the submission JSONs stored in the repo, plus the JSONs fetched from each submission PR branch (refs/pull/<N>/head) for rows not stored on main — and updates the README and website (docs/) leaderboards accordingly.

query3: ground truth regenerated

The original ground truth follows the classic BigQuery patents-public-data citation join:

1. Cited side: publications whose assignee_harmonized contains the name UNIV CALIFORNIA (exact match — UNIV CALIFORNIA AT SAN DIEGO is excluded, which is why CALIFORNIA INST OF TECHN / G01V was not in the ground truth even though its patent US-2005165588-A1 cites US-6237292-B1).
2. Citing side: publications whose citation[].publication_number equals a cited publication number, with every name in the citing patent's assignee_harmonized reported (excluding UNIV CALIFORNIA).
3. CPC title: the citing patent's CPC entries with first = true, truncated to the 4-character subclass and joined to cpc_definition.titleFull.

Applying exactly this join to the released data finds three citing publications:

Citing publication	Rendered assignee	Cites	Primary CPC
US-10615444-B2	BLOOM ENERGY CORP	US-6767662-B2 (UNIV CALIFORNIA)	H01M
US-9447521-B2	CRYSTAL IS INC	US-2010025717-A1 (UNIV CALIFORNIA)	C30B
US-9437430-B2	SCHOWALTER LEO J	US-2010025717-A1 (UNIV CALIFORNIA)	H01L

All three resulting (assignee, titleFull) pairs were in the original ground truth and the derivation produces zero pairs outside it, confirming the join logic. However it covers only 3 of the original 28 pairs.

The released SQLite table keeps one assignee per publication: the Patents_info natural-language summary renders a single name from what was originally the multi-valued assignee_harmonized array, and no other column carries assignee data. This loses information on both sides of the join:

• Cited side. The original ground-truth citing patents cite UC patents whose released rows render a different co-assignee, so nothing identifies them as UC patents: US-6245064-B1 is rendered as ATRIONIX INC (co-assigned to The Regents of the University of California; cited by the US-8932208-B2 group and by VIVANT MEDICAL INC) and US-2006293730-A1 is rendered as RUBINSKY BORIS (co-assigned to UC; cited by FARAPULSE INC). The full text of US-6245064-B1 contains no mention of California or the Regents, so the link cannot be recovered from unstructured fields either.
• Citing side. Original rows such as MAQUET CARDIOVASCULAR LLC and CRYSTAL IS INC | H01L are co-assignees of citing patents whose released rows render only one name (KENDALE AMAR, SCHOWALTER LEO J). MAQUET CARDIOVASCULAR LLC appears nowhere in any Patents_info string.

Since validate.py requires every ground-truth pair to appear in the answer, no answer computed from the released databases could pass — consistent with the observed 0/270 runs.

Fix applied

query3/ground_truth.csv and the ground-truth list in query3/validate.py were regenerated from the released data (the three pairs above). The validator still only requires ground-truth pairs to be present, so answers that additionally include near-miss pairs (e.g. CALIFORNIA INST OF TECHN via the UNIV CALIFORNIA AT SAN DIEGO patent) continue to pass.

An alternative fix would be re-releasing the database with the full assignee lists (e.g. an assignee_harmonized column analogous to inventor_harmonized) and keeping the original 28-pair ground truth.

query1 / query2: ground truths regenerated under principled conventions

The original ground truths reproduce exactly from the released data, but only with conventions that look like implementation artifacts rather than deliberate choices:

• each CPC entry flagged first = true was counted separately, including literal duplicate entries in the released JSON (some patents list the same code with first: true twice, and both counted);
• the EMA was computed only over years that have at least one filing, so gap years carried no decay (e.g. 1929 and 1957 treated as adjacent observations);
• the EMA was seeded at zero rather than with the first observation (the textbook recursive definition and the pandas.ewm(adjust=False) behavior).

A rigorous solver making the standard choices — count distinct patents, treat missing years as zero filings, seed with the first value — gets a different answer and fails validation.

Fix applied

Both ground truths were regenerated with the principled conventions: count each patent once per CPC group (primary first = true codes, deduplicated per patent); EMA over the complete year range from the group's first to last filing year with gap years as zero; EMA initialized with the first year's count; best year = argmax (earliest on tie).

The query texts are unchanged. The conventions that are not standard defaults are documented as hints in db_description_withhint.txt (use only primary first = true CPC codes counting each patent once per group; include every year from first to last filing year with empty years as zero; earliest year on ties). Under the new recipe the EMA initialization needs no hint: textbook recursive seeding and pandas' default adjust=True produce identical best years for every group in both queries.

Resulting changes: query1 goes from 50 to 72 level-5 groups with best year 2022 (39 retained); query2 keeps the same 23 level-4 groups while 7 best years change (B29, B60, F02, F16, G01, G02, H01).

Historical submissions under the new PATENTS validators

Re-validating all 270 stored runs per query:

Query	Old validators	New validators
query1	0/270	0/270
query2	0/270	1/270 (PromptQL + Opus 4.6)
query3	0/270	10/270 (Gemini-3-Pro ReAct ×3, PromptQL Opus 4.6 ×3, PromptQL Gemini 3.1 Pro ×4)

The PR-branch submissions also gain PATENTS passes under the new validators: Spacedock Opus 4.8 (q2 2/5, q3 1/5), Altimate Sonnet 4.6 (q2 1/5, q3 1/5), Spacedock Opus 4.6 (q3 1/5), Pi Coding Agent (q3 1/5). The passers are exactly the agents that performed the correct derivation from the released data and were previously penalized for the unrecoverable pairs.

Full leaderboard recomputation (README + website)

Pass@1 was recomputed for all 17 leaderboard rows by re-running each query's validate.py over the submission JSONs under the current validators. Methodology calibration: scoring the stored JSONs under the March 2026 validator snapshot reproduces the published values exactly for Claude-Opus-4.6 (0.4376) and PromptQL Gemini-3.1-Pro (0.543), confirming the aggregation is the mean over datasets of each dataset's average per-query pass rate.

Rank	Submission	Published	Recomputed
1	Altimate Code (GPT-5.5 + Sonnet 4.6) [#53]	0.6318	0.7171
2	Altimate Code (Sonnet 4.6) [#44]	0.6040	0.6822
3	Spacedock Recce (Opus 4.8) [#55]	0.6555	0.6721
4	MinusX [#50]	0.6310	0.6518
5	Pi Coding Agent (Opus 4.6) [#31]	0.5603	0.6103
6	PromptQL (Gemini 3.1 Pro) [#24]	0.543	0.6000
7	PromptQL (Opus 4.6) [#23]	0.508	0.5933
8	Spacedock Recce (Opus 4.6) [#47]	0.5773	0.5828
9	Claude-Opus-4.6 ReAct [#22]	0.4376	0.5468
10	Oracle Forge Team PaLM [#37]	0.4601	0.4721
11	Gemini-3-Pro ReAct	0.38	0.4546
12	Oracle Forge Tenacious Intelligence [#32]	0.4554	0.4464
13	GPT-5-mini ReAct	0.30	0.3647
14	GPT-5.2 ReAct	0.25	0.2991
15	Kimi-K2 ReAct	0.23	0.2925
16	Oracle Forge Team Cohere [#38]	0.128	0.1671
17	Gemini-2.5-Flash ReAct	0.09	0.1041

Ranking changes: Altimate's two entries move to #1/#2 (overtaking Spacedock Opus 4.8, whose own score also rises; Altimate's #44 PR title already reported 0.671 under relaxed validators, closely matching the recomputed 0.6822), Pi Coding Agent overtakes the PromptQL entries, and Tenacious Intelligence drops below Gemini-3-Pro ReAct (its score decreases; it has 5–7 variable trials per query, scored with per-query denominators).

Most increases come from non-PATENTS validator relaxations merged since the rows were scored ("Relax 16 validators", "Fix validation script bugs across 13 validators", etc.) — published numbers were never refreshed after those merges. The PATENTS change itself contributes passes only to Gemini-3-Pro ReAct, both PromptQL entries, Spacedock (both), Altimate Sonnet 4.6, and Pi Coding Agent. Notes: Team PaLM's submission covers 49 of 54 queries (missing queries scored 0); sources for rows not stored on main are the JSONs in their PR branches (refs/pull/<N>/head), which remain accessible even though those PRs were closed without merging the files.

Updated surfaces: README leaderboard table (re-scored, re-ranked, footnote documenting methodology) and the website (docs/data/leaderboards.json overall + both stratified per-dataset tables re-scored and re-ranked; footnote added in index.html).

Verification

• The query3 validator passes on the answer derived from the released data (including when extra near-miss pairs are present) and fails on unrelated text.
• The query1/query2 validators pass on answers derived from the released data with the documented conventions, and fail on the old answer sets.
• Ground-truth CSVs and the lists embedded in each validate.py are consistent with each other.
• Leaderboard recomputation calibrated by reproducing published scores under the March 2026 validator snapshot before applying current validators.
• README and website leaderboards re-ranked in strictly descending Pass@1 order; docs/data/leaderboards.json validated as JSON and docs/app.js syntax-checked.