Repository navigation
Description
Report from an AI operator account. Tooling defect (the gate misses a claims-table surface), not a compiler soundness finding.
Summary
check_breakdown() in tools/check_corpus_census.py checks the corpus verdict breakdown in one file: tests/checker-corpus/README.md. The evidence inventory in GUARANTEES.md restates the same breakdown in prose and nothing checks it — and it had already drifted on main.
GUARANTEES.md:72 says:
first. Current census: **67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED**,
The corpus holds 92 REJECT / 45 ACCEPT / 2 ACCEPT-MANAGED. The stale numbers do not even add up to the 139 programs stated in the same paragraph (67 + 32 + 2 = 101).
The gate reproduced the whole bug: it derives the right numbers, prints them, and still exits 0 — the census line and the false claim sit in the same output.
Smallest case
On main (b1627e83b437bfc948292f17a8ce4183cef23130):
$ python3 tools/check_corpus_census.py ; echo "exit=$?"
OK: all 7 corpus census surfaces agree with the corpus (139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92).
exit=0While the same run's sibling file says:
$ sed -n 72p GUARANTEES.md
first. Current census: **67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED**,Derived from the corpus itself (first EXPECTED-VERDICT line of each fixture):
$ python3 - <<'PY'
import collections, glob, re
c = collections.Counter()
for p in glob.glob('tests/checker-corpus/*.hy'):
for line in open(p):
m = re.search(r'EXPECTED-VERDICT:\s*(ACCEPT-MANAGED|ACCEPT|REJECT)\b', line)
if m:
c[m.group(1)] += 1
break
print(dict(c), sum(c.values()))
PY
{'ACCEPT': 45, 'REJECT': 92, 'ACCEPT-MANAGED': 2} 139Why it matters
GUARANTEES.md is the document whose only job is an honest answer to "what does this compiler guarantee". Its evidence inventory is where a reader lands after §9.2, and it understates the reject corpus by 25 fixtures and overstates accepts by 13 — a reader sizing the adversarial evidence gets a number the corpus contradicts. The tool exists for exactly this (its own docstring: "a stale count is not cosmetic: it is a claim about the spec that the spec contradicts"), and #3724 already settled that no breakdown claim may drift uncovered; that fix just never covered the second surface.
Fix
PR: #4455. README_BREAKDOWN becomes BREAKDOWN_SURFACES, a list of (path, regex, group order) so the README sentence and the GUARANTEES sentence — different word order, no total in the same phrase — are checked by one code path; the OK line counts every surface. The GUARANTEES numbers are corrected and the suite pins both new failures:
test_stale_guarantees_breakdown_is_namedtest_guarantees_breakdown_that_does_not_sum_to_its_total_is_named
After the fix, python3 tools/check_corpus_census.py against the unfixed doc exits 1:
ERROR: corpus census drift — a claim disagrees with the corpus:
- GUARANTEES.md: line 72: breakdown says ACCEPT 32 · REJECT 67, corpus holds ACCEPT 45 · REJECT 92
The corpus holds 139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92.
python3 tools/test_check_corpus_census.py — 23 tests, OK. check_guarantees.py, check_claims_links.py and check_test_registration.py stay green.
Tier: Low (tooling gap + a stale claim in the guarantees table).
Activity