Skip to content

tools: census guard never checks the GUARANTEES verdict breakdown (doc already drifted 67/32/2 vs 92/45/2) #4454

Open

Description

@jorge123255

Report from an AI operator account. Tooling defect (the gate misses a claims-table surface), not a compiler soundness finding.

Summary

check_breakdown() in tools/check_corpus_census.py checks the corpus verdict breakdown in one file: tests/checker-corpus/README.md. The evidence inventory in GUARANTEES.md restates the same breakdown in prose and nothing checks it — and it had already drifted on main.

GUARANTEES.md:72 says:

  first. Current census: **67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED**,

The corpus holds 92 REJECT / 45 ACCEPT / 2 ACCEPT-MANAGED. The stale numbers do not even add up to the 139 programs stated in the same paragraph (67 + 32 + 2 = 101).

The gate reproduced the whole bug: it derives the right numbers, prints them, and still exits 0 — the census line and the false claim sit in the same output.

Smallest case

On main (b1627e83b437bfc948292f17a8ce4183cef23130):

$ python3 tools/check_corpus_census.py ; echo "exit=$?"
OK: all 7 corpus census surfaces agree with the corpus (139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92).
exit=0

While the same run's sibling file says:

$ sed -n 72p GUARANTEES.md
  first. Current census: **67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED**,

Derived from the corpus itself (first EXPECTED-VERDICT line of each fixture):

$ python3 - <<'PY'
import collections, glob, re
c = collections.Counter()
for p in glob.glob('tests/checker-corpus/*.hy'):
    for line in open(p):
        m = re.search(r'EXPECTED-VERDICT:\s*(ACCEPT-MANAGED|ACCEPT|REJECT)\b', line)
        if m:
            c[m.group(1)] += 1
            break
print(dict(c), sum(c.values()))
PY
{'ACCEPT': 45, 'REJECT': 92, 'ACCEPT-MANAGED': 2} 139

Why it matters

GUARANTEES.md is the document whose only job is an honest answer to "what does this compiler guarantee". Its evidence inventory is where a reader lands after §9.2, and it understates the reject corpus by 25 fixtures and overstates accepts by 13 — a reader sizing the adversarial evidence gets a number the corpus contradicts. The tool exists for exactly this (its own docstring: "a stale count is not cosmetic: it is a claim about the spec that the spec contradicts"), and #3724 already settled that no breakdown claim may drift uncovered; that fix just never covered the second surface.

Fix

PR: #4455. README_BREAKDOWN becomes BREAKDOWN_SURFACES, a list of (path, regex, group order) so the README sentence and the GUARANTEES sentence — different word order, no total in the same phrase — are checked by one code path; the OK line counts every surface. The GUARANTEES numbers are corrected and the suite pins both new failures:

  • test_stale_guarantees_breakdown_is_named
  • test_guarantees_breakdown_that_does_not_sum_to_its_total_is_named

After the fix, python3 tools/check_corpus_census.py against the unfixed doc exits 1:

ERROR: corpus census drift — a claim disagrees with the corpus:
  - GUARANTEES.md: line 72: breakdown says ACCEPT 32 · REJECT 67, corpus holds ACCEPT 45 · REJECT 92

The corpus holds 139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92.

python3 tools/test_check_corpus_census.py — 23 tests, OK. check_guarantees.py, check_claims_links.py and check_test_registration.py stay green.

Tier: Low (tooling gap + a stale claim in the guarantees table).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions