Compact benchmark CI + non-UTF-8 detection fix

[wbarnha]

Summary

A more compact take on the features from #58 — same behavior and all shipped detection logic, with a smaller footprint: 1341 → 1297 added lines, 10 → 8 files changed.

What changed

Benchmark tooling (real redundancy — no risk to the library):

• Merged three scripts into one. benchmark.py + run.py + compare.py (349 lines, 3 files) → benchmarks/bench.py (331 lines, 1 file) with run and compare subcommands. run.py's subprocess crash-capture is now run --supervise, and parse_args / version() / _decode_equivalent are shared instead of triplicated. cchardet is imported lazily inside run, so compare still runs in the base CI Python without it installed.
• Compacted benchmark.yml. Loops over the three venvs instead of repeating each command six times, hoists the corpus SHA-256 into a workflow env var, and drops a duplicated integrity check (kept once, in the always-run verify step). Every other step is unchanged.
• Trimmed redundant docstrings in make_nonutf8_corpus.py and test_nonutf8_detection.py that restated shared context — the issue references (BUG: Incorrect Encoding Detection as Big5 #33 / Fix/pr37 artifact actions v4 #38 / Performance regression with 3.x #57), pickle-format description, decode-equivalence rationale, and threshold reasoning are all preserved.

Carried over unchanged (correctness-critical, deliberately not "compacted"): the _cchardet.pyx UTF-8 fast path + chunked handle_data, the nsMBCSGroupProber encoding-only overlay, meson.build, and the regression tests.

Validation

• Built cchardet with the overlay; full suite 130 passed, 1 skipped (including the new non-UTF-8 detection and buffer-overflow tests).
• Live-ran the run / run --supervise / compare pipeline on the generated corpus: UTF-8 mislabel rate 0.0%, table + gates render correctly, exit codes correct.
• An adversarial parity review confirmed the compacted tooling is behaviorally identical to Add benchmark CI for release performance comparisons #58, except one intentional, strictly-safer unification: the shared version() helper reports "unknown" instead of raising if package metadata is ever missing — which cannot occur in CI (every venv has faust-cchardet installed).

Notes

The bulk of the diff is the ~470-line native overlay (mostly license header + necessary detection logic), which I left intact rather than trade line count for detection-correctness risk. Further raw-size reductions would mean dropping overlapping features (e.g. the github-action-benchmark continuous check vs. the custom compare gate, or the CI non-UTF-8 benchmark vs. the unit tests) — happy to do that if wanted, but those are scope calls.

🤖 Generated with Claude Code

---

Generated by Claude Code