#799: Vim compat 1/9: land the extended Neovim-oracle harness (1,432 cases) with a known-deviation list and make it a CI gate - #808
Merged
JDonaghy merged 1 commit intoSep 4, 2026
Conversation
…IONS gate Grows tests/nvim_conformance.rs from 31 hand-picked cases to 1,432 across 16 areas (op, dot, undo, reg, mac, mark, search, ex, ins, vis, vb, num, scroll, word, to, misc), taken from the analysis artifact on commit 1159730. This is the only oracle-backed suite in the repo: nothing is hand-authored, so a case cannot encode the author's misconception the way the ~1,300 expectation-based Vim tests can. Result against nvim 0.9.5 (what ubuntu-24.04's apt ships, i.e. what CI runs): 1,432 cases — 788 pass, 644 deviate. No case is skipped and every case gets a real oracle answer. KNOWN_DEVIATIONS gate --------------------- The 644 deviating labels ship as `KNOWN_DEVIATIONS`, and the gate is bidirectional, so the list can only ever shrink: * an unlisted label that fails -> regression, test fails; * a listed label that passes -> the fix must delete its entry, test fails; * an entry naming no case at all -> stale, test fails. That last one matters because a stale entry would silently excuse the case if it were ever re-added. Labels are asserted unique, since they are the gate's identity. The eight follow-up issues in this chain each land by deleting lines from this list. The gate is extracted as a pure `classify()` and covered by `known_deviation_gate_is_bidirectional`, which needs no nvim and asserts all three failure directions plus the steady state — a gate never observed to fail is not a gate (#553). All three were also demonstrated end-to-end against the real corpus: deleting "scroll:H from 30" from the list produced `1 conformance REGRESSION(S) ... FAIL CUR [scroll:H from 30] cursor: nvim=(30,1) vimcode=(9,1)`; adding the passing "undo:xxx u" produced `1 case(s) listed in KNOWN_DEVIATIONS now PASS`; a bogus entry produced `1 KNOWN_DEVIATIONS entr(y/ies) match no case label`. Harness fidelity (each of these was silencing real failures) ----------------------------------------------------------- * `undolevels = -1` around the fixture write, restored to 1000 — nvim_buf_set_lines is itself an undo step, so `u` undid the fixture (41 spurious undo failures); * feedkeys mode "ntx" not "nx" — without `t` the keys are mapping-sourced, so `q` records nothing and undo is not synced between commands; * capture nvim_win_get_height and mirror it via set_viewport_lines (headless nvim reports a stable 22 rows regardless of $LINES/$COLUMNS); * ensure_cursor_visible() after placing the start cursor, because nvim_win_set_cursor scrolls and a raw engine cursor write does not; * pump macro_playback_queue after every key, as the UI does; * per-case Lua `setup` via cs(), for pinning Vim-vs-Neovim option defaults. Proof the fidelity fixes survived: `mac:qaxjq @a` and `undo:xxx u` PASS (they could not without the "ntx" and undolevels fixes), and `scroll:H from 30` produces a meaningful oracle answer — nvim (30,1) vs vimcode (9,1), a genuine deviation rather than a degenerate comparison. New here, beyond the artifact ----------------------------- * cases run across std::thread::scope workers — 1,432 serial nvim spawns is minutes of wall clock; the suite now finishes in ~12s. Output order stays deterministic (contiguous chunks, stable sort by category); * a probe is judged by whether it produced a parseable result, not by nvim's exit status: `<C-n>`/`<C-p>` leave nvim exiting non-zero even after `qa!`; * `ins:C-n/C-p completion` pin `completeopt=""` — with the default and two or more candidates nvim 0.9.x segfaults drawing the popup under `--headless -l`, so the oracle could never answer; * per-category pass/known/FAIL totals table; * CI's existing hard-fail-on-missing-nvim guard (#795) is preserved; * `CONFORMANCE_DUMP_DEVIATIONS=<path>` regenerates the list; `PROBE_FILTER` and `PROBE_VERBOSE` for triage. Docs ---- The oracle-first policy ("a Vim-behaviour PR must add oracle cases; hand-written expectations are second-class") is documented in docs/PATTERNS.md, NOT CLAUDE.md. The issue asked for CLAUDE.md, but this repo has now reverted worker edits to that file twice (f93fb3d/#657, 2d4d839/#796) — it is the coordinator's rulebook. The coordinator should fold a pointer into CLAUDE.md's Testing section; suggested wording is in the PR body. Depends on #795 (nvim installed in CI); without it this gate protects nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #799
Automated PR opened by coordinator for review of issue #799.