Vim compat 1/9: land the extended Neovim-oracle harness (1,432 cases) with a known-deviation list and make it a CI gate

[JDonaghy]

Part of the Vim-compatibility push. This is the first issue in a chain of 9 — it must land before the rest, because every later issue's acceptance criterion is expressed as "delete labels from this issue's known-deviation list".

Background

tests/nvim_conformance.rs is the only oracle-backed test in the repo: it runs the same keystrokes through nvim --headless and through Engine, then asserts buffer + cursor match. Neovim is the oracle, so no expectations are hand-authored. It has 31 cases.

Every other Vim test in tests/ (~1,297 tests) asserts against hand-written expectations — i.e. it encodes the author's belief about what Vim does. A shared misconception passes such a test forever. tests/operator_motions.rs:1047 test_dj_at_last_line_noop_or_delete_last is a live example: its comment states the wrong Vim behaviour, and its assertion (b.contains("aaa")) is too weak to catch either answer.

An analysis grew the harness to 1,432 cases. Result: 808 pass, 624 fail (43.6%). The failures are ordinary daily-use behaviour, not exotica.

The input artifact

The extended harness is committed on the pushed branch artifact-vim-oracle-harness (origin), commit 1159730 (tests/nvim_conformance.rs, 1,695 lines). Start from that file rather than rewriting it.

Harness fidelity fixes it contains — do not drop these

Each one was silencing real failures. Three were found only because probes produced impossible results:

Change	Why	Cost if omitted
vim.o.undolevels = -1 around the fixture write, restored to 1000	nvim_buf_set_lines is itself an undo step, so u undid the fixture and the buffer became ""	41 spurious undo failures
feedkeys(..., "ntx") not "nx"	Without t, keys count as mapping-sourced: q records nothing (every @a was a silent no-op in nvim) and undo is not synced between commands	every macro probe and most undo probes invalid
capture nvim_win_get_height(0), mirror via engine.set_viewport_lines(rows)	H/M/L/<C-d>/zt are meaningless with mismatched window heights	screen-relative motions incomparable
engine.ensure_cursor_visible() after placing the start cursor	nvim_win_set_cursor scrolls the window; a raw engine cursor write does not	12 spurious scroll failures
pump macro_playback_queue after every key	the UI normally pumps it; the harness must too	@a never executes on the vimcode side
per-case setup Lua field (cs(...) constructor)	pins a Vim-vs-Neovim default (startofline, joinspaces, nrformats, smarttab)	cannot distinguish "vimcode differs from Vim" from "Neovim differs from Vim"
PROBE_FILTER=<label-substring> env var; BUF/CUR/BUF+CUR failure tags	iteration speed and triage	—

What to do

1. Land the harness from 1159730, keeping Case backwards compatible via the c() / cs() const fn constructors.

2. Keep the per-area const arrays (CASES_OP, CASES_DOT, CASES_UNDO, CASES_REG, CASES_MAC, CASES_MARK, CASES_SEARCH, CASES_EX, CASES_INS, CASES_VIS, CASES_VB, CASES_NUM, CASES_SCROLL, CASES_WORD, CASES_TO, CASES_MISC) flattened by the runner — 1,432 cases in one literal is unworkable.

3. Add KNOWN_DEVIATIONS: &[&str], a list of currently-failing labels. The test must fail if:

   • a label in the list starts passing (so a fix is forced to delete its entry), or
   • a label not in the list fails (so regressions are caught).

   This is what lets 1,432 cases land green today and shrink monotonically.

4. Print per-category pass/fail totals in the run output.

5. Document in CLAUDE.md's Testing section: a Vim-behaviour PR must add oracle cases; hand-written expectations are second-class.

Explicitly not in scope

Fixing any of the 624 deviations. This issue lands the instrument only. The subsequent 8 issues in the chain do the fixing.

Acceptance

• cargo test --no-default-features --test nvim_conformance runs all cases and prints per-category totals.
• Green with the known-deviation list populated.
• Red if any listed label starts passing; red if any unlisted label fails. Demonstrate both in the PR — per CLAUDE.md, a test that cannot fail is not coverage (#448-F: GTK tab click (activate/close) dead when only ONE tab group exists #553).
• The macro (mac:qaxjq @a), undo (undo:xxx u) and scroll (scroll:H from 30) categories produce meaningful nvim-side results, proving the fidelity fixes survived.
• CLAUDE.md Testing section updated.

Depends on

#795 (install nvim in CI). Without it this harness runs nowhere in CI and protects nothing — the two should land together.

---

Artifacts on origin/artifact-vim-oracle-harness

Path	What
tests/nvim_conformance.rs @ 1159730	the 1,432-case harness — start here
docs/VIM_ORACLE_ANALYSIS.md @ 33668a0	full analysis, incl. the 621-row deviation table whose labels every issue in this chain cites
docs/vim-oracle-probe-run3.txt @ 33668a0	authoritative raw run (808 pass / 624 fail of 1,432)

The two docs/ files are reference material, not necessarily a merge target — this issue decides what lands on develop. The branch is not a claim on this issue; it predates any implementation work.

[JDonaghy]

Coordinator Assignment

Machine: dell64
Repo: vimcode
Files: (unspecified)
Do not touch: (nothing else in flight)

Briefing

Issue #799: Vim compat 1/9: land the extended Neovim-oracle harness (1,432 cases) with a known-deviation list and make it a CI gate

Part of the Vim-compatibility push. This is the first issue in a chain of 9 — it must land before the rest, because every later issue's acceptance criterion is expressed as "delete labels from this issue's known-deviation list".

Background

tests/nvim_conformance.rs is the only oracle-backed test in the repo: it runs the same keystrokes through nvim --headless and through Engine, then asserts buffer + cursor match. Neovim is the oracle, so no expectations are hand-authored. It has 31 cases.

Every other Vim test in tests/ (~1,297 tests) asserts against hand-written expectations — i.e. it encodes the author's belief about what Vim does. A shared misconception passes such a test forever. tests/operator_motions.rs:1047 test_dj_at_last_line_noop_or_delete_last is a live example: its comment states the wrong Vim behaviour, and its assertion (b.contains("aaa")) is too weak to catch either answer.

An analysis grew the harness to 1,432 cases. Result: 808 pass, 624 fail (43.6%). The failures are ordinary daily-use behaviour, not exotica.

The input artifact

The extended harness is committed on the local branch worktree-agent-a0ef291fdaf7cef3b, commit 1159730 (tests/nvim_conformance.rs, 1,695 lines). Start from that file rather than rewriting it.

Harness fidelity fixes it contains — do not drop these

Each one was silencing real failures. Three were found only because probes produced impossible results:

Change	Why	Cost if omitted
vim.o.undolevels = -1 around the fixture write, restored to 1000	nvim_buf_set_lines is itself an undo step, so u undid the fixture and the buffer became ""	41 spurious undo failures
feedkeys(..., "ntx") not "nx"	Without t, keys count as mapping-sourced: q records nothing (every @a was a silent no-op in nvim) and undo is not synced between commands	every macro probe and most undo probes invalid
capture nvim_win_get_height(0), mirror via engine.set_viewport_lines(rows)	H/M/L/<C-d>/zt are meaningless with mismatched window heights	screen-relative motions incomparable
engine.ensure_cursor_visible() after placing the start cursor	nvim_win_set_cursor scrolls the window; a raw engine cursor write does not	12 spurious scroll failures
pump macro_playback_queue after every key	the UI normally pumps it; the harness must too	@a never executes on the vimcode side
per-case setup Lua field (cs(...) constructor)	pins a Vim-vs-Neovim default (startofline, joinspaces, nrformats, smarttab)	cannot distinguish "vimcode differs from Vim" from "Neovim differs from Vim"
PROBE_FILTER=<label-substring> env var; BUF/CUR/BUF+CUR failure tags	iteration speed and triage	—

What to do

1. Land the harness from 1159730, keeping Case backwards compatible via the c() / cs() const fn constructors.

2. Keep the per-area const arrays (CASES_OP, CASES_DOT, CASES_UNDO, CASES_REG, CASES_MAC, CASES_MARK, CASES_SEARCH, CASES_EX, CASES_INS, CASES_VIS, CASES_VB, CASES_NUM, CASES_SCROLL, CASES_WORD, CASES_TO, CASES_MISC) flattened by the runner — 1,432 cases in one literal is unworkable.

3. Add KNOWN_DEVIATIONS: &[&str], a list of currently-failing labels. The test must fail if:

   • a label in the list starts passing (so a fix is forced to delete its entry), or
   • a label not in the list fails (so regressions are caught).

   This is what lets 1,432 cases land green today and shrink monotonically.

4. Print per-category pass/fail totals in the run output.

5. Document in CLAUDE.md's Testing section: a Vim-behaviour PR must add oracle cases; hand-written expectations are second-class.

Explicitly not in scope

Fixing any of the 624 deviations. This issue lands the instrument only. The subsequent 8 issues in the chain do the fixing.

Acceptance

• cargo test --no-default-features --test nvim_conformance runs all cases and prints per-category totals.
• Green with the known-deviation list populated.
• Red if any listed label starts passing; red if any unlisted label fails. Demonstrate both in the PR — per CLAUDE.md, a test that cannot fail is not coverage (#448-F: GTK tab click (activate/close) dead when only ONE tab group exists #553).
• The macro (mac:qaxjq @a), undo (undo:xxx u) and scroll (scroll:H from 30) categories produce meaningful nvim-side results, proving the fidelity fixes survived.
• CLAUDE.md Testing section updated.

Depends on

#795 (install nvim in CI). Without it this harness runs nowhere in CI and protects nothing — the two should land together.

[JDonaghy]

Coordinator: Assignment Complete

Machine: dell64
Status: done
Exit code: 0
Duration: 12m 36s
Log: /home/john/.coord/logs/dafe3eff6ef9.log

[JDonaghy]

Coordinator: Assignment Complete

Machine: dell64
Status: done
Exit code: 0
Duration: 3m 46s
Log: /home/john/.coord/logs/2136015577ab.log

[JDonaghy]

Coordinator: Assignment Complete

Machine: precision
Status: done
Exit code: 0
Duration: —

Summary

Verdict: approve

Review findings

Blocking findings

None.

Non-blocking concerns

• tests/nvim_conformance.rs:4429-4439: when a probe's nvim invocation itself breaks (Outcome::NvimBroke) under CI, the code always increments totals[*ci].2 (the printed "FAIL" column) and never totals[*ci].1 ("known"), even if case.label is already listed in KNOWN_DEVIATIONS. This is intentional per the adjacent comment ("never excusable via KNOWN_DEVIATIONS under CI") and does not affect the actual pass/fail gate (classify() correctly treats (known=true, passed=false) as a no-op, so the run won't fail spuriously) — it's only a minor cosmetic mismatch between the printed per-category table and the real gate outcome. Not worth blocking on.
• The issue text quotes "808 pass, 624 fail" from the original probe-run3.txt artifact; the landed harness (with the fidelity fixes baked in) measures 788 pass / 644 fail (1,432 total, verified against the file). This is well explained in the commit message (the fidelity fixes change which cases are considered pass/fail, e.g. macro/undo probes that were previously silent no-ops), so it's not a discrepancy worth chasing — just flagging that a reviewer skimming only the issue text might be confused by the different numbers in the PR.

Nits

• None beyond the above.

---

Verification performed:

• Confirmed 85f7b65 touches only docs/PATTERNS.md and tests/nvim_conformance.rs — tightly scoped to the issue, no CLAUDE.md edit (which is correctly reserved for the coordinator per repo convention and per this review's hard rule; prior commits 2d4d839/CI is advisory, not a gate: no required status checks on develop (unprotected) or main #796 and an earlier Put vimcode on the oracle loop: promote gtk/render/tui_main into vimcode_core, then seal a tests/acceptance/ suite #657 already reverted worker edits to that file, and the commit message explicitly explains the PATTERNS.md redirect and suggests wording for the coordinator to fold into CLAUDE.md).
• Extracted and counted all case labels programmatically: 1,432 total case labels across 16 CASES_* arrays (op, dot, undo, reg, mac, mark, search, ex, ins, vis, vb, num, scroll, word, to, misc), all unique, all wired into the flattened CATEGORIES table used by the runner — matches the issue's "16 areas" / "1,432 cases" requirement.
• KNOWN_DEVIATIONS has 644 entries, all unique, and every single one matches an existing case label (zero stale entries) — 788 pass + 644 known-fail = 1,432, self-consistent.
• Verified the specific proof cases cited in the PR: mac:qaxjq @A (tests/nvim_conformance.rs:1095) and undo:xxx u (:853) are present and NOT in KNOWN_DEVIATIONS (i.e. they pass, proving the "ntx" feedkeys mode and undolevels fixes survived); scroll:H from 30 (:2708) is present and IS in KNOWN_DEVIATIONS (:4071), consistent with the commit message's demonstration of the regression-direction gate.
• Read the gate implementation (classify(), Verdict, lines ~4291-4334) and the bidirectional test known_deviation_gate_is_bidirectional (lines 4558-4600): it exercises all four states (steady/regressed/improved/stale) plus the filtered-run exemption for the stale check, satisfying the "a gate that can't be observed to fail is not a gate" (#448-F: GTK tab click (activate/close) dead when only ONE tab group exists #553) bar without requiring nvim to be installed.
• Confirmed all harness-fidelity fixes described in the issue are present in the diff: vim.o.undolevels = -1/1000 around the fixture write (:113-136), feedkeys(..., "ntx") (:145-147), nvim_win_get_height captured and mirrored via engine.set_viewport_lines(rows) (:153, :280), engine.ensure_cursor_visible() after cursor placement (:285), pump() draining macro_playback_queue after every key (:200-223), and the cs() constructor for per-case Lua setup (:321-337), used e.g. for op:cc noautoindent (:409-416).
• Confirmed per-category pass/known/FAIL totals are printed (:4464-4486).
• Confirmed the CI-vs-local nvim-missing behavior (hard fail under CI env var, soft skip locally) matches Neovim conformance test silently no-ops in CI — nvim is never installed, so the only real Vim oracle always passes #795's existing guard and is unchanged in spirit from the prior 31-case version.
• Confirmed all helper APIs used (engine_with, Engine::set_viewport_lines, Engine::ensure_cursor_visible, macro_playback_queue, advance_macro_playback) already exist elsewhere in the tree — no missing symbols, nothing this diff should have added but didn't.
• No security concerns: nvim is invoked via std::process::Command with argument-array (.arg(...)), not a shell string, so embedded quotes/backslashes in case data cannot cause command injection; the Lua script content is separately escaped before being written to a temp file. Temp file names are process-id + atomic-counter qualified, safe under the new parallel (std::thread::scope) runner.
• Platform neutrality: this is a tests/ file plus a doc; no src/gtk/ or src/tui_main/ code touched, so the Platform-Neutrality Rule doesn't apply.
• Test-coverage / driver-test rule: this PR is itself the test infrastructure (adds an oracle-backed test suite), not a user-visible behaviour change to src/, so the GtkDriver/TuiDriver black-box-coverage requirement doesn't apply here — consistent with the issue's explicit "not in scope: fixing any of the 624 deviations."

Verdict: approve. The PR does exactly what #799 asks — lands the harness verbatim in spirit from the 1159730 artifact, preserves every fidelity fix, adds a properly bidirectional and independently-tested KNOWN_DEVIATIONS gate, prints per-category totals, and documents the oracle-first policy in the correct (non-coordinator-owned) doc. Two minor non-blocking observations noted above; nothing rises to a blocking level.

[JDonaghy]

Review Complete — ✅ Approved

Reviewer findings could not be posted directly to PR #808 (gh pr review was rejected — likely a self-review restriction). Findings are reproduced here.

Note: a completion comment was posted earlier but findings could not be extracted at that time. These are the retroactive findings.

Blocking findings

None.

Non-blocking concerns

• tests/nvim_conformance.rs:4429-4439: when a probe's nvim invocation itself breaks (Outcome::NvimBroke) under CI, the code always increments totals[*ci].2 (the printed "FAIL" column) and never totals[*ci].1 ("known"), even if case.label is already listed in KNOWN_DEVIATIONS. This is intentional per the adjacent comment ("never excusable via KNOWN_DEVIATIONS under CI") and does not affect the actual pass/fail gate (classify() correctly treats (known=true, passed=false) as a no-op, so the run won't fail spuriously) — it's only a minor cosmetic mismatch between the printed per-category table and the real gate outcome. Not worth blocking on.
• The issue text quotes "808 pass, 624 fail" from the original probe-run3.txt artifact; the landed harness (with the fidelity fixes baked in) measures 788 pass / 644 fail (1,432 total, verified against the file). This is well explained in the commit message (the fidelity fixes change which cases are considered pass/fail, e.g. macro/undo probes that were previously silent no-ops), so it's not a discrepancy worth chasing — just flagging that a reviewer skimming only the issue text might be confused by the different numbers in the PR.

Nits

• None beyond the above.

---

Verification performed:

• Confirmed 85f7b6589289252101c30dd7b554efaff2afbf0a touches only docs/PATTERNS.md and tests/nvim_conformance.rs — tightly scoped to the issue, no CLAUDE.md edit (which is correctly reserved for the coordinator per repo convention and per this review's hard rule; prior commits 2d4d839/CI is advisory, not a gate: no required status checks on develop (unprotected) or main #796 and an earlier Put vimcode on the oracle loop: promote gtk/render/tui_main into vimcode_core, then seal a tests/acceptance/ suite #657 already reverted worker edits to that file, and the commit message explicitly explains the PATTERNS.md redirect and suggests wording for the coordinator to fold into CLAUDE.md).
• Extracted and counted all case labels programmatically: 1,432 total case labels across 16 CASES_* arrays (op, dot, undo, reg, mac, mark, search, ex, ins, vis, vb, num, scroll, word, to, misc), all unique, all wired into the flattened CATEGORIES table used by the runner — matches the issue's "16 areas" / "1,432 cases" requirement.
• KNOWN_DEVIATIONS has 644 entries, all unique, and every single one matches an existing case label (zero stale entries) — 788 pass + 644 known-fail = 1,432, self-consistent.
• Verified the specific proof cases cited in the PR: mac:qaxjq @a (tests/nvim_conformance.rs:1095) and undo:xxx u (:853) are present and NOT in KNOWN_DEVIATIONS (i.e. they pass, proving the "ntx" feedkeys mode and undolevels fixes survived); scroll:H from 30 (:2708) is present and IS in KNOWN_DEVIATIONS (:4071), consistent with the commit message's demonstration of the regression-direction gate.
• Read the gate implementation (classify(), Verdict, lines ~4291-4334) and the bidirectional test known_deviation_gate_is_bidirectional (lines 4558-4600): it exercises all four states (steady/regressed/improved/stale) plus the filtered-run exemption for the stale check, satisfying the "a gate that can't be observed to fail is not a gate" (#448-F: GTK tab click (activate/close) dead when only ONE tab group exists #553) bar without requiring nvim to be installed.
• Confirmed all harness-fidelity fixes described in the issue are present in the diff: vim.o.undolevels = -1/1000 around the fixture write (:113-136), feedkeys(..., "ntx") (:145-147), nvim_win_get_height captured and mirrored via engine.set_viewport_lines(rows) (:153, :280), engine.ensure_cursor_visible() after cursor placement (:285), pump() draining macro_playback_queue after every key (:200-223), and the cs() constructor for per-case Lua setup (:321-337), used e.g. for op:cc noautoindent (:409-416).
• Confirmed per-category pass/known/FAIL totals are printed (:4464-4486).
• Confirmed the CI-vs-local nvim-missing behavior (hard fail under CI env var, soft skip locally) matches Neovim conformance test silently no-ops in CI — nvim is never installed, so the only real Vim oracle always passes #795's existing guard and is unchanged in spirit from the prior 31-case version.
• Confirmed all helper APIs used (engine_with, Engine::set_viewport_lines, Engine::ensure_cursor_visible, macro_playback_queue, advance_macro_playback) already exist elsewhere in the tree — no missing symbols, nothing this diff should have added but didn't.
• No security concerns: nvim is invoked via std::process::Command with argument-array (.arg(...)), not a shell string, so embedded quotes/backslashes in case data cannot cause command injection; the Lua script content is separately escaped before being written to a temp file. Temp file names are process-id + atomic-counter qualified, safe under the new parallel (std::thread::scope) runner.
• Platform neutrality: this is a tests/ file plus a doc; no src/gtk/ or src/tui_main/ code touched, so the Platform-Neutrality Rule doesn't apply.
• Test-coverage / driver-test rule: this PR is itself the test infrastructure (adds an oracle-backed test suite), not a user-visible behaviour change to src/, so the GtkDriver/TuiDriver black-box-coverage requirement doesn't apply here — consistent with the issue's explicit "not in scope: fixing any of the 624 deviations."

Verdict: approve. The PR does exactly what #799 asks — lands the harness verbatim in spirit from the 1159730 artifact, preserves every fidelity fix, adds a properly bidirectional and independently-tested KNOWN_DEVIATIONS gate, prints per-category totals, and documents the oracle-first policy in the correct (non-coordinator-owned) doc. Two minor non-blocking observations noted above; nothing rises to a blocking level.