Skip to content

vms-6a2: give the corpus conformance gate real teeth (floor 90/229, vacuity guard, path fix) - #492

Merged
baron-3dl merged 1 commit into
mainfrom
vms-6a2-corpus-baseline
Aug 13, 2026
Merged

baron-3dl merged 1 commit into
mainfrom
vms-6a2-corpus-baseline

Conversation

@baron-3dl

Copy link
Copy Markdown
Contributor

Problem (vms-6a2)

The corpus conformance gate was documented as a falsifiable floor but had real gaps:

  1. Vacuity risk — a total=0 baseline/report makes the whole gate exit 0 forever (an empty floor can never be undercut). This is the original toothless-gate defect the item was filed on; nothing mechanically prevented the baseline reverting to total=0.
  2. Container-path hardcodingrun_corpus.sh hardcoded /src/... for includes, -L, and LD_LIBRARY_PATH, so it only ran correctly inside the CI docker mount, not from a plain host checkout/worktree.
  3. Flaky boundary count — the run-pass count flapped (90 vs 86 across back-to-back runs), so a floor set at the high value would redden CI intermittently.

What changed

tests/conformance/run_corpus.sh

  • Derive REPO_ROOT from SCRIPT_DIR (= /src in CI, unchanged) instead of hardcoding /src; adds OVMX_BUILD_LIB_DIR override. Runs from any checkout/worktree now.
  • Redirect program stdin from /dev/null. The input routines (lib$get_input/get_command/get_foreign/lookup_key) flapped run-pass↔run-fail on an inherited tty. /dev/null = the closed stdin CI already gives, so the measurement is deterministic (root-cause fix, not a retry/timeout band-aid).
  • Vacuity guard: hard-fail (exit 2) when total==0.

.github/workflows/ci.yml

  • Floor step rejects total==0 on the report or the committed baseline (independent backstop to run_corpus.sh's exit 2).

tests/conformance/corpus_baseline.json

  • Real measurement against current HEAD libs, floor ratcheted 89 → 90 run-pass / 229 total (compile-fail 66, link-fail 52, run-fail 7, run-crash 14). Timestamped, per-program status. Stable over 5 runs after the stdin fix.

Real measured numbers (current HEAD, x86_64 workshop)

metric value
total 229
compile-fail 66
link-fail 52
run-pass (the conformance number) 90 (39%)
run-fail 7
run-crash 14
compile-or-better 111 / 229 (48%)

Note for the record: the closed vms-801.4 "80%+ compile and run" claim is not backed by the tree — real run-pass is 39%, compile-or-better 48%. The committed baseline now carries the honest number either way.

Proof the gate has teeth (all foreground, exit codes read directly)

control result
current state (positive) run_corpus.sh exit 0, floor held 90 ≥ 90
per-entry regression run_corpus.sh exit 1 (faked a baseline run-pass entry → flagged lib_attach)
aggregate floor CI step fails (current 90 < simulated floor 91)
total==0 (report) run_corpus.sh exit 2 + CI floor step exit 1
total==0 (baseline) CI floor step exit 1

Runs clean from a fresh worktree checkout (proved here — the worktree is not at /src), and CI behavior is unchanged since REPO_ROOT resolves to /src there.

🤖 Generated with Claude Code

The corpus gate was documented as falsifiable but had gaps the item flagged:
a total=0 baseline would make it vacuous, run_corpus.sh hardcoded /src paths
(only runnable inside the CI container), and the run-pass count was flaky at
the boundary so any floor set to the high value would redden CI intermittently.

Changes:
- run_corpus.sh: derive REPO_ROOT from SCRIPT_DIR (= /src in CI, unchanged)
  instead of hardcoding /src for includes, -L, and LD_LIBRARY_PATH, with an
  OVMX_BUILD_LIB_DIR override. The harness now runs from any checkout/worktree.
- run_corpus.sh: redirect program stdin from /dev/null. The input routines
  (lib$get_input/get_command/get_foreign/lookup_key) flapped run-pass<->run-fail
  on an inherited tty, making the run-pass count non-deterministic (90 vs 86).
  /dev/null = the closed stdin CI already gives, so the measurement (and thus
  the floor) is honest and reproducible: 90 run-pass, stable over 5 runs.
- run_corpus.sh: hard-fail (exit 2) when total==0 — a zero-total run is never
  a conformance pass; that vacuity is the exact defect this item exists to kill.
- ci.yml: reject total==0 on the report OR the committed baseline in the floor
  step (independent backstop), and document the vacuity guard.
- corpus_baseline.json: real measurement against current HEAD libs, floor
  ratcheted 89 -> 90 run-pass / 229 total (compile-fail 66, link-fail 52,
  run-fail 7, run-crash 14). Timestamped, per-program status.

Proven teeth (all foreground, exit codes read directly):
- current state PASSES (run_corpus exit 0, floor held 90>=90);
- per-entry regression reddens run_corpus.sh (exit 1);
- aggregate floor reddens the CI step (current 90 < simulated floor 91);
- total==0 hard-errors both run_corpus.sh (exit 2) and the CI floor step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@baron-3dl
baron-3dl force-pushed the vms-6a2-corpus-baseline branch from 00d3564 to 8d0bfa3 Compare August 13, 2026 19:46
@baron-3dl
baron-3dl merged commit 3e0ccad into main Aug 13, 2026
74 of 75 checks passed
baron-3dl pushed a commit that referenced this pull request Aug 13, 2026
…for MMK's clean rundown (no leaked executive resources)

test_syssvc_mmk_drive gated on MMK.EXE reaping within a tight grace
("MMK.EXE completed ... rather than deadlocking in $HIBER"). Under
contended TCG (the Kernel Executive job runs ~76 suites in one QEMU VM
near a 120s wall) MMK echoes the DCL-computed marker OVMXB23:42 and drives
the build correctly, but its teardown-and-exit lags the marker → the reap
assertion reddens → the whole Kernel Executive gate goes red on unrelated
PRs (#499, #492).

Fix, keying on the PROOF like spine #6 (test_syssvc_mmk_build, vms-d1b):
success is the marker echoed back over the mailbox drive (the DCL-computed
independent oracle), NOT MMK's self-exit timing. A genuine mid-drive
$HIBER deadlock echoes no marker → got_marker stays 0 → the bound elapses
→ fails HARD. The negctl (facility_defects.sh mmk-drive-command-not-sent,
zero command bytes → wedged DCL → no marker) still reddens via the
require_fail assertion; its knock_on_fail (the removed reap assertion) is
cleared so the observed red set still equals require_fail + knock_on_fail
exactly.

Round 2 (teardown hygiene against the REAL shared executive, Rule 9):
success is keyed on the marker, but the parent STILL WAITS (bounded,
unasserted) for MMK to exit on its OWN. MMK $CREMBXed its command + result
mailboxes, armed a write-attention AST, and lib$spawned a persistent DCL
subprocess, and it tears all of that down in close_subprocess()/sp_close()
($FORCEX + $DELPRC the DCL, $DASSGN both mailboxes) as its last act before
exiting. SIGKILLing MMK the instant the marker appears skipped that
teardown, orphaning the DCL grandchild (which held this suite's stdout
FIFO open and sat blocked on the leaked mailbox) and leaking the
mailboxes/AST into the executive the next suite shares — deterministically
wedging the suite AFTER this one (test_syssvc_procnam), stalling the whole
run at ~60/76 at any wall. Now the parent polls for the marker while
draining MMK's output until MMK exits on its own, letting sp_close()
reclaim everything; SIGKILL is a last resort reached only by a genuine
wedge, which occurs only under the isolated one-defect-per-boot negctl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
baron-3dl added a commit that referenced this pull request Aug 13, 2026
…for MMK's clean rundown (no leaked executive resources) (#502)

test_syssvc_mmk_drive gated on MMK.EXE reaping within a tight grace
("MMK.EXE completed ... rather than deadlocking in $HIBER"). Under
contended TCG (the Kernel Executive job runs ~76 suites in one QEMU VM
near a 120s wall) MMK echoes the DCL-computed marker OVMXB23:42 and drives
the build correctly, but its teardown-and-exit lags the marker → the reap
assertion reddens → the whole Kernel Executive gate goes red on unrelated
PRs (#499, #492).

Fix, keying on the PROOF like spine #6 (test_syssvc_mmk_build, vms-d1b):
success is the marker echoed back over the mailbox drive (the DCL-computed
independent oracle), NOT MMK's self-exit timing. A genuine mid-drive
$HIBER deadlock echoes no marker → got_marker stays 0 → the bound elapses
→ fails HARD. The negctl (facility_defects.sh mmk-drive-command-not-sent,
zero command bytes → wedged DCL → no marker) still reddens via the
require_fail assertion; its knock_on_fail (the removed reap assertion) is
cleared so the observed red set still equals require_fail + knock_on_fail
exactly.

Round 2 (teardown hygiene against the REAL shared executive, Rule 9):
success is keyed on the marker, but the parent STILL WAITS (bounded,
unasserted) for MMK to exit on its OWN. MMK $CREMBXed its command + result
mailboxes, armed a write-attention AST, and lib$spawned a persistent DCL
subprocess, and it tears all of that down in close_subprocess()/sp_close()
($FORCEX + $DELPRC the DCL, $DASSGN both mailboxes) as its last act before
exiting. SIGKILLing MMK the instant the marker appears skipped that
teardown, orphaning the DCL grandchild (which held this suite's stdout
FIFO open and sat blocked on the leaked mailbox) and leaking the
mailboxes/AST into the executive the next suite shares — deterministically
wedging the suite AFTER this one (test_syssvc_procnam), stalling the whole
run at ~60/76 at any wall. Now the parent polls for the marker while
draining MMK's output until MMK exits on its own, letting sp_close()
reclaim everything; SIGKILL is a last resort reached only by a genuine
wedge, which occurs only under the isolated one-defect-per-boot negctl.

Co-authored-by: alice <alice@workspace.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant