Skip to content

vms-562: mmk_drive — key completion on the DCL-computed marker, not MMK self-exit timing - #502

Merged
baron-3dl merged 1 commit into
mainfrom
vms-562-mmk-drive-flake
Aug 13, 2026
Merged

baron-3dl merged 1 commit into
mainfrom
vms-562-mmk-drive-flake

Conversation

@baron-3dl

@baron-3dl baron-3dl commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem

tests/qemu/test_syssvc_mmk_drive (self-host spine #4) asserted MMK.EXE reaped within a tight grace:

MMK.EXE completed (it detected the MMK____status= end-of-command marker and exited, rather than deadlocking in $HIBER)

Under contended TCG (the Kernel Executive job runs ~76 suites in ONE QEMU VM near a 120s wall), MMK echoes the DCL-computed marker OVMXB23:42 and drives the build correctly, but its teardown-and-exit lags the marker — so the reap assertion reddens and the whole Kernel Executive (vms.ko via /dev/vms, QEMU) gate goes red on unrelated PRs (bit #499, #492).

Fix — key success on the marker, but wait for MMK's clean rundown

Like spine #6 (test_syssvc_mmk_build, vms-d1b), success is keyed on the proof captured — the DCL-computed marker OVMXB23:42 echoed back over the mailbox drive (the independent oracle) — not on MMK's self-exit timing. DRIVE_TIMEOUT_MS unchanged (25s); no timeout bump.

Teardown hygiene against the REAL shared executive (Rule 9): the parent still waits (bounded, unasserted) for MMK to exit on its own. MMK $CREMBXed its command + result mailboxes, armed a write-attention AST, and lib$spawned a persistent DCL subprocess, and tears all of that down in close_subprocess()/sp_close() ($FORCEX + $DELPRC the DCL, $DASSGN both mailboxes) as its last act before exiting. SIGKILLing MMK the instant the marker appears skipped that teardown — orphaning the DCL grandchild (which held this suite's stdout FIFO open and sat blocked on the leaked mailbox) and leaking the mailboxes/AST into the executive the next suite shares — deterministically wedging test_syssvc_procnam and stalling the run at ~60/76 at any wall. Now the parent polls for the marker while draining MMK's output until MMK exits on its own, letting sp_close() reclaim everything; SIGKILL is a last resort reached only by a genuine wedge, which occurs only under the isolated one-defect-per-boot negctl.

Discriminating power preserved

A genuine mid-drive \$HIBER deadlock echoes no markergot_marker stays 0 → the bound elapses → fails hard. Negctl facility_defects.sh mmk-drive-command-not-sent (zero command bytes → wedged DCL → no marker) still reddens via the untouched require_fail assertion; its knock_on_fail (the removed reap assertion) is cleared so the observed red set equals require_fail + knock_on_fail exactly (manifest selftest passes). No-executive honest-skip branch unchanged (Rule 9 / INV-6).

Proof

Built tests/qemu/Dockerfile from a clean git archive HEAD (414632aa) and ran the FULL Kernel Executive suite under pure TCG:

=== FINAL RESULTS: 76 suites passed, 0 suites failed ===

76/76 — the run reaches test_syssvc_procnam and everything after mmk_drive (the old teardown-leak stall at ~60/76 is gone), with 0 failed assertions. CI green-by-SHA on this commit for Build & Test + Kernel Executive.

🤖 Generated with Claude Code

…for MMK's clean rundown (no leaked executive resources)

test_syssvc_mmk_drive gated on MMK.EXE reaping within a tight grace
("MMK.EXE completed ... rather than deadlocking in $HIBER"). Under
contended TCG (the Kernel Executive job runs ~76 suites in one QEMU VM
near a 120s wall) MMK echoes the DCL-computed marker OVMXB23:42 and drives
the build correctly, but its teardown-and-exit lags the marker → the reap
assertion reddens → the whole Kernel Executive gate goes red on unrelated
PRs (#499, #492).

Fix, keying on the PROOF like spine #6 (test_syssvc_mmk_build, vms-d1b):
success is the marker echoed back over the mailbox drive (the DCL-computed
independent oracle), NOT MMK's self-exit timing. A genuine mid-drive
$HIBER deadlock echoes no marker → got_marker stays 0 → the bound elapses
→ fails HARD. The negctl (facility_defects.sh mmk-drive-command-not-sent,
zero command bytes → wedged DCL → no marker) still reddens via the
require_fail assertion; its knock_on_fail (the removed reap assertion) is
cleared so the observed red set still equals require_fail + knock_on_fail
exactly.

Round 2 (teardown hygiene against the REAL shared executive, Rule 9):
success is keyed on the marker, but the parent STILL WAITS (bounded,
unasserted) for MMK to exit on its OWN. MMK $CREMBXed its command + result
mailboxes, armed a write-attention AST, and lib$spawned a persistent DCL
subprocess, and it tears all of that down in close_subprocess()/sp_close()
($FORCEX + $DELPRC the DCL, $DASSGN both mailboxes) as its last act before
exiting. SIGKILLing MMK the instant the marker appears skipped that
teardown, orphaning the DCL grandchild (which held this suite's stdout
FIFO open and sat blocked on the leaked mailbox) and leaking the
mailboxes/AST into the executive the next suite shares — deterministically
wedging the suite AFTER this one (test_syssvc_procnam), stalling the whole
run at ~60/76 at any wall. Now the parent polls for the marker while
draining MMK's output until MMK exits on its own, letting sp_close()
reclaim everything; SIGKILL is a last resort reached only by a genuine
wedge, which occurs only under the isolated one-defect-per-boot negctl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@baron-3dl
baron-3dl force-pushed the vms-562-mmk-drive-flake branch from 5efcc92 to 414632a Compare August 13, 2026 22:48
@baron-3dl
baron-3dl merged commit 01d2184 into main Aug 13, 2026
76 checks passed
baron-3dl added a commit that referenced this pull request Aug 15, 2026
…es to TCG variance) (#584)

The single Kernel Executive job booted ONE QEMU VM running all ~79 executive
suites sequentially under a fixed 600s whole-VM wall. A pathologically slow
(~10x) GitHub TCG runner reached only 60/79 in 600s and WALLED, reddening
proven work non-deterministically (#502 failed 3x at escalating walls). A
fixed wall buys only linear headroom against unbounded runner variance, so it
cannot fix the flake; raising it again is rejected (Rule 8).

Partition the suites into 6 shards, each a separate CI matrix leg booting its
own QEMU VM over ~1/6 of the suites (~11-16/shard, measured) under the SAME
unchanged 600s wall -- comfortably covered even at 10x-slow. The assignment is
by name-hash: shard(name) = (first 6 hex of md5(name)) mod N, computed
identically in tests/qemu/init.sh (busybox ash) and in the ci.yml verification
(GNU bash) -- proven byte-identical for all 79 suites. It is a residue-class
partition: every suite lands in exactly one shard, the union is the whole set,
no gap and no overlap. No suite dropped; coverage unchanged, only spread.

- tests/qemu/run_tests.sh: read SHARD_INDEX/SHARD_TOTAL env (default 0/1 =
  run everything, so the negative-control image and bare `docker run` are
  unaffected), forward them onto the guest kernel cmdline as
  ovmx.shard/ovmx.shards. Wall unchanged at 600s.
- tests/qemu/init.sh: parse ovmx.shard/ovmx.shards from /proc/cmdline and run
  only this VM's residue class. Malformed/empty falls back to run-all.
- .github/workflows/ci.yml: the positive KE job becomes `kernel-executive-shard`
  (6-way matrix), each leg asserting only its assigned suites' verdict lines +
  both module-load lines + `0 suites failed`. A NEW aggregate job
  `kernel-executive` KEEPS THE EXACT ORIGINAL required-check name
  'Kernel Executive (vms.ko via /dev/vms, QEMU)', needs all shards, and is
  green iff every shard is green (if: always() so a failed shard reddens it
  rather than skipping it green). The aggregate also runs a union/coverage
  proof: every suite maps to exactly one shard, sum over shards == total.

Preserves the gate's teeth: the negative-control job (job 3c, unsharded) still
runs the full set with the executive absent and must red; a dropped/failed
suite in shard K reds shard K -> reds the aggregate.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant