ci(negctl): fix 0.5-integration per-facility shard reds — full-suite wall truncation - #686
Merged
Merged
Conversation
…c=0) in the exec-absent gate
The "Kernel Executive — Negative Control (proves the gate can fail)" job
was RED on the 0.5 integration (and on the flip base -- not a cascade
regression). Diagnosed on the dind rail by building the NEGATIVE_CONTROL=1
image and reading the per-suite verdicts: every test_syssvc_* returns the
honest-skip 77, vms.ko load fails (expected), vmsfs.ko loads -- and the
SOLE offender is test_kmod_ods2_codec returning rc=0, which the gate's
catch-all ("every other test_kmod_* MUST be nonzero") flagged as a faked
executive.
It is not faking anything. test_kmod_ods2_codec (vms-dcd) mounts the
real-VAX ODS-2 fixture through the "ods2ro" filesystem IN vmsfs.ko and
byte-compares a kernel-resident-codec read against a committed golden. It
depends on vmsfs.ko (which init.sh insmods even in NEGATIVE_CONTROL), NOT
on vms.ko/the executive: it opens no /dev/vms and reaches no vms_kif entry
point (its only ioctl is LOOP_SET_FD), and its own header states it
follows test_kmod_vmsfs_blkdev's skip convention. So rc=0 with the
executive absent is CORRECT and honest -- it is a positive control for the
kernel-resident ODS-2 codec, exactly like the test_kmod_vmsfs* suites.
This is the vms-1d9-documented miscategorisation (a legitimately added
executive-independent suite the gate wrongly demanded fail), NOT an INV-6
fake-success. Move it into the rc-MUST-be-0 positive-control category with
test_kmod_vmsfs*; it keeps a real verdict (must PASS -- an indiscriminate
harness failure still trips it), it is NOT allowlisted out, and the gate's
teeth are unchanged. Its executive-facing teeth live in the OTHER gate:
the per-facility negctl's ods2-read-content-vbn defect reddens this suite
against a real codec mutation.
Verified on the dind rail: gate BAD set empty -> PASS (all 64 test_syssvc_*
= 77, vmsfs+ods2_codec = 0, every other test_kmod_* nonzero, markers
present, harness exits nonzero).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nutes (0.5 suite growth) Second half of the 0.5-integration negctl reds. The per-facility "Kernel Executive — Per-Facility Negative Controls" shards 0/1/2 (and 4/5) were reported red, but the CI job data (run 32447601184 on work/vms-0.5-final-dryrun) shows they were CANCELLED, not FAILED -- killed at EXACTLY 30.2 minutes, i.e. they hit their own timeout-minutes: 30. Shard 3 alone COMPLETED (24.5m) and PASSED. So facility_defects.sh is NOT inconsistent: coverage and selftest pass, and shard 3 completing all its defects proves the manifest/gate is sound and the mutations still redden. What changed is capacity: the flip + cascade grew the suite set (~76 -> ~97 suites), so every per-defect harness boot runs more suites; at ~20 defects/shard that no longer fits in 30m under the 95-job run's runner contention (shards 0/1/2 ran in the heavily-contended first max-parallel batch). Fix the capacity, faithfully -- do NOT weaken the gate, delete a defect, or shorten the suite run. Split 6 -> 10 shards: 120 defects / 10 = ~12 each; at shard 3's measured ~1min/defect + ~5m fixed build+positive- control that is ~17m, comfortably inside 30m even under first-batch contention. The partition is `NR % strategy.job-total`, so extending the matrix list is the only change; nothing is hand-maintained and every defect still runs, in a smaller batch. The inner 600s per-boot wall in run_tests.sh is untouched (it is not hit on CI -- shard 3 ran ~1min/ defect, ~0.75s/suite, huge margin; a rail run only truncates it because a shared 2-CPU slice runs QEMU ~8x slower). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
baron-3dl
changed the title
baron-3dl
changed the title
Contributor
Author
Rail verification complete — all four failing defects now reach FINAL RESULTS with the 1800s wallEach of the four defects that failed CI run 32487119712 (all on "the harness never reached FINAL RESULTS", control reddening correctly) was re-run on the dind rail with the raised wall:
Sizing is grounded in the actual CI (GitHub TCG) truncation points: eflag-readef ran ~75/97 suites in 600s (→ ~776s full run), image-rundown ran later ~86/97 (→ ~677s). 1800s is ~2.3× the slowest, and CI boots complete at their real ~677-776s (the wall is a ceiling, not the runtime), so the 50m shard budget holds comfortably — CI's failing shards already ran 24-27m and a formerly-truncated defect only adds ~3m. No |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third and final fix in the negctl cluster on the 0.5 reap gate. The first two (test_kmod_ods2_codec exec-absent classification; 6→10 shard split for the outer 30m timeout) are already merged into the base
work/vms-0.5-final-dryrun; this PR carries the fix the split unmasked.What the split exposed
With the outer 30m timeout fixed, run 32487119712 (389d129) showed shards 1, 2, 3, 5 genuinely FAILED. Read from the CI logs, all four are the SAME thing:
eflag-readef-status-invertedlnm-manager-delete-noopeflag-dacefc-status-wrongimage-rundown-without-entryIn every case the control WORKED — the facility's own suites reddened, red set EXACTLY the manifest's. The 1 "failure" is uniformly
the harness never reached FINAL RESULTS, with ~20 later suitesNEVER RAN. It is a wall-timeout truncation, not a stale control — none of these targets a flip-retired/vmspath (eflag = kernel event flags, lnm = name manager, image-rundown = image activation); none is the rms/setuai re-anchor class.facility_defects.shneeds no reconciliation.Root cause
The per-facility negctl runs the entire ~97-suite set in one VM per defect (check 5's isolation attribution must observe every suite, so it can't shard suites like the positive job does). The 0.5 flip grew that set
76→97, and a mutation that makes event-flag$WAITFRs wait out their timeouts adds enough per-suite slack that the full run passes 600s on GitHub's slow TCG runner and truncates at ~75/97 suites. Measured on real CI: ~75 suites at the 600s wall → the full 97 need ~776s.Fix (faithful — no control weakened, no defect deleted, no suite dropped)
Exactly what
run_tests.sh's own comment prescribes: "raise TIMEOUT... do not drop or skip a suite."TIMEOUTis now env-overridable (KE_WALL_TIMEOUT), default 600 unchanged — the positive job shards suites and keeps it.KE_WALL_TIMEOUT=1800(~2.3× the measured ~776s) so a slowed boot completes instead of truncating.timeout-minutes: 30 → 50: a formerly-truncated defect now runs ~13m (completes) instead of failing at 10m; CI shard durations (failing shards ran 24-27m) confirm ~30m actual, 50m ceiling.Every control still must redden and reach FINAL RESULTS; a genuine hang (>1800s or a panic) still fails the defect, and the 50m outer bound backstops a wedged shard.
Rail verification
The rail uses KVM (a different, faster speed profile than CI's TCG, so it can't reproduce the truncation itself), but it confirms the checked behaviour — controls redden and the run reaches FINAL RESULTS with the raised wall:
eflag-readef,lnm-manager-delete,eflag-dacefcall PASS (red set EXACTLY the manifest's).eflag-dacefctruncated at 1200s on a contended 4-CPU slice, which is why the wall is 1800 not 1200. [eflag-dacefc + image-rundown @ 1800/LIM=4 result appended below]🤖 Generated with Claude Code