Repository navigation
test(ci): the heaviest host gates build in ReleaseSafe (#2242) - #2249
Merged
ashhart merged 2 commits intoNov 6, 2026
Merged
Conversation
devopsfortunato
force-pushed
the
ci/heavy-host-gates-releasesafe
branch
from
November 5, 2026 11:39
4bba4b3 to
5b68662
Compare
Owner
|
Thanks @devopsfortunato. Same safety checks, same tests, and the Qwen4 gate at 18 s instead of 4 minutes is a clear win for every contributor's local run. Queued for after the 1.2.3 cut. |
ashhart
force-pushed
the
ci/heavy-host-gates-releasesafe
branch
from
November 6, 2026 02:11
5b68662 to
9be58c6
Compare
The Qwen4 host gate and the CUDA host family's gates ran their synthetic arithmetic in Debug. ReleaseSafe keeps every runtime safety check and, on one 20-core aarch64-linux host, took the Qwen4 gate's 113 checks from 240 s to 18 s (compile 15 s to 50 s) and the CUDA family's heavy binaries from 43/42/60/18/12/10 s to 4/3/10/1/1/0.7 s, for about the same compile cost, with the whole cold suite at 5m01 -> 4m24 and 0 failures.
ashhart
force-pushed
the
ci/heavy-host-gates-releasesafe
branch
from
November 6, 2026 02:39
9be58c6 to
d4f76b9
Compare
Owner
|
Thank you @devopsfortunato. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The Qwen4 host gate and the CUDA host family's host gates ran their synthetic-graph arithmetic in Debug, so their checks spent minutes in an unoptimized path while their compile took seconds. Both now build in ReleaseSafe, which keeps every runtime safety check (integer overflow, bounds,
unreachable,std.debug.assert) and only lets the optimizer work. No test source, no engine code and no served behaviour changes.zig/qwen4_family.zigalone takes over 6 minutes on Linux and over 3 on Windows").610c67bf7.atc overlap --json: 3 changed files; the snapshot iscomplete: false(its owninvalidPullsandpullFilesLimitlimitations, soatc claimrefuses every write repo-wide). One entry reportedconflictswith qwen3-asr: host slice, the shared tower and Qwen3 schedules over a backend binding (#2105) #2239, andgit merge-tree --write-treeconfirmed it was theCHANGELOG.mdbullet alone — both pull requests prepend one under## Unreleased, andzig/build/cuda.zigauto-merged. Resolved by rebasing onto610c67bf7with both bullets kept. Nozig/file conflicts with any open pull request.Checks on this head
5b68662capython3 tools/zig/lean_check.py: base610c67bf75501 files / 425 problems, this head 5501 files / 425 problems, 0 new findings.zig build test --summary all: 545/545 steps, 7066/7125 tests passed, 59 skipped, 0 failed.roce_test: "a peer that stops makes the waiting rank time out, and abort ends a wait at once", a wait-time test that the box's load disturbs; it passes on this head.zig build test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon: exit 0. Goldens: 322/335 and 55/55 equal to the frozen corpus, 0 unexpected differences;server-security: 24 passed, 0 failed, 0 skipped.zig fmt --check build.zig zig: clean.zig22m55s,zig-windows29m45s,release-paths5m25s,zig-macos-dev-tools1m55s,fmt14s,benchmark-clients14s and 34s.The
zigjob's ownzig build teststep took 19m12s here. Same-window pull requests on this repo ranged from 9m23s to 16m52s for that same step, so runner spread and the cache a pull request cannot share (until CI: zig and zig-windows keep main's build cache, and the quick checks run first #2140 lands) dominate this signal: this run recompiles every module that importscore,lanesor a family, which is what the mode change costs once. Read CI's job time as unproven either way, and the measurements above as what the change does to the tests themselves.Receipt
The change cannot reach inference (it is a build-graph and changelog change), so the inference part of the receipt is "n/a".
0311c9280(the base the numbers were taken on; the rebase onto610c67bf7moved only the changelog bullet) + this change, Zig 0.17.0, aarch64-linux (Debian 13), 20 cores, no GPU, no checkpoint.zig build testandzig build test-cuda-host/test-qwen4-host, host-only binaries, run on Linux aarch64. Windows and macOS build the same steps from the same scripts; not run here.Run times from
zig build test --summary allon the machine above (cold cache, wall clock per step):test-qwen4-hosttest-cuda-hosttest-cuda-hosttest-cuda-hosttest-cuda-hosttest-cuda-hosttest-cuda-hostWhole suite on the same host: 5m01 s wall, 1517 s compile CPU and 426 s measured run time before; 4m24 s wall, 1661 s compile CPU and 297 s run time after. The extra compile CPU is the price of optimizing those modules once; it is paid once per cache key, and the tests themselves no longer take minutes. On a runner with fewer cores the 4-minute serial floor is what CI's
zigandzig-windowsjobs were paying, so the wall-clock gain there should be larger than the 37 s seen on 20 cores — the pull request's own job times are the check.The one exception, named:
zig/src/families/nemotron/kv_pages.zig'sholds()is abuiltin.mode-gatedstd.debug.assertprecondition, so inside these ReleaseSafe family modules it no longer runs. It is a developer precondition, not an assertion any test makes: the test totals are unchanged (7,125 both sides) and no test fails. The Debug family copies incuda.zig(check_mods, and the copies loop) still compile it in Debug for their own filtered steps. A follow-up could widen that guard to skip onlyReleaseFast/ReleaseSmall, which would change shipped-engine behaviour and does not belong in this CI-only change.Checklist