Skip to content

test(ci): the heaviest host gates build in ReleaseSafe (#2242) - #2249

Merged
ashhart merged 2 commits into
ashhart:mainfrom
devopsfortunato:ci/heavy-host-gates-releasesafe
Nov 6, 2026
Merged

ashhart merged 2 commits into
ashhart:mainfrom
devopsfortunato:ci/heavy-host-gates-releasesafe

Conversation

@devopsfortunato

@devopsfortunato devopsfortunato commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

The Qwen4 host gate and the CUDA host family's host gates ran their synthetic-graph arithmetic in Debug, so their checks spent minutes in an unoptimized path while their compile took seconds. Both now build in ReleaseSafe, which keeps every runtime safety check (integer overflow, bounds, unreachable, std.debug.assert) and only lets the optimizer work. No test source, no engine code and no served behaviour changes.

Checks on this head

  • Head commit: 5b68662ca
  • Lean checker, python3 tools/zig/lean_check.py: base 610c67bf7 5501 files / 425 problems, this head 5501 files / 425 problems, 0 new findings.
  • Tests run on this head, all on one aarch64-linux host (Zig 0.17.0, 20 cores, no GPU), cold cache:
    • zig build test --summary all: 545/545 steps, 7066/7125 tests passed, 59 skipped, 0 failed.
    • Unmodified main on the same host, same command: 543/545 steps, 7065/7125 tests, 1 failed — roce_test: "a peer that stops makes the waiting rank time out, and abort ends a wait at once", a wait-time test that the box's load disturbs; it passes on this head.
    • zig build test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon: exit 0. Goldens: 322/335 and 55/55 equal to the frozen corpus, 0 unexpected differences; server-security: 24 passed, 0 failed, 0 skipped.
    • zig fmt --check build.zig zig: clean.
  • CI: green on this head — zig 22m55s, zig-windows 29m45s, release-paths 5m25s, zig-macos-dev-tools 1m55s, fmt 14s, benchmark-clients 14s and 34s.
    The zig job's own zig build test step took 19m12s here. Same-window pull requests on this repo ranged from 9m23s to 16m52s for that same step, so runner spread and the cache a pull request cannot share (until CI: zig and zig-windows keep main's build cache, and the quick checks run first #2140 lands) dominate this signal: this run recompiles every module that imports core, lanes or a family, which is what the mode change costs once. Read CI's job time as unproven either way, and the measurements above as what the change does to the tests themselves.
  • Not run, and why: macOS Metal device checks, the CUDA/HIP/XPU device gates and the Windows job — this host has no such GPU and no macOS or Windows machine. The change only selects a host test binary's optimize mode and does not touch kernels, device code or the served path; the Linux and Windows unit jobs both build these same two steps.

Receipt

The change cannot reach inference (it is a build-graph and changelog change), so the inference part of the receipt is "n/a".

  • Environment: 0311c9280 (the base the numbers were taken on; the rebase onto 610c67bf7 moved only the changelog bullet) + this change, Zig 0.17.0, aarch64-linux (Debian 13), 20 cores, no GPU, no checkpoint.
  • Paths and platforms the change reaches: zig build test and zig build test-cuda-host / test-qwen4-host, host-only binaries, run on Linux aarch64. Windows and macOS build the same steps from the same scripts; not run here.
  • Exactness: n/a — no arithmetic, kernel or request path changes; the same 7,125 tests run as before.
  • Decode speed / prompt speed: n/a, no served path.

Run times from zig build test --summary all on the machine above (cold cache, wall clock per step):

Step Binary Debug ReleaseSafe
test-qwen4-host 113 checks 240 s run + 15 s compile 18 s run + 50 s compile
test-cuda-host 546 checks 43 s 4 s
test-cuda-host 335 checks 42 s 3 s
test-cuda-host 495 checks 60 s 10 s
test-cuda-host 118 checks 18 s 1 s
test-cuda-host 67 checks 12 s 1 s
test-cuda-host 139 checks 10 s 0.7 s

Whole suite on the same host: 5m01 s wall, 1517 s compile CPU and 426 s measured run time before; 4m24 s wall, 1661 s compile CPU and 297 s run time after. The extra compile CPU is the price of optimizing those modules once; it is paid once per cache key, and the tests themselves no longer take minutes. On a runner with fewer cores the 4-minute serial floor is what CI's zig and zig-windows jobs were paying, so the wall-clock gain there should be larger than the 37 s seen on 20 cores — the pull request's own job times are the check.

The one exception, named: zig/src/families/nemotron/kv_pages.zig's holds() is a builtin.mode-gated std.debug.assert precondition, so inside these ReleaseSafe family modules it no longer runs. It is a developer precondition, not an assertion any test makes: the test totals are unchanged (7,125 both sides) and no test fails. The Debug family copies in cuda.zig (check_mods, and the copies loop) still compile it in Debug for their own filtered steps. A follow-up could widen that guard to skip only ReleaseFast/ReleaseSmall, which would change shipped-engine behaviour and does not belong in this CI-only change.

Checklist

  • The title and description match the head commit.
  • The suite fails on nothing this change introduced; the one failure seen on unmodified main is a load-dependent wait test that passes here.
  • Comments are one line, within 120 columns. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

@devopsfortunato
devopsfortunato force-pushed the ci/heavy-host-gates-releasesafe branch from 4bba4b3 to 5b68662 Compare November 5, 2026 11:39
@ashhart

ashhart commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Thanks @devopsfortunato. Same safety checks, same tests, and the Qwen4 gate at 18 s instead of 4 minutes is a clear win for every contributor's local run. Queued for after the 1.2.3 cut.

devopsfortunato and others added 2 commits November 6, 2026 02:38
The Qwen4 host gate and the CUDA host family's gates ran their synthetic
arithmetic in Debug. ReleaseSafe keeps every runtime safety check and, on
one 20-core aarch64-linux host, took the Qwen4 gate's 113 checks from
240 s to 18 s (compile 15 s to 50 s) and the CUDA family's heavy binaries
from 43/42/60/18/12/10 s to 4/3/10/1/1/0.7 s, for about the same compile
cost, with the whole cold suite at 5m01 -> 4m24 and 0 failures.
@ashhart
ashhart force-pushed the ci/heavy-host-gates-releasesafe branch from 9be58c6 to d4f76b9 Compare November 6, 2026 02:39
@ashhart
ashhart merged commit a189233 into ashhart:main Nov 6, 2026
0 of 7 checks passed
@ashhart

ashhart commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Thank you @devopsfortunato. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants