Skip to content

host tests: six named steps that fail on main pass again, each fixture brought up to the change that moved it - #2024

Merged
ashhart merged 6 commits into
ashhart:mainfrom
jishnuvenugopal:test/stale-host-step-fixtures
Nov 3, 2026
Merged

ashhart merged 6 commits into
ashhart:mainfrom
jishnuvenugopal:test/stale-host-step-fixtures

Conversation

@jishnuvenugopal

@jishnuvenugopal jishnuvenugopal commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Thank you, @ashhart, for the named host steps and the recording devices behind them (mtl.Record, the fake CUDA drivers): they
check drafted trees, residency after a failed allocation and chunk grids with no GPU at all, which is what made each of these breaks
easy to pin to one commit. And thank you for the #1327 review that settled GLM's chunk grid; one of these tests just had not caught up.

What this changes

Six host steps fail on main because a later change moved code under a test fixture that was not updated with it. Each commit
updates one fixture to the change that moved it, in tests only, except one line: qwen3_5/cuda.zig now exports cuda_cache.zig
as cache, beside engine and draft, so the shared-stream test can count pendBytes instead of copying its formula.

Commit here Change on main it follows What the fixture missed
qwen35 drafter fixtures 3e88d9b: DFlash2's facts read the target's shape (copyMatch(d.e.g)) Draft fixtures never set d.e, an unset pointer
Nemotron residency fixture d643104: kv_arena, kv_tables, kv_pages, kv_pipes on the backend Metal = undefined left them undefined
Qwen27 frame replacement ede6249: a frame slot reserves 64 KiB a 1 KiB replacement fit the old slot, so nothing was replaced
GLM opt-in test 7bf9711 (#1327): GLM's step takes the cap hint as unused the test still expected the off-grid BadChunk
qwen35 Keeps fixtures 58c5085: Keeps.apply_pend, and streamBytes adds pendBytes six Keeps literals and one stream-size sum

The two Draft fixtures get a small target shape (hidden 256, one layer), so copyMatch returns 0, the target's own copy
match, as before 3e88d9b. The GLM test keeps its opt-in setup and its check that the 512-row cap reaches the driver, then asserts
the property your review settled: every chunk start stays on the 2048 grid.

Verified outcome

  • Why: six of the named test steps that check drafting, buffer residency and prompt chunking fail on main: three crash or fail,
    and three no longer compile, so the checks they were written to make have not run since late October.
  • What was done: each test's setup was brought up to date with the code change that moved under it; no engine behavior changes.
  • Result: all six steps pass (152 tests) where main has 6 crashed or failed tests and 3 builds that stop, and the repaired
    tests again catch a planted break of what they check.

Checks on this head

  • Head commit: 6f3c2c8.

  • Lean checker on the base (6a50486) and on this head: 424 and 424, 0 new.

  • Tests run, uncached (--cache-dir on an empty directory), M1 Pro, macOS 26.6:

    Step main 6a50486 this branch
    test-qwen35-draft-trees-host 1 pass, 3 crash 4 pass
    test-qwen35-draft-frame-host does not compile: missing struct field: apply_pend x3 12 pass
    test-qwen35-rounds-host does not compile: apply_pend x1 8 pass
    test-qwen35-shared-host does not compile: apply_pend x2 13 pass
    test-metal-residency-owners-host 66 pass, 1 fail, 1 crash 68 pass
    test-glm-scheduling-host 46 pass, 1 fail 47 pass
    test-qwen35-frame-invalidation-host (shares an edited file) 2 pass 2 pass
  • Each break pinned to one commit: on its parent the step passes, on the commit it fails (3e88d9b40^ 4/4 and 14/14, 3e88d9b40
    3 and 1 crash; d64310496^ 65/65, d64310496 1 crash; git bisect run from d643104 to main finds ede6249 for the Qwen27
    test). The GLM test already fails at 7bf9711, where it landed, so it is not bisectable.

  • Planted breaks, each run against the repaired step: Qwen27's replacement keeping the old buffer in the residency set (caught);
    GLM's step honoring the hint (caught: BadChunk); residency.openCache without errdefer cache.deinit (caught); streamBytes
    without pendBytes (caught). Not caught: Nemotron's residency.apply without &b.scratch; that gap was already there and this
    branch does not change it.

  • zig build test --test-timeout 10m, uncached, on main and on this branch: 8336 of 8375 tests pass, 39 skip, 0 fail on both; the
    same 55 steps fail on both, each a Metal library build that needs Xcode on this Mac.

  • CI's other steps, on main and here: test-golden 375 equal, 0 differ, 11 known on both; test-kernel-contract 138/138,
    fuzz-cortheon 19/19 and test-server-security 19/19 on both; test-pull-security stops on both at the same 44 Metal library
    builds. Plain zig build: the same 64 steps fail on both, all Metal library builds.

  • Sweep of all 393 test-* steps on main, one zig build each: 361 pass, 23 need Xcode's Metal compiler or -Dxgrammar, and 9
    fail in code: the six above, test-qwen35-live-host (cuda: tune dense 27B MLP tiles on RTX PRO 4500 #1730), and test-gradient-serving and test-qwen-serving-host, whose
    reviewed-source hashes no longer match (ck_experts.cu, gdn.cu). Those two are review pins, so this branch leaves them to you.
    The same sweep on this branch differs from main in exactly the six steps above; the other 387 keep their result.

  • zig fmt --check build.zig zig and git diff --check origin/main: clean.

  • CI: runs test, test-golden, test-kernel-contract, test-server-security, test-pull-security and fuzz-cortheon, none of
    which builds these steps, so CI cannot show this change either way.

  • Not run: CUDA devices, and Linux or Windows hosts.

Receipt

n/a: tests and one module export, nothing that reaches inference.

  • Environment: main 6a50486, Zig 0.17.0, Apple M1 Pro, macOS 26.6 (Command Line Tools, no Xcode). No checkpoint.
  • Paths and platforms: host tests only (zig/build/cuda_family_host.zig, the residency and GLM scheduling steps). No kernel.
  • Exactness, decode speed, prompt speed, licence: n/a, no engine change.

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new tests: each repaired test
    fails or does not build on main and passes here; the planted breaks above show they test their behavior again.)
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @jishnuvenugopal, this is a clean, careful fix. I checked it on current main: all five commits still apply
cleanly, the lean count stays at 424, and each of the six steps still fails there for the reason you give. The
copyMatch target, the apply_pend field, the four Nemotron KV fields, the 64 KiB frame slot and GLM's ignored cap
hint are all still on main. Nothing landed in the meantime covers them, so every fix here is still needed. Each test
still asserts what it was written for: the Qwen27 test now reaches the replacement it meant to check, and the GLM test
keeps its cap check while asserting the grid GLM actually keeps. The one production line, exporting cache so the
shared test reads pendBytes, makes sense. Thank you also for pinning each break to its commit and planting the
breaks to show the steps catch them again. We'd like to land this soon, and #2031 would keep these steps from going
stale again.

@ashhart
ashhart force-pushed the test/stale-host-step-fixtures branch from 6f3c2c8 to 3ad6592 Compare November 3, 2026 20:48
@ashhart
ashhart merged commit e8fe0b0 into ashhart:main Nov 3, 2026
3 of 7 checks passed
@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thank you @jishnuvenugopal. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

@ashhart

ashhart commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Thanks @jishnuvenugopal! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants