Repository navigation
HRX: fix the MoE router expert-id fault (engine#123) - #170
Merged
Merged
Conversation
Pins third_party/llama.cpp at the fix for engine#123: router_top8_f32 could publish its 0x7FFFFFFF no-winner seed as an expert id, which qwen3_moe_routed_gate_up_swiglu_q4k_q8 then turned into a wild weight address (HSA_STATUS_ERROR_MEMORY_FAULT / llama_decode ret=-3). Upstream PR: 1bit-MONSTER/llama.cpp#25 Re-point at the merge commit once #25 lands.
|
Docs7 for 1bit-monster/engine
Commit |
… leaves open Re-points third_party/llama.cpp from the PR head (53c0715) to the merge commit of 1bit-MONSTER/llama.cpp#25 and regenerates registry/architectures.json (the registry_pins check failed on the moved pin). docs/hrx.md: the merge commit, and that the fix turns NaN router logits into a valid-but-wrong expert instead of a fault; the NaN source stays open (#140), with the page-migration measurement that points at it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Reviewed. The root cause and fix hold up: seeding with
Worth a follow-up: now that the fault is traced to the router rather than flash attention, #148's decode-split default can be revisited. #140's nondeterminism (up to 3.66 nats between identical requests) is a separate issue and still has to be ruled out with repeated runs before the kernel goes back on. |
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 1, 2026
…ation, #170 -ot/input-layer) + held hrx-system (#786) * hw-gate: pass the gfx device library to its own configure, and recognise the zero-cases exit The gate's build failed at #772 with clang++: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library - not a kernel result and not the pin: hw-gate.sh configures third_party/llama.cpp directly, so cmake/hrx.cmake's fix (engine#713, #771) never reached it, and the kernel-corpus compile is its own amdclang++ invocation that does not inherit the llama.cpp build's CMAKE_C/CXX_FLAGS either. The box only builds because build-candidate{19,20}.sh export HIP_DEVICE_LIB_PATH into the whole run - which the workflow does not. - Read the gfx device library from the resolved toolchain (its pip tree, where the 2026-10-31 layout put it) and pass it as GGML_HRX_HIP_FLAGS="-O3 --rocm-device-lib-path=…" plus GGML_HRX_HIP_COMPILER=<toolchain>/bin/amdclang++ - the same pair cmake/hrx.cmake now passes - and export HIP_DEVICE_LIB_PATH as a fallback for anything else in the tree. - A changed flag set drops the cached CMakeCache.txt (a marker beside .gpu-kernels-pin), so an existing build tree cannot keep configuring without it. - ops: the pin carries the zero-cases guard (tests/test-backend-ops.cpp, fork engine#657) - a run that executed no case exits 1 and says "error: no test case ran (check -b/-o/-p and --test-file): refusing to report a pass". Recognise that as its own outcome: ops SKIP naming the selection that matched nothing, never a PASS (an empty selection must not look green) and never a bare FAIL that reads like a kernel fault. Verified on the box without a GPU, with HIP_DEVICE_LIB_PATH/ROCM_PATH/HIP_PATH unset: the gate's own configure arguments against /opt/rocm-therock produce all 50 gfx1151 code objects and the embedded .inc, where the same configure without the flag reproduces the CI error. bash -n clean. * Bump HRX: llama.cpp 1d32c22609d6 + held hrx-system 82804197bcbe The fork line moved two commits past 1991650479db - #169 (hrx: host-weight residency instrumentation, weak leases, and an opt-in bound, engine #669) and #170 (loader: an explicit -ot wins over the arch's lazy-read hint, LLAMA_INPUT_LAYER_ON_DEV moves the input layer) - so this cut carries the new tip 1d32c22609d6eb23d48a26ba14e680e40afd6e17. Replaces the 1991650479db cut (#772), which was gated on the previous run. - third_party/llama.cpp: 1991650479db -> 1d32c22609d6 - hrx-system: unchanged at the held 82804197bcbe (config/hrx-system-hold.config); the hold is not lifted by this PR - registry/architectures.json: regenerated against the new gitlink (tools/registry_build.py): 294 hrx / 7 npu, 0 architectures added or removed, so the only change is the recorded source pin; --check-pins says "records the committed pins" Checked on the new tip before cutting: the ggml-hrx HIP add-on hook (GGML_HRX_HIP_ADDON_DIR) is present, tools/mtmd/models/models.h exists (the file whose absence in the runner's checkout failed the first #772 gate), and test-backend-ops still carries the zero-cases guard the improved gate row now recognises. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 2, 2026
… 82804197bcbe (#810) The fork line moved three commits past the pin main carries today (1d32c22609d6, #170): d8bd0ff hrx: refuse to run when the residency budget is below the working set (engine #788) (#173) ba4fdf0 hrx: make the host-weight residency budget mean the budget (engine #788) (#172) 197e775 kv share: an optional guard page after an imported region, so a tile that writes past it faults instead of corrupting (#171) So the first two are the residency budget actually binding (and now refusing rather than over-committing), and the third gives an imported KV-share region a guard page - a mapping change, which is exactly the kind of thing the hardware gate below is for. - third_party/llama.cpp: 1d32c22609d6 -> d8bd0ff6052a - hrx-system: unchanged at the held 82804197bcbe (config/hrx-system-hold.config); the hold is not lifted here - registry/architectures.json: regenerated against the new gitlink (tools/registry_build.py reads HEAD, so this runs after the gitlink commit): 294 hrx / 7 npu, 0 architectures added or removed - the change is the recorded source pin only; --check-pins says "records the committed pins" Checked on the new tip before cutting: the GGML_HRX_HIP_ADDON_DIR hook is present (6 refs), tests/test-backend-ops.cpp still carries the zero-cases guard the gate row recognises, and tools/mtmd/models/models.h exists. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bumps
third_party/llama.cppto the fix for #123 and documents it indocs/hrx.md.Upstream: 1bit-MONSTER/llama.cpp#25
Root cause
llama_decodeintermittently died withHSA_STATUS_ERROR_MEMORY_FAULT(ret = -3) on MoE models. It was reported against the decode-split multipass path, but it is not in flash attention — disabling that dispatch only changed the layout enough to mask it. A serial-execution trace put the fault inqwen3_moe_routed_gate_up_swiglu_q4k_q8:router_top8_f32.loomseeded each lane's argmax withbest_id = 0x7FFFFFFFand only replaced it on an orderedogtor the equal-value tie-break. A lane whose candidate logits are unordered (NaN) or below-FLT_MAXpublished the sentinel verbatim intoroute_ids; consumers treat the id as bounded viaindex.assume(a hint, not a check) and computeexpert * expert_stridein 32 bits, so0x7FFFFFFF * 884736 mod 2^32 = 0xFFF28000(~4.29 GB) walked off the end of the weights buffer. The logged fault page matchedup_binding + 0xFFF28000 + 25*1152exactly.Verified
gfx1151(Strix Halo),Qwen3-Coder-30B-A3B-Instruct-Q4_K_M,-dev HRX0,llama-bench -p 0 -n 8, quiet box, on the superset tree withGGML_HRX=ON:-d 2100: 5/5 faults → post-fix 0/5-d 3000/-d 4800: 0/3 / 0/3<=2048path (-d 1500): 0/3llama-serveroutput on HRX0 == CPU backend of the same buildRe-point the pin at the PR merge commit once #25 lands.