Repository navigation
ggml-hrx: MoE and MLA fixes for engine #95 (GLM-4.7-Flash, Qwen3-Coder at full context) - #4
Merged
Conversation
The HRX0 flash-attention kernel caps the KV cache at 32768 tokens, so at a
MoE model's full context (262144) llama.cpp disables flash attention and the
graph falls back to the non-FA attention path. Two bugs in that path made the
chat request return HTTP 500 "Compute error.":
- tensor_storage_relative_offset rejected a zero-byte (empty) source tensor
whenever a degenerate view into it had a non-zero offset, so the graph cache
validation failed on empty per-expert FFN views ("metadata does not match").
- the fused attention-output and routed-FFN dispatches requested an in-place
buffer alias whose target (the layer output, e.g. l_out) is an external,
persistent buffer, so alias_storage failed with "value alias target is not
transient". Only request the in-place alias when both the source and target
are transient.
Also narrow the MoE router SUM_ROWS/CLAMP claim to the top-k router chain
(SOFT_MAX -> ARGSORT -> GET_ROWS), so a sigmoid router (GLM-4.7-Flash) leaves
those nodes to another backend instead of an unsupported SUM_ROWS on HRX0.
Verified: serve_e2e answers "Paris" on Qwen3-Coder-30B-A3B-Instruct Q4_K_M
and Qwen3.6-35B-A3B Q8_0 (no regression), and a 2048-token prompt returns HTTP
200 (was 500) on both Qwen3-Coder-30B-A3B and GLM-4.7-Flash.
HRX0 declined flash attention for two MoE shapes, so llama.cpp fell back to the non-FA path: - GLM-4.7-Flash (deepseek2/MLA) has key_length=576, above the FA kernel's 512 LDS head-size cap. Raise the cap to 576 and, because MLA pads the KV cache row to key_length, match the value against the QK head-size stride and pass the value stride to the kernel. - Qwen3-Coder-30B-A3B (qwen3moe) has a 262144 context, above the FA matcher's 32768 KV cap. Raise the prefill/decode KV caps and the kernels' KV loop bounds to 262144 so FA is claimed at the model's full context. Verified: serve_e2e answers "Paris" on Qwen3-Coder-30B-A3B Q4_K_M and Qwen3.6-35B-A3B Q8_0 (no regression); GLM-4.7-Flash loads with FA enabled and a 2048-token prompt returns HTTP 200 on Qwen3-Coder via FA.
llama.cpp's route ids are a [n_expert_used, n_tokens] view of the [n_expert, n_tokens] argsort, so the stride between a token's route ids is n_expert, not n_expert_used. The mul_mat_id routing bundle assumed route_stride == route_count, so the expert-table builder read the wrong route ids for batched prompts and the MoE FFN produced corrupt outputs (GLM-4.7-Flash answered garbled at the default ubatch). Read route_stride from route_ids->nb[1] and size the route-ids span as (token_count - 1) * route_stride + route_count. With this, GLM-4.7-Flash answers "Paris" in serve_e2e at the default ubatch.
…witches
GGML_HRX_DISABLE_DISPATCH takes comma-separated substrings of dispatch
registration names to skip ("fused" skips every fused registration);
GGML_HRX_LOG_DISPATCH=1 prints each registration the first time it
matches. Bisecting with them found the MUL_MAT_ID route-stride bug (#95).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
value_stride is the value's row stride in elements, n_kv_head x head_size
(1024 for Qwen3-0.6B, 576 for GLM's MLA cache row). Its config range stopped
at 576, so every GQA model whose KV row is wider failed to compile the FA
kernel ("value_stride 1024 violates constraint 'range'", 500 on hrx).
Before value_stride the kernel used n_kv_head x value_head_size here, with
no such bound. Allow up to 64 heads x 576.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 25, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the fixes for 1bit-MONSTER/engine#95 on the engine's pinned branch:
GGML_HRX_DISABLE_DISPATCH/GGML_HRX_LOG_DISPATCHdebug switches.value_striderange covers a whole KV row. Without it, hrx-backend-test: drop the use of the deleted dispatch-symmetric-i4.h (#422) #103 broke every GQA model whose KV row is wider than 576 (Qwen3-0.6B: "value_stride 1024 violates constraint 'range'").Verified on Strix Halo at this exact tree:
test-backend-ops -b HRX0: 790/790.tests/serve_e2e.sh ... hrx: 7 of 7 answer "Paris": Qwen3-0.6B, Qwen2.5-7B, Qwen3-Coder-30B-A3B, Qwen3.6-35B-A3B Q8_0, GLM-4.7-Flash, MiniCPM4-8B, MiniCPM5-1B.🤖 Generated with Claude Code