SYCL: MUL_MAT_ID prefill path produces wrong results on Intel Arc Pro B70 (Battlemage G31) — causes garbage output on MoE models

Description

test-backend-ops -b SYCL0 -o MUL_MAT_ID fails 28/792 cases on Intel Arc Pro B70 (Battlemage G31), all in the non-decode (prefill/batched, ne12 > 1) path of ggml_sycl_mul_mat_id(). Errors are large numerical mismatches (0.3–1.9, tolerance is 0.0005), not close-call rounding.

I originally hit this while debugging garbage/gibberish output from a real MoE model (Qwen3.6-35B-A3B) running on the SYCL backend — the same model on the Vulkan backend of the same GPU produces fully correct, coherent output at similar speed, so the corruption is SYCL-specific.

Investigation so far

Confirmed not related to any of GGML_SYCL_ENABLE_OPT, GGML_SYCL_ENABLE_VMM, GGML_SYCL_ENABLE_FUSION, GGML_SYCL_USE_ASYNC_MEM_OP — disabling each individually (and all together) does not change the failure set.
Suspected the counting-sort refactor from [SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 (mmid_counting_sort_rows / current k_copy_src1_to_contiguous) — tested by temporarily reverting ggml_sycl_mul_mat_id()'s ne12 > 1 branch to the pre-[SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 atomic-scan implementation (old k_copy_src1_to_contiguous/k_copy_dst_from_contiguous with dev_cur_src1_row atomic counter). Same failure set reproduces, and in fact gets worse — several cases that were "just" large numerical errors under the new code produce outright NaN/inf under the old code. This points to a bug in a lower-level shared path (e.g. ggml_sycl_op_mul_mat/ggml_sycl_op_mul_mat_sycl when fed the row-gathered/contiguous tensor shapes used by mul_mat_id), not in the [SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 refactor itself.
Already has SYCL: fix use-after-free bug with async memcpy in MoE prefill #24676 (use-after-free fix) and sycl: support reordered Q4_K/Q5_K/Q6_K MoE MUL_MAT_ID #24452 (reordered Q4_K/Q5_K/Q6_K MoE support) merged — neither affects this.
Decode path (ne12 == 1, ggml_sycl_mul_mat_id_mmvq_fused) is unaffected — only the prefill/batched path with more than one token per sequence.

Failing cases (764/792 pass; 28 fail)

MUL_MAT_ID(type_a=bf16,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=50,n=200,k=64)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=1,m=50,n=200,k=64)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=iq1_m,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq1_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_xxs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq3_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq3_xxs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq4_nl,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=iq4_xs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q1_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=384)
MUL_MAT_ID(type_a=q2_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q3_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q4_1,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q5_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q5_1,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q5_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)

Sample error magnitudes:

[MUL_MAT_ID] ERR = 1.564748538 > 0.000500000   MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96): FAIL
[MUL_MAT_ID] ERR = 0.327848188 > 0.000500000   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=50,n=200,k=64): FAIL

Steps to reproduce

llama-bench --list-devices   # confirm SYCL0 loaded
test-backend-ops test -b SYCL0 -o MUL_MAT_ID

Environment

GPU: Intel Arc Pro B70 (Battlemage G31)
Level Zero driver: 1.14.36605
oneAPI: 2026.1.0 (icx 2026.1.0 build 20260617)
Build flags: -DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DGGML_SYCL_DEVICE_ARCH=bmg-g31 (AOT via spir64_gen)
OS: Windows 11
All SYCL runtime env vars at default (GGML_SYCL_ENABLE_OPT=1, GGML_SYCL_ENABLE_VMM=1, GGML_SYCL_ENABLE_FUSION=1, GGML_SYCL_USE_ASYNC_MEM_OP=1) — disabling any/all of these does not change the result.

Other backends (CUDA, Vulkan, CPU) on the same build pass all MUL_MAT_ID tests cleanly.

@malakhovskiy

@malakhovskiy
I didn't reproduce the failed UT cases.
Here is my driver and code release. Please follow them：

Code:
32e41fa (HEAD -> master, tag: b9934

Driver:

dpkg -l | grep -i zero
ii  libze-dev:amd64                               1.27.0-1~24.04~ppa2                              amd64        oneAPI Level Zero -- development files
ii  libze-intel-gpu1                              26.22.38646.4-0                                  amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
ii  libze1:amd64                                  1.27.0-1~24.04~ppa2

MUL_MAT_ID UT worked fine for me , but I also have a similar issue. Currently, gemma 4 26b a4b Q5_K MOE support appears to be broken.

Tested Q4_K MOE, Q6_K MOE models work correctly. However, Q5_K MOE model start producing gibberish after several multi-turn chat.

Q5_K dense model have no issue.

Thank you.

@jlionhan

@jlionhan
Could you share the url of Q5_k MOE gguf for this issue?

Dumb theory, but does GGML_SYCL_ENABLE_DNN=0 alter anything?

@mndodd

@mndodd
In some cases of mul_mat_id, the oneDNN or oneMKL API is used to execute the gemm().
oneMKL is default enabled.

So, disable oneDNN by GGML_SYCL_ENABLE_DNN=0 won't change any thing.

I tested with

https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF/resolve/main/google_gemma-4-26B-A4B-it-Q5_K_M.gguf

https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen_Qwen3.6-35B-A3B-Q5_K_M.gguf

Models produce gibberish or inconsistent responses after several multi-turn chat or from first prompt.

I can't verify this myself but AI said "MoE routes reordered Q5_K to the DMMV path, whose reorder kernel drops half each block at WARP_SIZE=16, run with GGML_SYCL_ENABLE_OPT=0".

It seems to work fine with GGML_SYCL_ENABLE_OPT=0.

Thank you.

@jlionhan

@jlionhan
Could you try with the PR: #25690

@malakhovskiy

@mndodd Could you try with the PR: #25690

I'm unable to reproduce the original issue locally. Suspect this is a Windows vs. Linux issue. (Linux 6.17.0-35-generic here.)

I suspect you intended @malakhovskiy ?

@mndodd

@mndodd
Sorry,it's my mistake.

@mndodd Sorry,it's my mistake.

No worries. I spent a bit of time trying to reproduce this locally when I ran across this issue and am tracking it against local errors that might match.

@jlionhan

Tested PR #25690 against the MUL_MAT_ID prefill/batched failures reported above (fresh origin/master @ 12127defd, Intel Arc Pro B70). The 28/792 failing cases are unchanged before and after cherry-picking the PR's 4 commits — same test cases, same error magnitudes (a few even show NaN/inf now instead of just a large numeric error).

This PR touches ggml-sycl/dmmv.cpp (the decode-path dequantize-mul-mat-vec kernels — the K_QUANTS_PER_ITERATION row calculation and the reordered Q5_K kernel), which isn't in the code path exercised by these tests (ggml_sycl_mul_mat_id()'s ne12 > 1 prefill/batched branch, and the shared ggml_sycl_op_mul_mat/ggml_sycl_op_mul_mat_sycl path it feeds into). It looks like #25690 fixes a different, real bug (the Q5_K MoE decode-path corruption reported by @jlionhan above), but not the one in this issue.

@jlionhan Could you try with the PR: #25690

It works well for my use case.

Hello, author of #25690 here. Another data point to confirm what's already been described: on Linux with B70, I wasn't able to reproduce the MUL_MAT_ID failures with either commit 00fa7cb (prior to the PR) or 45ee550 (the HEAD of the PR).

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

SYCL: MUL_MAT_ID prefill path produces wrong results on Intel Arc Pro B70 (Battlemage G31) — causes garbage output on MoE models #25455

Description

Investigation so far

Failing cases (764/792 pass; 28 fail)

Steps to reproduce

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SYCL: MUL_MAT_ID prefill path produces wrong results on Intel Arc Pro B70 (Battlemage G31) — causes garbage output on MoE models #25455

Description

Description

Investigation so far

Failing cases (764/792 pass; 28 fail)

Steps to reproduce

Environment

Activity

arthw commented on Jul 9, 2026

jlionhan commented on Jul 12, 2026

NeoZhangJianyu commented on Jul 14, 2026

mndodd commented on Jul 14, 2026

NeoZhangJianyu commented on Jul 14, 2026

jlionhan commented on Jul 14, 2026

arthw commented on Jul 15, 2026

mndodd commented on Jul 15, 2026

arthw commented on Jul 15, 2026

mndodd commented on Jul 15, 2026

malakhovskiy commented on Jul 15, 2026

jlionhan commented on Jul 15, 2026

malsbat commented on Jul 15, 2026

github-actions commented on Aug 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions