Docker action shouldn't run on PRs by etra0 · Pull Request #241 · ggml-org/llama.cpp

etra0 · 2023-03-17T16:36:51Z

This action would run for every PR action which means it'll create a docker image every time the CI runs on contributions.

j-f1 · 2023-03-17T17:37:36Z

IMO it makes sense to build the image so that the pipeline doesn’t unexpectedly break — since the images aren’t pushed anywhere it is safe to generate them.

etra0 · 2023-03-17T18:00:29Z

Oops, you're right, I did missed that the docker.yml conditionally pushes depending if it's a PR or a commit. Closing this.

* WIP CUDA FA with Dk != Dv * WIP * CUDA FA WIP - It actually works! No TG yet, but for PP I can run FA with fp16 cache and it gets the same answer. * CUDA FA WIP - it now works for Q8_0 + Q8_0 for KV cache * CUDA FA WIP - TG, not working yet. * CUDA FA with Dk != Dv: it works now for DeepSeek --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…nt comment - copy_to_quant.comp: turbo3 signs packing read ballot.x with a shift of (sg_lane/8)*8, which is UB on wave64 subgroups where lanes 32-63 land in ballot.y - corrupting half of every signs byte at write time on GCN/Polaris (fork commit 11a8377, found in the post-rebase sweep of TheTom/llama-cpp-turboquant, 2 commits past our backup tip). Select the ballot component by lane/32 and the byte by (lane%32)/8. - ggml-rpc.h: comment said 'upstream 97 + 5'; upstream is at 101 and already contains LIGHTNING_INDEXER and DSV4_HC_*, so our only op delta is TURBO_WHT (102 = 101 + 1). Assisted-by: Claude

…org#241) (ggml-org#271) The Windows CUDA x64, Linux CUDA and Windows HIP release jobs build with GGML_NATIVE=OFF but without GGML_BACKEND_DL/GGML_CPU_ALL_VARIANTS. With native detection off and no explicit ISA flags, ggml enables AVX/AVX2/FMA/F16C by default, so each bundle ships a single AVX2-only ggml-cpu library. On a CPU without AVX2 the process dies at startup with no message: even `llama-server --version` exits silently (ggml-org#241). The CPU and Vulkan bundles already build every variant and pick one at runtime (Vulkan since ggml-org#198). Build those three bundles with GGML_BACKEND_DL=ON and GGML_CPU_ALL_VARIANTS=ON as well. The arm64 CUDA leg already used BACKEND_DL with GGML_CPU=OFF and is unchanged. Claude-Session: https://claude.ai/code/session_012Q6ecxozUXFFFLE3N2SMiR Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…org#241) (ggml-org#271) The Windows CUDA x64, Linux CUDA and Windows HIP release jobs build with GGML_NATIVE=OFF but without GGML_BACKEND_DL/GGML_CPU_ALL_VARIANTS. With native detection off and no explicit ISA flags, ggml enables AVX/AVX2/FMA/F16C by default, so each bundle ships a single AVX2-only ggml-cpu library. On a CPU without AVX2 the process dies at startup with no message: even `llama-server --version` exits silently (ggml-org#241). The CPU and Vulkan bundles already build every variant and pick one at runtime (Vulkan since ggml-org#198). Build those three bundles with GGML_BACKEND_DL=ON and GGML_CPU_ALL_VARIANTS=ON as well. The arm64 CUDA leg already used BACKEND_DL with GGML_CPU=OFF and is unchanged. Claude-Session: https://claude.ai/code/session_012Q6ecxozUXFFFLE3N2SMiR Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 9a4dd89)

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

@slavap

…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).

Docker image shouldn't be published on PRs

c6a0f83

This action would run for every PR action which means it'll create a docker image every time the CI runs on contributions.

etra0 closed this Mar 17, 2023

j-f1 deleted the fix-docker-action branch March 17, 2023 18:36

AAbushady pushed a commit to AAbushady/llama.cpp that referenced this pull request Jan 27, 2024

Fix Makefile for CUBLAS. (ggml-org#241)

b1b8dc3

Soket1 pushed a commit to Soket1/llama.cpp-xdna that referenced this pull request Aug 22, 2026

xdna(ggml-org#241): hash source content into flowkv/relay cache identity

c4d9794

Navigation Menu

Repository navigation

Docker action shouldn't run on PRs - #241

Docker action shouldn't run on PRs#241
etra0 wants to merge 1 commit into
masterfrom
fix-docker-action

etra0 commented Mar 17, 2023

Uh oh!

j-f1 commented Mar 17, 2023

Uh oh!

etra0 commented Mar 17, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

etra0 commented Mar 17, 2023

Uh oh!

j-f1 commented Mar 17, 2023

Uh oh!

etra0 commented Mar 17, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants