Repository navigation
llama : scale moe down projection per expert token - #30861
Merged
Merged
Conversation
github-actions
Bot
added
the
ggml
ggerganov
commented
Oct 20, 2026
ggerganov
force-pushed
the
gg/amax-pool1d-meta
branch
from
October 20, 2026 07:57
1f6a17e to
9e7f76c
Compare
This was referenced Oct 20, 2026
ggerganov
marked this pull request as ready for review
October 20, 2026 09:35
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggerganov
force-pushed
the
gg/amax-pool1d-meta
branch
from
October 20, 2026 13:47
9e7f76c to
0c5b7cb
Compare
Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
Member
Author
|
Tested on CUDA - all good. Should be ready to merge. |
ggerganov
added
the
merge ready
|
Tested on Vulkan with Mellum2.1, tps and prefill are much better now |
3 of 6 tasks
Member
Author
|
There is one remaining issue - some quantization implementations use the @jeffbolznv Let me know if you have any thoughts. For now I think I'll adjust the scaling to be more conservative. |
voidstackloop
added a commit
to voidstackloop/llama.cpp
that referenced
this pull request
Oct 21, 2026
Resolve the conflict with 8baabd5 (llama : scale moe down projection per expert token, ggml-org#30861): master replaced the MISTRAL4/MELLUM F32 precision override on the MoE down projection with an amax rescale of src1. Move that rescale into the PR's build_experts lambda, the single path every expert computation goes through (plain, cache, hybrid hot/cold, mailbox), in place of the old ggml_prec_set_src there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
I think there was some proposal to use bf16 for the q8_1 scale factor, maybe we should do that? |
Member
Author
Ok, I'll look into that. |
adderek
pushed a commit
to adderek/llama.cpp
that referenced
this pull request
Oct 21, 2026
297 upstream commits. Conflicts and how they were resolved: - ggml ops: upstream added GGML_OP_TOP_K_MASK next to the fork's GGML_OP_TURBO_WHT; both are kept, upstream's first, GGML_OP_COUNT 103. - MoE down projection: upstream replaced the MISTRAL4 F32 src1 precision hint (its own, ggml-org#29029) with a per-row amax rescale (ggml-org#30861) for MISTRAL4 and MELLUM. It now sits in build_moe_ffn_exps, so every tiered call (hot, cold, each arena window) rescales its own rows; rows skipped by a partial mul_mat_id are zero, so their scale is 1. - MoE post_exp_norm (new upstream): applied in build_moe_ffn after the hot/cold merge and before the router weights; the two halves fill disjoint expert slots, so the norm sees whole rows. - build_lora_mm_id: upstream's Hadamard input transform (prism) after the fork's partial/slots assert. Expert tensors split by LLAMA_MOE_HOT are not in the Hadamard map, so prism and tiering do not stack on one layer. - kv-cache attention rotation: the fork's default (off unless LLAMA_ATTN_ROT_DISABLE=0) is kept; upstream's llama_model_set_attn_rot can still turn it off per model. - model loader: hugetlb per-tensor populate kept before upstream's multi-buffer lookup of the mmap buffer. - llama-model: MoE arena setup and upstream's load_tensors_hadamard both run at the end of load_tensors. - qwen4exp: turbo Q pre-rotation kept; the selection mask is now `sel`. - server: upstream added draft-token logprobs (ggml-org#27196) read back from the batch logits, without post_sampling_probs. The fork's on_sample path, which records them during verification and covers post_sampling_probs, is kept; upstream's test_draft_token_probs passes with it. Upstream's local spec_i_batch is taken. - mmq: fork's IQ1_M MMQ gains upstream's prec_src1 parameter. - fattn-tile: the fork's D=640 case and HIP guard kept, upstream's default-case style. - argsort.cu: upstream's CCCL guard; the fork's hipcub arm was dead since HIP does not define GGML_CUDA_USE_CUB. - tests/CMakeLists.txt: fork tests kept, upstream's LINK and consolidated test-llama-context taken. Tested on gfx1100: ctest 78/80 (the two failures are environmental: the ggml-vocabs clone has git-lfs pointers, and system python has no jinja2). All fork tests pass (turbo-kv x8, moe-tier, fattn-turbo4, fattn-long-kv, mmvq-src1-cache, fusions, mul-mat-f16-range, top-k-graph-capture, op-offload-devices). test-llama-archs 390/390. Server test_speculative.py 10/10, completion/probs pytests 14/14. Needle at 20.5k tokens, Qwen3.8-27B IQ4_NL, turbo4 K/V: correct, prefill 925 tok/s vs 909 before the merge (single run).
anemicsoftware
added a commit
to anemicsoftware/mx-llama.cpp
that referenced
this pull request
Oct 23, 2026
…#30861) Cherry-pick of upstream 8baabd5, minus the MoE-cache slots argument mx does not have. The F32 src1 precision on the mistral4 down projection (from ggml-org#29029) made the CUDA backend reject the MUL_MAT_ID, so every layer's down projection ran on CPU. With layer split that staged each layer's expert weights in a ~101 GiB host buffer, and copying out of the repacked buffer (no get_tensor) segfaulted. The graph now rescales src1 per expert token instead and the op stays on the GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stanley00
pushed a commit
to stanley-fork/ggml
that referenced
this pull request
Oct 28, 2026
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the per expert-token amax scaling done in the compute graph (ref: ggml-org/llama.cpp#30861), which keeps the MoE down-projection activations within f16 range before the op runs. Drop the two-stage reduction kernels, their scratch buffer and the GGML_PREC_F32 dispatch added in ggml-org/llama.cpp#26223, as well as the test cases that drive MUL_MAT_ID src1 out of f16 range. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
1 task done
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
ggerganov
added a commit
to ggml-org/whisper.cpp
that referenced
this pull request
Oct 29, 2026
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the per expert-token amax scaling done in the compute graph (ref: ggml-org/llama.cpp#30861), which keeps the MoE down-projection activations within f16 range before the op runs. Drop the two-stage reduction kernels, their scratch buffer and the GGML_PREC_F32 dispatch added in ggml-org/llama.cpp#26223, as well as the test cases that drive MUL_MAT_ID src1 out of f16 range. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
… per-device scalar) This block is a LOCAL, TEMPORARY relaxation of upstream behaviour, not a feature of the delivery. It exists so the MoE amax arches (flint, mellum, mistral4) can run -sm tensor today; it is expected to be replaced once upstream lands a real fix, and it can be dropped without touching any other block. Limitations are recorded in patches/README.md (block 14) and GREEDY-PURITY.md; do not treat `-sm relaxed-tensor` as a supported mode. Why upstream refuses today: ggml-org#30861/ggml-org#31081 -- the MoE amax rescale pools a whole matmul row (`pool_1d(abs(cur), MAX, k = s = n_ff_exp)`) to divide `cur` by a power of two and undo it on the down-projection output. Under `-sm tensor` no device holds a complete row, so each one can only compute the max of its own slice. ggml-org#31081 removed the Meta POOL_1D case (which claimed MIRRORED, a false claim) and denied -sm tensor for those three arches. Add a split mode instead of a blanket deny. `-sm relaxed-tensor` places the weights exactly like `-sm tensor` (same Meta wrapper, FA required, expert tables split: 0 extra VRAM) but also admits a replicated tensor whose values differ per device: - ggml: new GGML_BACKEND_SPLIT_AXIS_PER_DEVICE = 12 -- same shape/positions everywhere, per-device values. `handle_pool_1d` produces it for an axis-0 split (whole-row MAX only), `handle_bin_bcast` lets it be consumed elementwise and broadcast, and every copy/gather/read-back path aborts with a message instead of silently returning one device's value. - The delayed-reduce MUL-chain in graph_compute accepts it, so the reduce point stays AFTER the per-device undo: each device applies its own `e` to its own partial and only then is the sum taken. That is what makes the per-device value cancel (`e` is a power of two, so the scaling commutes with the sum), and `e_local <= e_global` means every device keeps at least the reference's f16 headroom. - llama: LLAMA_SPLIT_MODE_RELAXED_TENSOR = 4, `-sm relaxed-tensor`; every "is this the tensor path?" site goes through llm_split_mode_is_tensor_like(). `-sm tensor` keeps upstream's strict behaviour and the deny, but the error now points at the new mode. Measured (Flint Q8_0, 2x R9700): greedy text byte-identical to `-sm layer` and to itself at 3 GPUs; with the amax divisor forced to 1 (so `e` != 1) the POOL_1D pooled max differs across devices on ~half of the ~185 per-run calls and the text still matches `-sm layer` exactly. A non-amax arch (Qwen3.5-4B) is byte-identical under tensor and relaxed-tensor. `ggml_pool_1d` has exactly one producer in the graph, so the relaxed state is unreachable for any other arch. NOT claimed: bit-identity to a single device on a backend whose in-kernel src1 rescale is non-linear (Vulkan's clamp; RDNA/HIP is exact), any behaviour under -ncmoe (the expert arena still refuses tensor-like modes), and mellum/mistral4 at all (enabled by the same change, unvalidated -- no model).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
ref #30656
alt #30733
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. The same exact power-of-two factor is undone on the output, so the roundtrip preserves the matmul result. Handle POOL_1D in the Meta split-state logic.
Additional information
Requirements
pi:llama.cppMac Studio, M2 Ultra, 192GB◷ 3.4h · ⚙ 10.6m · ↑ 497k · ↓ 259k · ≡ 915 · ⌨ 28