[GenerelSchwerz]

Overview

Fuse F32 weighted RMS normalization with an independent same-shape SiLU or sigmoid gate in one CUDA kernel. The existing two-kernel sequence becomes one, removing an intermediate write/read. This extends the previous SiLU-only revision of this PR; it remains independent of activation-reuse PRs #117/#118.

The matcher selects the graph's actual activation, and the kernel specializes SiLU/sigmoid at compile time using the existing scalar formulas. Normalization order, epsilon and weighting remain unchanged. Qwen3.5/Qwen3Next SiLU and Qwen4Exp sigmoid builders expand the independent gate before normalization so eligible graph nodes are adjacent. This is graph dependency ordering, with no model whitelist or new operator.

The four-op closure, F32/layout/shape checks and allocator dependencies remain in place. Broadcast gates, unweighted RMS, extra intermediate consumers, dependent gates, in-place intermediates and other activation functions keep their existing paths. The gated matcher additionally checks output overlap with leaf inputs: a custom output aliasing a broadcast normalization weight reproduced incorrect SiLU results in the previous revision. This revision falls back and matches upstream. Other fusion callers retain their original overlap policy.

Measured operator benefit

RTX 5070 Ti / SM120a, CUDA 13.4.59, release build. Four alternating upstream/previous/new triples per row; complete unprofiled operator-sequence replay, two fresh input/gate passes. Separate CUDA-node profiles prove two kernels become one for eligible SiLU/sigmoid graphs; negative controls keep their kernel counts.

Gate / F32 columns x rows	Upstream us	Previous PR us	This revision us	Less time vs upstream
Sigmoid / 63 x 30	2.333	2.457	2.050	12.1%
Sigmoid / 128 x 16	3.552	3.215	2.049	42.3%
Sigmoid / 128 x 8192	22.542	22.560	18.444	18.2%
Sigmoid / 1024 x 30	4.094	4.094	4.094	flat
SiLU / 128 x 16	3.412	2.049	2.050	39.9%
SiLU / 128 x 8192	22.543	18.480	18.457	18.1%

Captured SiLU performance is essentially unchanged versus the previous PR. A tiny graphs-disabled SiLU case is 1.439->1.475us (+2.54% time); this exposes a small direct-execution regression; the added safety/admission work may contribute, but its exact share was not isolated. The matching sigmoid case is 2.758->1.488us versus upstream (46.1% less time). Broadcast-gate controls are flat. Ranges and all negative/neutral results are retained. These component gains are not model throughput gains.

Model regression and limits

Qwen3.5-0.8B Q4_0 supplied-token C1/C4 evaluations use 128 calls, 32 warmups excluded, all GPU layers, F16 KV, FlashAttention and four CPU threads. Four alternating triples: complete first-two/last-two float-logit rows match upstream/previous/new. Previous/new means are 1488.751/1488.215us at C1 and 1948.236/1949.478us at C4, essentially flat.

Full no-MTP Qwen3.5 requests use 9303 real-text prompt tokens and 2048 generated tokens. Eight previous/new requests in ABBA then BAAB have identical complete token IDs and output bytes. The first block shows -2.31% candidate decode throughput, the reversed block +0.51%. Combined means are 488.894/484.366tok/s, with overlapping ranges 478.137-502.212/480.321-490.973. No serving improvement or universal performance guarantee is claimed.

No Qwen4Exp model is installed for a full-model sigmoid run. Sigmoid is proved in upstream operator graphs; its model builder ordering is source-reviewed. Full-model sigmoid speed, a Qwen3Next request, older GPUs, HIP, Windows and multi-GPU are not validated. F16/BF16, broadcast gates, nonadjacent matching and additional gate functions remain outside this scoped fusion.

Validation and reproduction

• 479/479 focused CUDA-vs-CPU backend cases each with graphs on/off and fusion disabled, including both gates, boundary widths, strided inputs/gates and unsafe/extra-consumer/broadcast/in-place fallbacks.
• 3513/3513 matrix, routed-ID and fusion regression cases each with graphs on/off.
• 35 three-build external cases meet CPU NMSE 1e-12 and have byte-identical outputs across fresh input updates, strides, operand order, extreme gates, unsupported GELU/unweighted/dependent/broadcast paths and graph/fusion controls.
• Twelve further leaf weight/input/gate alias controls match upstream. Two old-SiLU weight-alias controls reproduce the inherited bug; all sigmoid variants match. Two compute-sanitizer memchecks report zero errors. Independent final source review: GO.

Both shared build/GPU locks are held for each complete suite. The external correctness resume uses two workers; all profiles, memory checks, performance repeats and serving requests run sequentially. Completed backend regressions are retained rather than repeated after an external-harness construction correction. The existing tester supports -j for future correctness runs.

Build with -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120. Focused command: test-backend-ops test -b CUDA0 -j 2 -o RMS_NORM_SIGMOID,RMS_NORM_SILU,RMS_NORM,RMS_NORM_MUL_ADD,RMS_NORM_MUL_ROPE,L2_NORM,MOE_REDUCE,GATED_DELTA_NET; repeat with graphs disabled and focused fusion disabled. Parallel backend mode is a reproduction option; this revision's recorded backend passes used one worker.

Official base dcd387a412ca54e172a8d60eb71ef6753850c8ca; previous PR 40957049604ed0fc37892bc37fbe142224ecf8e0; measured revision 4b88fc8c60b32ebf7fa593cbb5adad699877700c. Exact commands, source/binary/model hashes, helper sources, raw results, outputs and teardown records are retained locally under /home/gencoolpc/moe-cache-tests/results/upstream-kernel-series-20261001: RMS-GATED-EVIDENCE.md, RMS-GATED-MANIFEST.json and RMS-GATED-*.json.

Requirements

• Contributing guidelines and the owner-fork policy were read. This draft targets the owner fork; the owner handles human review, maintenance and any upstream submission.
• AI usage disclosure: YES. Codex implemented and tested the owner-discussed design, prepared this description and used a bounded independent source review. The owner remains responsible for all submitted changes.