[BlueKingMuch]

Every verify window calls native_expert_grouped twice per layer: for the experts in VRAM and for the ones that come over PCIe. Both calls launched gate/up and down with one block row per possible group (cap_groups = window tokens x top-k) and ran SwiGLU and the q8_1 quantization as two kernels over all cap_entries. The PCIe call has no group at all at pcie_frac 0 and rarely more than one otherwise, yet every layer paid cap_groups x (2 n_ff / 8 + n_embd / 8) blocks for it that read n_groups and returned (about 17k with this model's 3.4-token windows), plus two passes over entries that were not its own. The graph is captured once per session and a request may still set pcie_frac afterwards, so the call stays in the graph (the graph without the PCIe path was left out when #109 was merged); this PR makes it cheap instead.

What changes

• Group stride: the four grouped kernels (native_gu_kernel, native_down_kernel and IQ kernels: decode each weight part once for every column and entry, bitwise identical #242's _multi variants) run group g in block row g % gridDim.y (rows y, y + gridDim.y, ... below *n_groups). The verify window's PCIe call launches kPcieGroupRows = 4 block rows; the VRAM call and every other caller keep one per possible group (grid_groups defaults to 0 = cap_groups). A group's rows are computed by the same warp code in the same order whichever block row runs it, so the results do not depend on the grid.
• SwiGLU and q8_1 in one kernel, over the call's own entries [grp_start[0], grp_start[n_groups]). The SwiGLU product is rounded on its own before the q8_1 block sums, as it was when it went through memory (__fmul_rn). The quantizer body is quantize_q8_1_kernel's, factored out into a device function; that kernel's PTX is unchanged.
• STRATA_GROUPED_V1=1 keeps the old launches, for A/B timing.

It is on by default since it changes no bit. If you prefer it opt-in, that is the default of g_grouped_v1.

Exactness

• native_grouped_parity (a GPU ctest next to iq_multi_parity) runs the new launches against STRATA_GROUPED_V1's and compares the outputs bit for bit: every gate/up format with every down format the dispatcher takes (gate/up IQ2_XXS, IQ2_XS, IQ3_XXS, IQ3_S, IQ2_S, IQ4_XS, IQ1_M, Q2_0, Q4_K, Q5_K, Q8_0; down IQ4_NL, IQ4_XS, Q2_0, Q5_1, Q8_0), calls of 0 to cap groups, entries from 0 and from past 0 (the PCIe call's entries follow the VRAM call's), scattered destinations, stale scratch, and grid_groups 0, 1, 2, 3, 4 and above cap. The rows a call must not write are compared too (both runs start from the same NaN pattern). On this branch (0.1.31): 0 failures.
• End to end: requests without anything timing-dependent (no prompt-lookup drafts, no expert swaps, pcie_frac=0) give the same 256 tokens in the same number of rounds with and without STRATA_GROUPED_V1=1, greedy and sampled (T 1.0, top_p 0.95, top_k 20, seed 1).

Measured

RTX 4080 SUPER 32 GB (sm_89), Ryzen 7 5800X3D, 64 GB DDR4, PCIe 3.0 x16, Windows 11, CUDA 13.3.

The calls alone, on this branch: native_grouped_parity --bench captures a window's 48 calls in a graph, each layer with its own experts (2560 x 768, IQ3_S gate/up, IQ4_XS down), so the weights stream from DRAM as in the engine. The two variants' launches alternate; medians of 40 launches each:

call	before (STRATA_GROUPED_V1=1)	this PR
PCIe call, no group (pcie_frac 0)	16.51 us	4.01 us
PCIe call, one group of two entries	19.11 us	10.71 us
VRAM call, 28 groups / 40 entries	149.72 us	148.95 us

Kernel by kernel (Nsight Systems on the same bench), the VRAM call's gate/up and down take the same time in both variants (86.2 / 85.8 us and 54.7 / 54.8 us), and the fused SwiGLU + q8_1 kernel takes 2.06 us against 1.41 + 1.47 us for the two it replaces.

One thing to watch when timing this on a desktop card: under sustained load this one steps its clock down after a fraction of a second (gate/up went from 80 to 89 us within one run), so timing one variant after the other can show a difference of that size that has nothing to do with the change. That is why the bench alternates them.

In the engine: 0.1.31 (9259cad) with this commit only, plus two measurement aids that change no computed value: #320 (so the profile's columns add up) and a local patch that lets a request ask for a fixed length and for no timing-dependent parts (ignore_eos, suffix=0, adapt=0). IQ3_S, --max-context 262144, int8 KV, 11055 experts in VRAM; every comparison within that build (STRATA_GROUPED_V1 unset against =1).

GPU stages (STRATA_VERIFY_PROFILE=1, 8K context, ms per verify window, DeltaNet and attention layers summed):

	before (STRATA_GROUPED_V1=1)	this PR
experts over PCIe	0.67	0.20
experts in VRAM	4.56	4.52
waiting for the plan and the CPU experts	1.45	1.88
whole window	19.99	19.91

The PCIe call loses its 0.47 ms in every run I made, also in a 0.1.30-based build and next to other open PRs. How much of it the window keeps depends on what the GPU waits for next: in these runs it mostly waited longer for the host side instead (+0.43 ms for the plan and the CPU experts), so the window got 0.08 ms shorter; in the 0.1.30-based run it kept all of it (18.97 -> 18.50 ms).

The deterministic requests above (the same 256 tokens in 97 / 103 rounds; with expert swaps off the CPU experts take about 7.5 ms per round): round 24.79 -> 24.49 ms greedy and 25.13 -> 24.85 ms sampled (-0.30 / -0.28 ms, +1.1 % tokens/s), GPU wait per round -0.31 / -0.26 ms.

Decode, 5 requests of 256 tokens per cell, prompt-lookup drafts and expert swaps on (8K greedy / sampled, 64K greedy / sampled):

	before	this PR
GPU wait per round, ms	14.76 / 14.86 / 16.35 / 16.43	14.60 / 14.77 / 15.89 / 16.10
tokens/s	115.5 / 114.3 / 111.9 / 114.1	118.0 / 113.9 / 119.6 / 116.4
--pcie-frac 0.28: GPU wait per round, ms	14.85 / 14.94 / 16.46 / 16.50	14.41 / 14.57 / 16.05 / 16.05
--pcie-frac 0.28: tokens/s	115.5 / 109.8 / 115.7 / 114.5	113.3 / 112.7 / 117.9 / 112.6

The GPU wait per round drops in all eight cells (by 0.09 to 0.46 ms). The tokens/s stay within the scatter of single requests (104 to 130 within one cell here; without the SSD keep-alive of #317 some requests also waited for SSD reads): +2.2 / -0.3 / +6.9 / +2.0 % at pcie_frac 0, -1.9 / +2.6 / +1.9 / -1.7 % at 0.28.

In short: a fixed 0.47 ms of GPU time per window, tokens and bits unchanged. What it buys in tokens/s depends on whether the GPU or the CPU experts set the pace on a given machine; about 1 % in the deterministic runs here.

To reproduce: STRATA_GROUPED_V1=1 against unset, STRATA_VERIFY_PROFILE=1 for the stages, native_grouped_parity --bench for the calls alone.

[bsorensen110]

Tested on the R9700 (gfx1201, ROCm 10.2.0a / hipBLASLt 1.5.0, Swift 1.5 IQ3_XXS, --kv int8, --spec 4 --mtp, MMQ ON). The launch reduction reproduces, but I could not measure a decode-speed gain.

Build / correctness

• Base is 9259cad (v0.1.31) here, but it cherry-picks cleanly onto current main — git auto-merged verify.cpp/iq_kernels.cu with zero conflicts despite 8 commits touching verify.cpp in 0.1.32. 54/54 ctest including the new native_grouped_parity.

Kernel launches drop as claimed (rocprofv3 --stats, a 64-token decode):

• Total 70,948 → 69,051 (−2.7%, ≈ −30 kernels/step).
• swiglu_entries_kernel 1,920 + quantize_q8_1_kernel 2,880 → fused swiglu_q8_1_entries_kernel 1,920 (+ residual quantize_q8_1_kernel 960).

Decode speed: a wash (same binary, STRATA_GROUPED_V1=1 vs unset, fresh 512-token trials):

• First 3×512 run: 66.45 → 69.42 tok/s (looked like +4.5%).
• 6-trial retest: 69.22 (V1) vs 68.85 (lean) — per-trial ranges overlap completely (63–71 tok/s both arms).

So the ~2.7% fewer kernels don't move throughput on this card — decode here is ~34% idle with median kernel 2.64 µs, and the removed blocks were the ones that already returned immediately. I believe it's still a real (if small) reduction in GPU work — it just isn't the decode bottleneck on gfx1201 at ~69 tok/s. Possibly more visibly useful at low clocks, on multi-stream/MTP-heavy servers, or on other cards; happy to share the rocprof CSVs if useful.

[BlueKingMuch]

Thanks @bsorensen110 for testing it on gfx1201, and for the launch counts: good to see the reduction reproduce too on AMD.

On CUDA it does show up in decode, I think because there the GPU sets the pace. 0.1.32 with this PR on top (it merges without conflicts; I run it that way), RTX 4080 SUPER 32 GB, IQ3_S at 262K, 256-token requests, STRATA_GROUPED_V1=1 against unset in the same build: all 12 requests (8K / 64K, greedy / sampled, 3 each) gave the same tokens in the same number of rounds, and every round was 0.38-0.54 ms shorter with this PR:

	greedy 8K	greedy 64K	sampled 8K	sampled 64K
ms per round, STRATA_GROUPED_V1=1	21.8	22.1	21.0	22.2
ms per round, this PR	21.4	21.6	20.6	21.7
tokens/s	+2.0 %	+2.2 %	+2.2 %	+2.3 %

Here the GPU wait is 64-71 % of a round, and it drops by the same 0.4-0.5 ms. With your card ~34 % idle in decode, the removed blocks come off time the GPU spent waiting anyway, which fits your numbers.

btw: i don't only work on/with Nvidia, i have a 2nd System with an RDNA2 RX6800 too where I am looking into enabling (better) support here and maybe do some performance tuning. Might as well help on newer gfx Archs too then.

[Niko1221]

Released in 0.1.38: the PCIe call's group stride and SwiGLU + q8_1 in one pass. Bitwise here as well (STRATA_GROUPED_V1=1 against the default). Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!