[ruibeikaa]

Problem and result

Measured on PH402 SKU 200 boards: four GP100 dies, 32 GB HBM2 each (128 GB in all), every expert resident in VRAM, no CPU share, no PCIe share.

On a GP100 die (compute capability 6.0: no __dp4a, no tensor cores) the verify window's routed down projection for the native packs' IQ4_NL and Q2_0 rows is latency bound, not bandwidth bound: expert mode 0's native_down_multi_kernel reads each weight with 2-byte loads, one item at a time, straight from global memory (IQ4_NL at ~170-185 GB/s of the die's ~730). The cc 6.0 default, mode 8 (native_down_lds_kernel), is faster than mode 0 for Q2_0 but slower for IQ4_NL on this card.

With STRATA_DOWN_WS=1, on a cc 6.0 device, native_down_ws_kernel takes the down step instead: a block copies its rows (IQ4_NL 32, Q2_0 64: 11,520 bytes) into shared memory with 16-byte loads, all in flight at once, reads the plan words before the group test and the destination rows before the weights land, and then runs mode 0's row dots on the copy. Per layer on one GP100 die (Swift 1.5 IQ3_S, real rows, T-token window = 10T groups of one entry, us per call, the minimum of 3 interleaved rounds of 300 calls):

down	T	mode 8 (cc 6.0 default)	mode 0	STRATA_DOWN_WS=1	vs default	vs mode 0
IQ4_NL	1	72.6	54.5	42.3	-41.7%	-22.4%
IQ4_NL	2	132.9	103.3	79.4	-40.3%	-23.1%
IQ4_NL	3	195.2	151.9	115.9	-40.6%	-23.7%
IQ4_NL	4	257.8	200.4	152.3	-40.9%	-24.0%
Q2_0	1	53.5	65.4	38.6	-27.9%	-41.0%
Q2_0	2	99.5	126.1	70.3	-29.3%	-44.3%
Q2_0	3	146.1	186.7	102.2	-30.0%	-45.3%
Q2_0	4	192.6	247.2	132.9	-31.0%	-46.2%

Swift 1.5 IQ3_S has 39 IQ4_NL and 9 Q2_0 down layers; weighted that way a layer's down step is 69.0 -> 41.6 us at T = 1 (-39.7% against the default, -26.4% against mode 0) and 245.6 -> 148.7 us at T = 4 (-39.5% / -28.9%). The output is mode 0's bit for bit, and mode 0's equals mode 8's here, so it is also the default's. Unset (the default) nothing changes.

End to end on four GP100 dies, on a different revision (engine 0.1.47 plus my GP100 stack, expert mode 0 in both arms, details below): decode +2.41% with every reply hash identical.

Related discussion

No scope issue. Same card and setup as #3455 and #3752 (GP100 decode and prompt-path experts); this change is independent of both and applies to plain main. My comment on #3455 measured expert mode 0 decoding about 2% faster than the cc 6.0 default mode 8 on this card, with the difference in the down projection; this kernel makes the down step faster than either mode without changing their results.

Approach

• src/kernels/cuda/iq_kernels.cu: native_down_ws_kernel<TD, NCMAX> (TD 20 = IQ4_NL, 42 = Q2_0; NCMAX 1 for a one-token window, 4 otherwise), down_ws_on, launch_down_ws, and a seven-line branch at the start of native_expert_grouped's down step, before the mode 8 / AMD / CUDA-layout choice.
• Why the bits are mode 0's: each (row, entry) is computed by the same 8- or 16-lane sub-warp, the same lane and the same items as row_dot_40_sub8 / row_dot_20_sub16, with the same float expressions in the same order (s += item, s16 / s8 / s24, the __fadd_rns, the xor reduction, * sd). Only an item's integer sum changes form: the dp4a emulation's byte products are taken with vmad byte selectors straight from the table lookup's bytes (no interleaving byte_perm), and IQ4_NL's lookup uses raw prmt selectors (bit 3 of a selector only affects bytes the low/high merge drops, and the merge selector's nibbles are 0..7). The same int8 x int8 products give the same int32, so the floats that follow are mode 0's.
• Gating: compiled only in the CUDA build with STRATA_EXPERIMENTAL_SM60 (not HIP, not Metal; a local STRATA_DOWN_WS_BUILT). At run time only with STRATA_DOWN_WS=1, on a compute capability 6.0 device (checked per device; vmad is emulated on Volta and later), for IQ4_NL / Q2_0 down rows at n_ff 640, and not with STRATA_OLD_IQ_MMVQ or STRATA_NO_SUB16_GU. When on it takes the down step in expert mode 0 and mode 8 alike.
• Resources (ptxas, sm_60): 63-64 registers, no spill, 12,240 / 14,400 B shared memory, four blocks per SM.
• docs/OLDER_GPUS.md: one bullet in the GP100 section.

Validation

Environment note: measured only on PH402 SKU 200 boards - four GP100 dies of 48 SMs and 32 GB HBM2 each (128 GB in all), application clocks 715 / 1050 MHz - with every expert resident in VRAM (no CPU share, no PCIe share, no adaptive tier). The single-die numbers use one of those dies.

Rig: Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022; build -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60, ggml at the pinned 3cf03257f. Model: Swift 1.5 IQ3_S (abliterated GSQ-RCO GGUF).

Passed:

• This branch (main 7ddc17a + this PR), one GP100 die: native_expert_bench <shard> 0,2,1,8 <G> 1 <ref> <mode> 300 with BENCH_CAP=G, G = 10 / 20 / 30 / 40 (T = 1..4 windows, one entry per group), layers 0 / 2 (IQ4_NL) and 1 / 8 (Q2_0). Without the switch: mode 8 bitwise equal to mode 0 on all 16 (layer, G) cases. With STRATA_DOWN_WS=1: the output's FNV-1a hash equal to the switch-off run's (that is, mode 0's and mode 8's) on all 16. The table above is from these runs.
• During development the kernel was also compared byte for byte with mode 0 on all 48 layers, all 512 experts, 1-6 entries a group and T = 1..4 routing plans (an out-of-tree harness on the same die): equal.
• Default off: cuobjdump of iq_kernels.cu, sm_60 build: the 442 kernels of main have identical SASS with this PR; the four native_down_ws_kernel instances are the only additions. The sm_86 build without STRATA_EXPERIMENTAL_SM60 compiles none of them (442 kernels, as main) and builds clean.
• End to end, four GP100 dies, not this branch's revision: engine 0.1.47 (main 011c9e6) plus 26 opt-in GP100 commits of mine ([Engine] GP100 (sm_60): 4-bit slots for resident IQ2_S / IQ3_XXS / IQ3_S gate/up experts, decode +9.7% (opt-in, STRATA_EXPERT_NIBBLE=1) #3455's 4-bit gate/up slots on 47 of 48 layers, [Engine] GP100 prompt path: a fully resident layer's experts weight-stationary, one launch per product (STRATA_MMQ_WS=1, opt-in) #3752, --pipeline-windows 3 and others not posted), expert mode 0 (STRATA_SM60_TABLE=0) in both arms, so the gain is against mode 0's down, not the default mode 8's. Swift 1.5 IQ3_S, layer split 12,24,36, MTP, one fresh engine per arm, eight decode prompts (five short prompts, 8K / 26K / 88K-token contexts) token-weighted: 57.53 -> 58.92 tok/s (+2.41%), each prompt +2.2..+2.5%; every reply hash and the draft acceptance identical on all 9 requests; prompt reads within 0.05%. One run per arm. Against the default mode 8 the down step saves more per layer (table above) but that was not run end to end.

Failed: none observed.

Not run: HIP (gfx906), SYCL and Metal builds (no such machine here; the kernel is under the STRATA_EXPERIMENTAL_SM60 CUDA-only guard). An end-to-end run on this branch's own revision. A single Tesla P100 16 GB with experts partly in RAM. Packs other than Swift 1.5 IQ3_S. Cards other than GP100 (the switch does nothing below or above cc 6.0).

Limits

• GP100 (cc 6.0) only, STRATA_EXPERIMENTAL_SM60 builds only, IQ4_NL and Q2_0 down rows at n_ff 640 only.
• Measured only on PH402 dies with every expert resident in VRAM.
• The kernel is now bound by the emulated dp4a (most of its time is the integer math), so further gains need a different weight layout, not more staging.

CI note: the red metal-link check is not caused by this PR. The job fails while compiling upstream's metal/src/mtl_rt.mm (lines 69 and 92: MTLGPUFamilyMetal4, MTLCompileOptions.mathMode, MTLMathModeFast / MTLMathModeSafe), which the macos-14 runner's SDK does not provide. The workflow runs on any pull request that touches src/ or include/; this PR changes no file under metal/.

[Niko1221]

Thank you. Fixed in 0.2.7: GP100 decode down projection staged in shared memory, opt-in STRATA_DOWN_WS=1. Your name is on the commit.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7