[amitmi704]

Overview

Add vector path for an f16 accumulator operation when no dot-product instruction exists. Two scalar FMA operations become one f16vec2 FMA accumulated as v_pk_fma_f16 in the same order so results are bit-identical. Uses a helper with a guard so RADV packs them when applicable hardware executes the code.

The guard keeps pipelines identical to master: 5/61 MUL_MAT (the remaining 56 are the f16-accumulate ones that are subject to change under this PR), 61/61 forced f32 accumulation, 61/61 fp16 off, 650/650 FLASH_ATTN_EXT on Navi 10.

Only applies to devices that meet these criteria:

1. The device supports f16 arithmetic but has no cooperative matrix
2. The weights don't require f32 accumulation (f32, bf16)
3. Activations aren't q8_1 (branches to mul_mmq instead of mul_mm)
4. The model graph doesn't request f32 precision
5. No dot-product instruction

Additional information

AMD BC-250 (RADV GFX1013) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
40 CUs are enabled (BC250_WGP_MASKS=0x1f on all shader arrays); GPU is overclocked to 1.8 GHz with stock voltage.

Found a peak actual vs theoretical throughput gap in prefill-shaped MUL_MAT shapes (n=512) measured with test-backend-ops perf on a GFX1013 (BC-250) using q8_0 and f16 for control. Both showed the same throughput peak of ~37% of theoretical throughput (6 TFLOPS vs 16.4 TFLOPS) ruling out dequantization bottlenecks.

• Sweeping tile-sizes found nothing.

Results

• CI passed: ci/run.sh with Vulkan, 19029/19029 test-backend-ops on 99b95488c using Mesa 26.0.8.

Note: Original PR ran 19011/19011 test-backend-ops on 5fc4f3c using Mesa 26.0.8.

Measured

• Tested for speed and quality on the GFX1013 controlling for driver version (Mesa 26.0.8 and 26.2.3)
• Llama 2 7B Q4_0 (TheBloke/Llama-2-7B-GGUF)
• llama-bench -ngl 100 -fa 0,1 (the scoreboard command from discussion Performance of llama.cpp with Vulkan #10879)

Note: These tests were run twice, except llama-perplexity hence both results are shown where applicable. Also, these are from the original PR.

test	26.0.8 no patch	26.0.8 patch	26.2.3 no patch	26.2.3 patch	change 26.0.8	change 26.2.3
test-backend-ops -o MUL_MAT vs CPU	1132/1132	1132/1132	1132/1132	1132/1132	none	none
pp512, fa 0 (t/s)	519.49, 519.79	694.03, 694.33	513.30, 513.39	697.83, 697.88	+33.6%	+35.9%
pp512, fa 1	531.40, 531.44	714.45, 714.47	525.20, 525.23	719.11, 719.40	+34.4%	+36.9%
tg128, fa 0	72.70, 72.75	72.66, 72.52	70.82, 71.13	71.13, 70.71	-0.2%	-0.1%
tg128, fa 1	77.09, 77.06	77.33, 77.02	75.32, 75.32	75.13, 75.31	+0.1%	-0.1%
llama-perplexity	5.3488 +/- 0.07439	5.3488	5.3488	5.3488	identical	identical

Static Analysis

• Statically analyzed RADV-generated shader code:
  • Used AMDGPU shim to generate: VEGA10, RAVEN, RAVEN2, RENOIR and NAVI10, VEGA20 and NAVI21 as controls
  • No shim required: GFX1013 (BC-250)
• Counted scalar v_fma_f16 against packed v_pk_fma_f16 in each f16acc mul_mm pipeline to validate changes
• Current PR is byte-identical to the original PR on (1) Navi 10 (Mesa 26.2.3, shim), (2) the BC-250 (Mesa 26.0.8)
• 56/61 pipelines differ from master (the f16 accumulate ones now packed).

Note: Used master 99b9548 (Original PR: a868c3e) on Mesa 26.2.3 and the GFX1013 (BC-250) numbers came from real hardware (no shim)

Chip (as RADV saw it)	Before (scalar / packed)	After (scalar / packed)
GFX1013 (BC-250), small tile	512 / 0	0 / 256
GFX1013 (BC-250), large tile	1024 / 0	0 / 512
Vega 10, Raven, Raven2, Renoir	512 / 0	0 / 256
Navi 10, small tile	512 / 0	0 / 256
Navi 10, large tile	1024 / 0	0 / 512
Vega 20, Navi 21 (controls)	integer-dot q8_1 shaders	unchanged

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, an agent:
  • found the performance gap, ran benchmarks and CI/CD to iterate positive and negative testing
  • analyzed instruction counts from disassembly dumps
  • wrote the code
  • tried various permutations of changes based upon your review feedback

[jeffbolznv]

This all feels very roundabout to me. Like, the dot_product function (when DOT2_F16 is not defined) should just be return dot(a,b)+acc; and rely on the compiler to produce f16vec2 instructions.

[amitmi704]

I tried a couple of methods but couldn't get RADV to pack:

1. dot(a,b)+acc doesn't compile
2. ACC_TYPE(dot(a,b))+acc compiles, arithmetic order can differ while the instruction count doesn't change and doesn't pack (RADV doesn't figure out the pairing)
3. (Previous commit) Two f16vec2 FMAs pack and the f32 accumulate pipeline is byte-identical to master.
4. (Updated commit) The guarded dot_product packs on Mesa 26.0.8 and 26.2.3 and the guard keeps the f32 pipelines identical to master.

The CI tests passed on 26.0.8. The test_backend_ops stage only consistently passed without locking the GPU after a 100 mV voltage increase. I validated this occurs on master.

[0cc4m]

I don't mind the change, but the function name shouldn't be the same, it's not a normal dot product anymore. You can name it e.g. dot_product_v2(.

[amitmi704]

I made this change. Let me know if there is anything else want to see done!