[thelittlefireman]

Overview

Older AMD GPUs (pre-CDNA / pre-RDNA3) do not have native BF16 matrix acceleration, and rocBLAS has no tuned BF16 kernel for them. On these older card a BF16 GEMM can land on a 64x32x8 stub tile.

This PR add a fallback from BF16 to F32 on pre-CDNA / pre-RDNA3 AMD GPUs because rocBLAS does not provide a properly tuned BF16 GEMM path there.

Applies to UD/*_XL quants that keep BF16 tensors.

BF16 path:
    convert activations F32 -> BF16: O(K * N)
    BF16 x BF16 GEMM with F32 accumulation

F32 fallback:
    convert weights BF16 -> F32: O(M * K)
    F32 x F32 GEMM

During testing on my gfx906 this gives large prefill improvements for sufficiently large GEMMs, but benchmarking showed that doing the conversion unconditionally introduces a significant regression for small N.

After some tests on multiple models (differents count of BF16 and different GEMM shapes (M,N,K)) I added small check to keeps BF16 on AMD when N <= 32 and switches to F32 when N > 32.

        if (GGML_CUDA_CC_IS_AMD(ggml_cuda_info().devices[ctx.device].cc) && src1->ne[1] > 32) {
            compute_type = GGML_TYPE_F32;
        }

src1->ne[1] > 32 corresponds to the GEMM N dimension.

I've only made the change for AMD. Nvidia GPU before Ampere does not have native BF16 matrix acceleration eather but I do not own a Nvidia card to test this. Feel free to complete/remove the todo if revelant.

Additional information

Why N > 32 ?

Whitout the thresold, on small N <=32 :

• Qwen3-14B: F32 was ~10.5% slower
• Qwen3-4B: F32 was ~7% slower
• Llama 3.1 8B: F32 was ~3% slower
• Gemma 3 12B: F32 was ~8% slower

N=32 Seems to be the crossover. The generic HIP fallback library contains: Cijk_Ailk_Bjlk_BSS_BH_MT64x32x8_...
which seems to always fallback to the 64x32x8 macro-tile and several other fallback solutions also use a free-dimension tile of 32. This does not necessarily mean that the 64x32x8 macro-tile alone causes the crossover: rocBLAS can select different solutions depending on the complete (M, N, K) shape. However, the exact 32 -> 33 transition was observed across models with very different shapes, which makes the N > 32 threshold a good fit.

Tested models:

• Qwen3-4B
• Qwen3-8B
• Qwen3-14B
• Llama 3.1 8B
• Gemma 3 12B
• Gemma 4 E4B

Overall results

Model	BF16 tensors	N = 64	N = 128	N = 512
Qwen3-14B UD-Q8_K_XL	67	+1.74%	+19.47%	+21.70%
Qwen3-8B-128K UD-Q8_K_XL	63	+6.02%	+13.43%	+18.09%
Llama-3.1-8B-Instruct UD-Q8_K_XL	59	+6.51%	+14.20%	+19.67%
Qwen3-4B UD-Q8_K_XL	26	+1.53%	+5.21%	+14.19%
Gemma-3-12B-it UD-Q8_K_XL	26	+1.15%	+10.60%	+17.52%
Gemma-4 E4B UD-Q4_K_XL	1	+0.31%	+0.70%	+0.94%

Detailed threshold validation

Model	BF16 tensors	ubatch / N	master (t/s)	PR (t/s)	delta
Qwen3-14B UD-Q8_K_XL	67	16	77.35	77.39	+0.05%
		31	219.28	219.35	+0.04%
		32	226.69	226.89	+0.09%
		33	185.48	187.64	+1.17%
		64	279.49	284.35	+1.74%
		128	279.13	333.48	+19.47%
		256	327.08	386.13	+18.05%
		512	361.79	440.30	+21.70%
Qwen3-4B UD-Q8_K_XL	26	16	180.53	180.67	+0.08%
		31	518.89	519.73	+0.16%
		32	534.56	537.40	+0.53%
		33	464.83	471.11	+1.35%
		64	664.52	674.70	+1.53%
		128	727.35	765.26	+5.21%
		256	1033.15	1137.06	+10.06%
		512	1101.21	1257.48	+14.19%
Gemma-4 E4B UD-Q4_K_XL	1	16	354.12	354.99	+0.25%
		31	456.70	456.10	-0.13%
		32	417.33	416.67	-0.16%
		33	406.07	407.05	+0.24%
		64	509.58	511.15	+0.31%
		128	814.35	820.05	+0.70%
		256	1083.95	1089.94	+0.55%
		512	1262.18	1274.05	+0.94%

RDNA1 and RDNA2 also lack native BF16 matrix acceleration, and rocBLAS appears to use the same generic HIP BF16 fallback family, which includes the 64x32x8, but I cannot confirm that the same N > 32 threshold is optimal there since I do not own an RDNA1/RDNA2 GPU.. If someone has an RDNA2/RDNA1 gpu could test if N>32 is still correct on thoses architecture ?

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, use for advice, testing help, understanding

[JohannesGaessler]

Performance

Table generated with scripts/compare-llama-bench.py

GPU	Model	Microbatch size	Test	t/s b10930	t/s bbd6d46	Speedup
MI60 / MI50	llama 1B BF16	1	pp1024	190.75	190.50	1.00
MI60 / MI50	llama 1B BF16	2	pp1024	328.27	328.27	1.00
MI60 / MI50	llama 1B BF16	4	pp1024	375.92	376.34	1.00
MI60 / MI50	llama 1B BF16	8	pp1024	385.51	385.29	1.00
MI60 / MI50	llama 1B BF16	16	pp1024	474.59	474.46	1.00
MI60 / MI50	llama 1B BF16	32	pp1024	900.49	899.37	1.00
MI60 / MI50	llama 1B BF16	64	pp1024	1262.93	1306.87	1.03
MI60 / MI50	llama 1B BF16	128	pp1024	1503.76	2096.05	1.39
MI60 / MI50	llama 1B BF16	256	pp1024	1764.00	2676.84	1.52
MI60 / MI50	llama 1B BF16	512	pp1024	1951.61	3016.20	1.55
P40	llama 1B BF16	1	pp1024	141.96	141.96	1.00
P40	llama 1B BF16	2	pp1024	244.40	244.46	1.00
P40	llama 1B BF16	4	pp1024	283.89	283.73	1.00
P40	llama 1B BF16	8	pp1024	276.78	276.49	1.00
P40	llama 1B BF16	16	pp1024	310.44	310.37	1.00
P40	llama 1B BF16	32	pp1024	573.96	573.94	1.00
P40	llama 1B BF16	64	pp1024	1180.25	1180.47	1.00
P40	llama 1B BF16	128	pp1024	1842.19	1843.27	1.00
P40	llama 1B BF16	256	pp1024	2074.58	2504.82	1.21
P40	llama 1B BF16	512	pp1024	2397.26	3070.49	1.28
RX 6800	llama 1B BF16	1	pp1024	139.60	139.74	1.00
RX 6800	llama 1B BF16	2	pp1024	221.83	221.85	1.00
RX 6800	llama 1B BF16	4	pp1024	324.00	323.98	1.00
RX 6800	llama 1B BF16	8	pp1024	413.97	413.83	1.00
RX 6800	llama 1B BF16	16	pp1024	301.17	301.15	1.00
RX 6800	llama 1B BF16	32	pp1024	588.37	588.27	1.00
RX 6800	llama 1B BF16	64	pp1024	745.28	871.42	1.17
RX 6800	llama 1B BF16	128	pp1024	1361.35	1370.69	1.01
RX 6800	llama 1B BF16	256	pp1024	1470.73	1871.72	1.27
RX 6800	llama 1B BF16	512	pp1024	1518.87	2304.89	1.52
V100-PCIE-32GB	llama 1B BF16	1	pp1024	335.43	335.40	1.00
V100-PCIE-32GB	llama 1B BF16	2	pp1024	549.81	549.83	1.00
V100-PCIE-32GB	llama 1B BF16	4	pp1024	656.01	656.38	1.00
V100-PCIE-32GB	llama 1B BF16	8	pp1024	881.73	882.29	1.00
V100-PCIE-32GB	llama 1B BF16	16	pp1024	287.45	921.26	3.20
V100-PCIE-32GB	llama 1B BF16	32	pp1024	568.46	1818.90	3.20
V100-PCIE-32GB	llama 1B BF16	64	pp1024	1111.54	2637.65	2.37
V100-PCIE-32GB	llama 1B BF16	128	pp1024	2054.49	3876.30	1.89
V100-PCIE-32GB	llama 1B BF16	256	pp1024	3288.20	4742.68	1.44
V100-PCIE-32GB	llama 1B BF16	512	pp1024	4137.98	5539.32	1.34

I pushed the corresponding logic for NVIDIA as well. Unfortunately I don't have a Turing card so I'm using the logic I determined for Volta.

[IMbackK]

Simple enough and logical change they put mutch more effort into tuning fp32 in the era these cards where made. I dont have any old enough hardware to add anything to the switchover parameters.

[eliah]

It would be great to make this change switchable via some environment variable, because, for example on my RDNA2 (RX6900XT) it drastically increases the memory requirements for loading models with BF16 layers and does not increase the prefill speed a lot. It is not a very advantageous exchange.

[thelittlefireman]

It would be great to make this change switchable via some environment variable, because, for example on my RDNA2 (RX6900XT) it drastically increases the memory requirements for loading models with BF16 layers and does not increase the prefill speed a lot. It is not a very advantageous exchange.

I think you could do that with GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16 maybe ?
On what model do you test ? If it's not useful for rdna2 we could exclu de it from the if switch.

[JohannesGaessler]

The default should remain the FP32 conversion where it is faster. It can be overridden via setting the environment variable GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16. Still, it may make sense to explicitly mention this tradeoff in the documentation of the environment variable.

[eliah]

Setting the compute type via GGML_CUDA_CUBLAS_COMPUTE_TYPE is not a good solution as well, because it leads to the forced layer type conversion, which definitely is not desirable in some cases. I just added the GGML_CUDA_CUBLAS_CONVERT_BF16 environment variable in my local branch of the code and checking it in the BF16 conversion condition to make disabled it by default unless the environment variable set. I also tried to automatically choose what type to convert the BF16 to, based on fast_fp16_hardware_available(cc) value, but somehow the processing becames unstable and fails from time to time with the memory-related errors when bf16 is being converted there.
@thelittlefireman - i'm testin on the https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF (Q5_K_M quantization) which has the output.weight encoded as BF16. @JohannesGaessler - in case of this model, conversion of the output.weight to fp32 eats about 4Gb of VRAM compared to the unconverted BF16 layer.

[LostRuins]

A P40 user mentioned this does increase the VRAM usage quite a bit, so it being togglable would be useful.

[JohannesGaessler]

The toggle already exists via setting GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16. On a P40 this should 100% restore the old behavior.

[thelittlefireman]

@LostRuins @eliah @JohannesGaessler

After some digging, I've managed to fix the high temporary-memory usage on model like:
Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-Q5_K_M.gguf by chunking.

We are still getting a performance improvement over native bf16, but with a little performance drop compare to unchuncked.

Configuration	Mean time	Median time	Throughput vs BF16	Peak VRAM	PPL
Native BF16	5.287 s	5.290 s	baseline	20.079 GiB	3.7871
F32, unchunked	5.164 s	5.160 s	+2.38%	25.060 GiB	3.7871
F32, 512 MiB chunks	5.122 s	5.120 s	+3.22%	20.618 GiB	3.7871

The idea is to let the chunk size is configurable through GGML_CUDA_CUBLAS_CONVERT_CHUNK_SIZE Var. (0 =disable, default=512mb)

The intresting thing is that it could also be useful in term of vram peak for F16->F32 conversations.
I Will try to make a draft pr asap.