Repository navigation
Conversation
|
You should break this PR into 2 separate pieces. One for each change |
Sure, will split the changes |
cee25e2 to
6e6c343
Compare
|
why are the results so good with nvfp4 and noisy with q4_k_m? |
|
I actually couldn't reproduce these results on a RTX Pro 6000. I tried |
Q4_K_M checkpoint had GDN projections quantized to Q8_0, so we dequantize Q8_) in that path which affects perf |
I collected perf numbers with speedbench on coding dataset. I will collect perf numbers with llama-bench with small ub. |
Compared to the NVFP4 model which has f32? |
Sorry, missed to mention, for NVFP4 checkpoint from both unsloth and radixark are in bf16 prec while for Q4_K_M checkpoint from unsloth it is in Q8_0. |
|
I think what is happening is that bf16 path is falling to cublas for these thin matrices i.e. MMVF is only getting enabled for bs=1. What this points to is that we need better kernels for bf16 for n=2..8, the bf16 cublas one is too slow, forcing MMVF there seems to help |
…lat in server A/B; not adopted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018co7fcJoEQXMxLezg7HpMU
Sorry for the delay was out on holiday for some time. Performance RTX-6000 PRO-BLACKWELLCollected with -p 256 -n 0 -ub 1,2,3,4,5,6,7,8 -r 5
|
|
Yes, but you should make the bf16 change in separate PR, we can merge that and then independently check this PR again. |
|
Seams slightly beneficial for HIP gpus too, no surprise.
|
|
|
||
| const int cc = ggml_cuda_info().devices[cuda_ctx->device].cc; | ||
| if (!use_cuda_graph || cuda_ctx->curr_stream_no != 0 || | ||
| !GGML_CUDA_CC_IS_NVIDIA(cc) || cc < GGML_CUDA_CC_TURING) { |
There was a problem hiding this comment.
| !GGML_CUDA_CC_IS_NVIDIA(cc) || cc < GGML_CUDA_CC_TURING) { | |
| GGML_CUDA_CC_IS_MTHREADS(cc) || (GGML_CUDA_CC_IS_NVIDIA(cc) && cc < GGML_CUDA_CC_TURING)) { |
Signed-off-by: ynankani <ynankani@nvidia.com>
3d3d159 to
e75c82b
Compare
|
Performance improvements dropped to 2-4% after #29633. Not sure if this much complexity is ok to be introduced for small perf improvements. Performance RTX-6000 PRO-BLACKWELL
|
|
Let's hold off then. I would think something like automatic fusion using nvrtc is a better long term solution like the inductor framework. Perhaps worth a PoC. |
Agreed that a generic solution would be better long term. Few questions regarding this:
|
|
All are open for discussion. From what I experimented with, the virtual table doesn't yield much benefit vs nvrtc. I tried fusing elementwise point wise and row wise ops. |
Overview
Fuse GDN alpha/beta projections and MMVQ small-batch epilogues. This PR fuses below two paths
Additional information
Performance RTX-6000 PRO-BLACKWELL
It is showing up to 13% perf benefits on RTX-6000-PRO-BLACKWELL, will collect scores on other hardware also
Requirements