[0cc4m]

Overview

This is a prototype branch on top of @ORippler's FP8 branches, moving quant scale support into GGML. To keep it simple, it requires backends to support them for FP8 and NVFP4. Support is very simple to implement with an epilogue MUL. I also tested options for graph-based fallbacks, but they add a lot of complexity, so I think this is easier. It is backwards compatible with existing uses of MUL_MAT and MUL_MAT_ID, while supporting direct conversion of many FP8/NVFP4 safetensors models on HF directly.

For validation I also have Vulkan support added, because the primary purpose of this branch is just discussion. I can split it later if necessary.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Claude was used

[0cc4m]

@ggerganov As soon as we add FP8 support, we'll have two types that require scales. I would like to avoid the llama.cpp (non-ggml) workarounds and instead support this case in ggml directly. The simplest way for that is to allow a weight scale tensor and an optional activation scale tensor as src to ggml_mul_mat and ggml_mul_mat_id, only for the types that require these (nvfp4 and fp8).

This branch (mostly commit 394d7a8) is a rough example of how that could be done, based on @ORippler's work and discussed with him. There is a breaking change for nvfp4, since it removes the graph-based MUL + backend-internal fusion currently used for nvfp4, but for any other type it is completely backwards compatible. Since nvfp4 is still very fresh I don't think that's a big issue.

I think this is better than requiring each backend to fuse internally and also permits optimizations for src1-quantization through preset activation scales that models ship with.

Before we pursue this further, what is your opinion on the approach? We would really like to unblock FP8 and improve NVFP4 support.

[ggerganov]

I looked at the stack and the proposed changes are OK. IMO this functionality is overall quite orthogonal to most other stuff so we can afford to move forward quickly - even if we realize some better approach later on it would be simple to adjust (just like how now we will step back on the backend fusion approach).

Feel free to proceed with this as you find best. I think @jeffbolznv also had some thoughts/interest in FP8, so pinging to keep him in the loop.