Repository navigation

ggml: support quant scales for fp8 and nvfp4 - #28898

Draft

0cc4m wants to merge 28 commits into

osimons/ocp_fp8_cudafrom

0cc4m/fp8-scaled-mm

Draft

ggml: support quant scales for fp8 and nvfp4#28898
0cc4m wants to merge 28 commits into
osimons/ocp_fp8_cudafrom
0cc4m/fp8-scaled-mm

Commits on Oct 29, 2026
- ggml: add tensor scaling (per-tensor, per-channel, per-expert, per-expert-per-channel) support
  
  0cc4m
  authored and
  ORippler
  committed
  1750a6b
- add meta backend scale split
  
  0cc4m
  authored and
  ORippler
  committed
  544c64a
- add optional activation scales to mul_mat and mul_mat_id tensors
  
  0cc4m
  authored and
  ORippler
  committed
  117f1c3
- handle bitnet scales too, disable in backends
  
  0cc4m
  authored and
  ORippler
  committed
  78af0a0
- Don't treat single expert as per-channel-per-expert
  ORippler
  committed
  84ee0e6
- Temp workaround for moe-cache with scaled data-types
  ORippler
  committed
  7b48b40
- Also apply scales in CPU backend 's tiled mul_mat
  ORippler
  committed
  defc794
- Handle strided views on scales
  ORippler
  committed
  d1a2908
- Make *scale tensors use existing GGML_OP_MUL buffer probe
  ORippler
  committed
  bf1d8b9
- Make scale tensors required for ggml_type with needs_scale=true
  ORippler
  committed
  b94837a
- Make needs_scale=true for FP8
  ORippler
  committed
  6f301f6
- Use ggml_need_scale in code-base
  ORippler
  committed
  99800ea
- Transform companion FP8 scales
  ORippler
  committed
  71502d2
- Add comment explaining intended use of activation scales
  ORippler
  committed
  a25cc7e
- Fix pre-existing view-handling failure in meta backend
  ORippler
  committed
  0170784
- Enforce scale side-cars for scaled data-types during conversion
  ORippler
  committed
  df67d5e
- tq quants also need scales
  
  0cc4m
  authored and
  ORippler
  committed
  ea4fea9
- llama: use map for scale tensors instead of manual handling
  
  0cc4m
  authored and
  ORippler
  committed
  889a239
- Fix conversion edge-case for FP8 block-scales
  ORippler
  committed
  ce4bb48
- Revert "tq quants also need scales"
  ORippler
  committed
  b4fade2
- Disable dedicated kernels also for kleidiai in requant test
  ORippler
  committed
  55eb10f
- Add separate tests for scaled and unscaled DSV2
  ORippler
  committed
  83fc27e
- Also remove IQ from needs_scale related tests/behvaior
  ORippler
  committed
  4be2c38
- Wire activation scales and disallow broadcasting over K
  ORippler
  committed
  70bd1d1
- Adjust build_* in lfm3.cpp
  ORippler
  committed
  141302a
- Avoid internal includes in test-llama-arch for moe scales
  ORippler
  committed
  6138f6d
- djust build_* in limite.c
  ORippler
  committed
  62bff84
- Adjust build_* for mistral 4
  ORippler
  committed
  4c00151

Commits

Commits on Oct 29, 2026