Feature: BF16/FP16 conversion to f32 chunking by thelittlefireman · Pull Request #29442 · ggml-org/llama.cpp

thelittlefireman · 2026-09-25T19:47:26Z

Overview

Follow #28846
Chuncking the BF16/FP16 to FP32 convertion for lowering VRAM usage.
Set chunck by 512MB after Heuristics tests.

Additional information

Test with BF16 to FP32 and FP16 to FP32 (useally when FA is set to OFF and context if growing)

MI50/gfx906:

Test	Master / F32 unchunked	New 512 MiB sliced implementation	Difference
`pp512`	215.82 ± 3.07 t/s	216.04 ± 2.88 t/s	+0.10%
`pp1024`	223.04 ± 1.67 t/s	223.22 ± 2.33 t/s	+0.08%
PPL	3.7871	3.7871	identical
Peak VRAM	25.060 GiB	20.614 GiB	-4.446 GiB (-17.7%)

@LostRuins @eliah Can you test an tell me if it's ok for you ?

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES for help, understanding and code generation.

1luik · 2026-09-26T11:00:13Z

Did you upload the compilation script

thelittlefireman · 2026-09-26T15:50:55Z

Did you upload the compilation script

I don't understand. GGML_CUDA_CUBLAS_CONVERT_CHUNK_SIZE could be used at runtime does not need specific compilation script.

JohannesGaessler

The implementation is way too complicated for what it is. All that should be needed is the on-the-fly creation of tensors that are slices of the original tensors in a loop.
The size of 512 MiB should just be hard-coded. I don't consider it a parameter important enough to expose to users since that bloats the UI.

thelittlefireman · 2026-09-27T09:19:57Z

The implementation is way too complicated for what it is. All that should be needed is the on-the-fly creation of tensors that are slices of the original tensors in a loop.

The size of 512 MiB should just be hard-coded. I don't consider it a parameter important enough to expose to users since that bloats the UI.

Is it better now ?
I also re-ran the real-model tests on MI50/gfx906:

Test	Master / F32 unchunked	New 512 MiB sliced implementation	Difference
`pp512`	215.82 ± 3.07 t/s	216.04 ± 2.88 t/s	+0.10%
`pp1024`	223.04 ± 1.67 t/s	223.22 ± 2.33 t/s	+0.08%
PPL	3.7871	3.7871	identical
Peak VRAM	25.060 GiB	20.614 GiB	-4.446 GiB (-17.7%)

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

thelittlefireman · 2026-10-03T12:03:57Z

@JohannesGaessler Done. Also edit the initial PR message

thelittlefireman · 2026-10-04T22:44:58Z

@JohannesGaessler. Fix the dst stride for chunked, Sorry for the failling test.

Resolves the conflicts with upstream ggml-org#29535 (K2 Horizon) and ggml-org#29442 (cuBLAS ldc and F16/BF16 src0 chunking): - Inkling and K2 Horizon both register: parser decl, parser source, model mapping case, EOG token, output tensor name. model.unembed moves a few lines up the OUTPUT tuple, away from upstream's embedding_projection, so pins that carry that line still merge. - LLAMA_VOCAB_PRE_TYPE_INKLING moves to 62, since upstream claimed 61 for K2_HORIZON. The id is internal; the GGUF carries tokenizer.ggml.pre as a string. - ggml_cuda_mul_mat_cublas_impl keeps the F32_PEDANTIC GemmEx path and passes upstream's ldc to both F32 branches; the pedantic compute-type override stays ahead of upstream's F16/BF16 src0 chunking, which it cannot overlap (pedantic only applies to F32 src0).

* ggml-cuda: chunk large BF16/FP16 to F32 conversions * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * ggml-cuda: respect dst stride in chunked cuBLAS matmul --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> (cherry picked from commit ab09ea4)

* ggml-cuda: chunk large BF16/FP16 to F32 conversions * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * ggml-cuda: respect dst stride in chunked cuBLAS matmul --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

…he replay case measure its own noise floor test-llama-context went from "CUDA error: out of memory" to 15/15 on this machine. Two independent defects, both reproduced and both measured. cuda/mul_mat_cublas: ggml-org#29442 chunks the conversion of src0 to the compute type, but only for F16/BF16 sources. A quantized weight reaches the same code path whenever the device has no fast fp16 (Maxwell, consumer Pascal) -- exactly the devices with the least free memory -- and is converted whole: the 254 M element Q4_K output layer becomes a 970 MiB f32 temporary, which a 2 GiB card cannot hold, so the graph died inside cuMemSetAccess with the device at 0 bytes free. Extend the row chunking to quantized sources (one row is a whole number of blocks, so it is as self-contained as an f16 row) and take the chunk budget from the free memory (a quarter of it, capped at the old 512 MiB), because a fixed budget cannot fit a 2 GiB device. Splitting along rows leaves each output element's K reduction untouched, so the numbers do not change. tests/llama-context/replay: the case compares a rolled-back sequence against a reference that never advanced past the rollback point, but the two cannot share the batch geometry -- the rolled-back context decodes the anchor inside the 4-token ubatch it is about to remove, the reference decodes it alone. A token's own arithmetic is not independent of that geometry (bit-exact for f16/q8_0 on CUDA, 1.6e-3 nmse for a 2-bit cache, 1.3e-3 for f16 on CPU) and the recurrent state carries the difference through the whole replay, so the fixed 1e-4 bound only held for one backend/type combination -- and non-monotonically: a q4_0 K cache fails while q4_0/q4_0 passes. Measure the geometry sensitivity in the case itself, on two short lived probe contexts, and scale the bound with it. Evidence. - Pool probe: cuMemSetAccess out of memory on device 1 (GTX 950, 2047.88 MiB) with free=0.00 MiB; the growth log names the node: result_output MUL_MAT, 970.00 MiB requested, 254279680 elements == n_vocab*n_embd at 4 B. - After the change the same run reserves 338.91 MiB instead of 970.00, and -dev CUDA1 -p 64 (a 2 GiB card, which always failed before) runs at 379.58 t/s. - llama-bench -dev CUDA0 -p 512 -n 32: pp512 13395 t/s, tg32 259.72 t/s (13392 / 258.18 before) -- no regression from the chunking. - test-llama-context --model Qwen3.5-0.8B-Q4_K_M.gguf (both GPUs, default KV): 14 OK / 0 FAIL / 1 SKIP (rot, by design), RC=0, three runs in a row; -dev CUDA0 the same; -dev CUDA0 -ctk f16 -ctv f16 gives 15/15. - replay alone passes with the default KV, f16, q8_0, q5_0, turbo4, turbo2_tcq and q4_0/f16 cache types. - Reverse control: making s_copy() ignore the pending rs_idx turns the case red again on both the default and the f16 configuration (nmse 8.8e-3 / 8.6e-3), so the new bound still catches the failure the case exists for. - ctest test-vbr-* and test-kv-cells pass; test-download-model fails for lack of network and test-thread-safety is skipped as its dependency, but the latter runs to RC=0 by hand.

* ggml-cuda: chunk large BF16/FP16 to F32 conversions * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * ggml-cuda: respect dst stride in chunked cuBLAS matmul --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

thelittlefireman requested review from a team and ggerganov as code owners September 25, 2026 19:47

github-actions Bot added documentation Improvements or additions to documentation testing Everything test related devops improvements to build systems and github actions ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Sep 25, 2026

JohannesGaessler requested changes Sep 26, 2026

View reviewed changes

ggml-cuda: chunk large BF16/FP16 to F32 conversions

ffa6c80

thelittlefireman force-pushed the feature_bf16_f32_chunking branch from 9045583 to ffa6c80 Compare September 27, 2026 08:56

thelittlefireman requested a review from JohannesGaessler September 27, 2026 09:47

JohannesGaessler reviewed Sep 29, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Resolved

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Resolved

thelittlefireman and others added 2 commits October 3, 2026 13:52

Update ggml/src/ggml-cuda/ggml-cuda.cu

c634997

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

Update ggml/src/ggml-cuda/ggml-cuda.cu

6314f0c

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

JohannesGaessler approved these changes Oct 4, 2026

View reviewed changes

ggml-cuda: respect dst stride in chunked cuBLAS matmul

7363a5c

JohannesGaessler approved these changes Oct 5, 2026

View reviewed changes

JohannesGaessler added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 5, 2026

ggerganov merged commit ab09ea4 into ggml-org:master Oct 6, 2026
14 of 18 checks passed

danielhanchen mentioned this pull request Oct 7, 2026

unsloth: repin Inkling, #241 and #243 past b11453, drop #247, pin --moe-cache-mib auto (#251) unslothai/llama.cpp#252

Merged

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

Navigation Menu

Repository navigation

Feature: BF16/FP16 conversion to f32 chunking - #29442

Feature: BF16/FP16 conversion to f32 chunking#29442
ggerganov merged 4 commits into
ggml-org:masterfrom
thelittlefireman:feature_bf16_f32_chunking

thelittlefireman commented Sep 25, 2026 •

edited

Loading

Uh oh!

1luik commented Sep 26, 2026

Uh oh!

thelittlefireman commented Sep 26, 2026

Uh oh!

JohannesGaessler left a comment

Uh oh!

thelittlefireman commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

thelittlefireman commented Oct 3, 2026

Uh oh!

thelittlefireman commented Oct 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

thelittlefireman commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

1luik commented Sep 26, 2026

Uh oh!

thelittlefireman commented Sep 26, 2026

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

thelittlefireman commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

thelittlefireman commented Oct 3, 2026

Uh oh!

thelittlefireman commented Oct 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thelittlefireman commented Sep 25, 2026 •

edited

Loading