[askmyteapot]

Custom CUDA builds with explicit GPU architectures can miss CUDA graph support entirely. With graph support enabled, Qwen3.8-Flash-Next also hits an ADD shape heuristic that prevents capture on CUDA0 during single-token decoding.

Changes

• Move GGML_CUDA_USE_GRAPHS outside automatic architecture selection, retaining the existing toolkit condition and architecture defaults.
• Add GGML_CUDA_GRAPH_ALLOW_ADD=1 to bypass only the ADD shape rejection. Default behavior and other compatibility checks remain unchanged.
• Identify the GPU and node in capture rejection diagnostics.

Why the ADD check is suspect

Upstream removed it in ggml-org#19645 (ad8207af7, February 17), explaining that exhaustive node-property checks made it unnecessary. After a series of reverts, Kobold commit 06c0ffa (February 23, "with am17an fix for henk to test") restored newer MUL_MAT_ID handling but kept a broad ADD check and removed its model-specific exceptions.

Both the comparison llama.cpp checkout (4413a192) and Kobold's newer merged upstream parent (f12aaa344) lack that ADD rejection. Maintainer context on why it was retained would be useful; this PR keeps the bypass opt-in.

Validation

Windows, CUDA 13.4, RTX PRO 5000 Blackwell + RTX 3090; explicit architectures 86-real;120a-real. Qwen3.8-Flash-Next-Uncensored IQ4_XS (qwen4exp), 36,244-token prompt, 81,920 context, batch 2,048 / ubatch 512, Flash Attention, F16 KV, manual layer split 33,16, 49 GPU layers. No MTP or expert cache.

Configuration	Prompt tok/s	Generation tok/s
Original Autofit/build	1634.31	30.37
Manual split, graphs compiled in, ADD still restricted	2254.12	37.78
ADD opt-in enabled, graph logging off	2290.44	80.20

The total improvement includes build and placement changes; it is not solely attributable to ADD capture. These are real-workload runs with differing completion lengths/content, not a controlled benchmark.

Diagnostics showed CUDA0 rejecting ADD capture every decode step (src1 ne[1]=4), while CUDA1 replayed. With the opt-in enabled, paired GPU graph reuse appeared. Output looked correct, with no logged CUDA errors. Rebuilt against 589e97dd8 and observed 78.74 tok/s in a subsequent run. Numerical/logit equivalence and broader model/hardware coverage remain untested.

The two proposed files match the locally tested source exactly; whitespace checks pass. Model-memory changes, temporary profiling instrumentation, configs, and binaries are excluded.

Investigation, code changes, and PR preparation were assisted by OpenAI Codex.

[LostRuins]

As mentioned on discord:

just wanted to pinpoint if its really just the add operation that's causing so much speed difference, i find that very odd.

even between cuda graphs / no graphs the difference shouldnt be that big either. Are there other variables?

[askmyteapot]

no lazy, graph ADD Allow.txt
lazy, graph ADD Allow.txt
No lazy load.txt
Lazy Load.txt

This is using the benchmark tool this time, rather than real-world generations. Fully offloaded its very fast, but still has a large difference between the ADD/ADD Rejection

[askmyteapot]

And some more benchmarks using the autofit code with 128k context (so its forced to offload some experts to sys ram)

I'll also note that all of these are with my changes to the cmakelists to ensure cuda graphs are compiled properly.

128k ctx, default kobold.txt
128k, all new flags.txt

[LostRuins]

merging, eventually ill probably remove the opt out if everything is fine