[danielhanchen]

Overview

Repins ggml-org#25731 (Inkling) from 534c321 to e1b89cf. 534c321 does not merge on any upstream base from b11760 on, so the nightly fails at the merge step on every tag it can pick now (the preflight on #306 says "Tonight's nightly will fail on base b11761 unless pr-set.json is repinned").

The prebuilt is upstream's aged b#### tag plus the pins in scripts/unsloth/pr-set.json, merged in order (git diff3, then additive_merge.py for pure additions only). We do not carry a fork: once upstream takes a pin, the pin is deleted.

scripts/unsloth/pr-set.json  +1/-1  (ggml-org#25731: 534c32182 -> e1b89cfa0)

feature-checks.json is unchanged (no pin is added, removed or folded).

What broke it

Upstream ggml-org#30836 (b684498, first in b11760, "HIP/CUDA: Fix reduction in mmf getting tile widths not devisiable by 64") changes ggml/src/ggml-cuda/ggml-cuda.cu from the WARP_SIZE constant to a per-device warp_size in ggml_cuda_mul_mat_id_needs_sync() and ggml_cuda_mul_mat_id(). These are the same lines where ggml-org#25731 adds its f32_pedantic gate in front of both ggml_cuda_should_use_mmf(...) calls. The base side of that hunk is not empty, so additive_merge.py refuses it and the merge aborts. On master's pr-set at b11785, ggml-org#25731 is REFUSED. It lists parsers.h, sources.cmake, ggml-cuda.cu, llama-arch.*, llama-model-saver.cpp, test-llama-archs.cpp and mtmd-audio.h because the whole merge aborts, but only ggml-cuda.cu is a real conflict.

Fix (in ggml-org#25731's own branch)

e1b89cf = 534c321 + git merge --no-ff b11767. This is one Fix merge conflicts commit, with nothing rebased or force-pushed. b11767 was the aged base when the merge was made, so the merge brings no upstream lines onto any base the nightly can still pick.

• ggml-cuda.cu: upstream's const int warp_size = ... lines and its warp_size call arguments are kept verbatim. Only the pin's f32_pedantic declarations and the !f32_pedantic && prefix on both ggml_cuda_should_use_mmf calls are re-applied.
• common/parsers/parsers.h, common/parsers/sources.cmake and tools/mtmd/mtmd-audio.h are add/add only (upstream's harmony-xml / muse-spark entries next to the pin's inkling entries). Both sides are kept.
• I grepped the pin's added lines under ggml/src/ggml-cuda for other old call shapes upstream changed (git diff b11767 -- ggml/src/ggml-cuda | grep '^+'). Every remaining WARP_SIZE use is in the pin's own new kernel code, where the constant is still valid upstream.
• The pin's own change keeps its shape: diff(b11767, e1b89cfa0) is 70 files, +4119/-126, against +4120/-126 for 534c321. The one-line difference is a ggml_cuda_should_use_mmf call that is now on one line, as it is upstream.
• ggml-org#25731 merges cleanly with ggml-org master again (git merge-tree of e1b89cf against master 0a1ef30 is clean).

Checks

Merge simulation

llama_pinset.py sim with master's pr-set plus this repin, on a full-history clone:

b11767 .. b11796 (all 26 tags)   ok: #25731 additive, pin_contract all 20 pins intact, merge_checks clean
master pr-set unchanged, b11785  FAIL: #25731 REFUSED (the break above)

verify --cuda (sm_100, local B200)

                                  b11782                b11795
CPU gate build                    ok                    ok
feature_matrix (CPU)              13/13, 10 need GPU    13/13, 10 need GPU
full CUDA build                   ok                    ok
test-chat                         pass                  pass
test-backend-ops MUL_MAT          2484/2484             2484/2484
test-backend-ops MUL_MAT_ID       993/993               993/993
test-backend-ops PAD              37/37                 37/37
test-backend-ops GET_ROWS         349/349               349/349
feature_matrix --gpu              13/13 (incl. inkling) 13/13 (incl. inkling)

A standalone CPU build of e1b89cf also passes: test-llama-archs 140/140 (inkling OK), test-mtmd-impl and test-chat.

Inkling numeric gate

The gate uses unsloth/Inkling-Small-GGUF UD-IQ1_S, wikitext-2 -c 512 --chunks 20, B200, with A = 534c321 as the reference. Error bars as printed by llama-perplexity: PPL about ±2.1, KLD ±0.0098, top-1 ±0.53.

arm                              PPL(Q)      mean KLD   top-1
A  534c32182 (A/A floor)         38.371526   0.000000   100.000%
B  e1b89cfa0                     39.199589   0.216343    82.333%
C  e1b89cfa0 + ggml-org master   39.199589   0.216343    82.333%

Under the gate's rule (KLD <= 1e-6, top-1 100%), B and C fail. The whole delta is one upstream kernel change, ggml-org#29720 (623c163, "CUDA: HIP: rms_norm: add variants that load float4 at a time and store x in registers", first in b11691). The conflict resolution adds none of it. Bisected by merging upstream commits into 534c321, each measured against A:

534c32182 + 5e03a1ff4 (the commit just before #29720)        38.371526  KLD 0.000000  top-1 100%     = A
534c32182 + 623c16393 (#29720)                               39.199589  KLD 0.216343  top-1 82.333%  = B
534c32182 + 99db719bd / 4c1004c49 (just before #30836)       39.199589  KLD 0.216343  top-1 82.333%  = B

So everything from b11609 to b11767 apart from ggml-org#29720 (including ggml-org#30836 and this merge's conflict resolution) changes nothing in the numbers. ggml-org#29720 changes the RMS norm reduction order. On CUDA it passes test-backend-ops -o RMS_NORM 53/53 and -o RMS_NORM_MUL_ADD 37/37. This 1-bit MoE model (sigmoid routing, top 6 of 256 experts, 42 layers) amplifies last-bit changes: turning off CUDA fusion in A's own build (GGML_CUDA_DISABLE_FUSION=1) moves it by a similar amount (KLD 0.208155, top-1 81.569%, PPL 40.885748).

ggml-org#29720 has been in every base the nightly has picked since b11691, including the current b11734-mix release. Any base the nightly can pick now carries it, whatever ggml-org#25731 is pinned to.

audit --pr-set

After pushing e1b89cf to ggml-org#25731's branch, the ggml-org#25731 row is ok (base=master, +4119/-126, 70 files). The FIX rows (#70, #95, #157, #264, #266, #298) show on master's own pr-set too and are not touched here.

Interaction with open pin-set PRs

• Pin unslothai#269 (server-side preemption and exact concurrency) at 18365c117 #211 (server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11577) #269 refreshed to 15b4e3b): master + server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11577) #269 at 15b4e3b + this repin merges on b11768, b11782 and b11785 with all 21 pins intact and merge_checks clean.
• pr-set: pin #303 (CUDA FA MMA in-kernel q4_0/q8_0 K/V dequant for 3-16 tokens) #306 (CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 at e886a61): CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 is refused in ggml/src/ggml-cuda/fattn-mma-f16.cuh and tests/test-backend-ops.cpp, because it collides with Add TML Inkling architecture ggml-org/llama.cpp#25731's lines in those files. It is refused the same way with the old Inkling pin (b11759, 534c321, where Inkling still merges), so the cause is CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 against Add TML Inkling architecture ggml-org/llama.cpp#25731, not this repin. It needs a fix in CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303's branch.

Not run

• ROCm, Vulkan, Metal, Windows and ARM legs (only the nightly runs them).
• GPUs other than B200.
• Upstream tags newer than b11796.
• The CPU preflight (unsloth-pin-preflight.yml) on this branch.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T12:46:59.435084Z	6b811ac	PR opened
🔒 Security Review	⚠️ Failed 2026-10-21T12:45:27.228343Z We couldn’t complete this review. Comment @codex security review to try again.	6b811ac	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins.

[danielhanchen]

Superseded by #312, which repins ggml-org#25731 to the same commit (e1b89cf) together with #304, #311 and #314.