Repository navigation
Pin llama.cpp 96f81977: query-chunk prompt FA, ROPE token chunks, int4 SwiGLU prefill opt-in (27B KLD base re-recorded) - #493
Merged
Conversation
…4 SwiGLU prefill opt-in (27B KLD base re-recorded) llama.cpp 96f81977 (fork #120, #122, #121 on 5dcf6623): - #120: prompt flash attention above 2048 query rows as query chunks (27B -ub 4096: 6.5 -> 588 tok/s). - #122: ROPE above 2048 tokens as token chunks (Qwen3-4B now runs -ub 4096; KLD 0.000002). - #121: the int4-activation SwiGLU prefill (symmetric_i4_q8_plane) is opt-in, GGML_HRX_SYMMETRIC_I4_PREFILL=1. 27B prefill speed -0.0..-0.7% (interleaved x3). scripts/hw-gate.sh --bin on a private build (this llama.cpp + hrx-system 82804197 + gpu-kernels 08bb2241, strixhalo performance mode), check mode: PASS on every row (ops 667/667, kat, needle, placement, kld). The 27B kld row moved, as expected from #121: mean KLD 0.000000 -> 0.002222 (p99 0.01868, same top 97.65%, limit 0.01) against the base recorded with the int4 path on. The 27B base is re-recorded on that same build (scripts/hw-gate.sh --phases kld --models qwen3.8-27b --record): manifest sha256 d2f9a8af -> a31cb56c; the previous base file is kept on strixhalo as ~/.cache/hw-gate/kld/qwen3.8-27b.kld.pre-96f81977. Measured directly: the new default is the exact path - 27B -ub 2048 against the -ub 4096 logits 0.001949 -> 0.000003, PPL (c4096) 6.7183 -> 6.7097, PPL (c512, the gate args) 6.6835 -> 6.6381. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added
the
hw-gate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
llama.cpp 96f81977 (fork #120, #122, #121 on 5dcf6623):
GGML_HRX_SYMMETRIC_I4_PREFILL=1. 27B prefill speed -0.0..-0.7% (interleaved x3).
scripts/hw-gate.sh --bin on a private build (this llama.cpp + hrx-system 82804197 + gpu-kernels
08bb2241, strixhalo performance mode), check mode: PASS on every row (ops 667/667, kat, needle,
placement, kld). The 27B kld row moved, as expected from #121: mean KLD 0.000000 -> 0.002222
(p99 0.01868, same top 97.65%, limit 0.01) against the base recorded with the int4 path on.
The 27B base is re-recorded on that same build (scripts/hw-gate.sh --phases kld --models
qwen3.8-27b --record): manifest sha256 d2f9a8af -> a31cb56c; the previous base file is kept on
strixhalo as ~/.cache/hw-gate/kld/qwen3.8-27b.kld.pre-96f81977. Measured directly: the new
default is the exact path - 27B -ub 2048 against the -ub 4096 logits 0.001949 -> 0.000003,
PPL (c4096) 6.7183 -> 6.7097, PPL (c512, the gate args) 6.6835 -> 6.6381.
Auto-merge is off on purpose: merge by hand after the hw_gate job passes. Until this merges, a gate run of main on strixhalo will fail the 27B kld verify (the shared store now holds the new base; the old one is the .pre-96f81977 file next to it).
🤖 Generated with Claude Code