Repository navigation
Conversation
j-f1
reviewed
Mar 13, 2023
Co-authored-by: Jed Fox <git@jedfox.com>
Member
|
Hi, thanks for contributing! |
|
argparse is part of the Python standard library, so this PR does not add any new dependency. |
aldehir
pushed a commit
to aldehir/llama.cpp
that referenced
this pull request
Jan 4, 2026
jinja whitespace tests
spiritbuun
referenced
this pull request
in spiritbuun/buun-llama-cpp
Apr 6, 2026
…wer) Fused MMA flash attention is fundamentally faster because it avoids materializing the O(nq×nkv) score matrix. cuBLAS GEMM must write/read S as intermediate, adding bandwidth overhead that dominates tensor core gains. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
spiritbuun
referenced
this pull request
in spiritbuun/buun-llama-cpp
Apr 6, 2026
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* iqk_mul_mat: better iq4_nl implementation on Zen4/AVX2 PP-512 performance for LLaMA-3.1-8B goes to 162.6 t/s up from 133.2 t/s. * Fix AVX2 In addition to fixing iq4_nl, it seems I never adhusted the AVX2 implementation for iq2_tn to the block scale removal? This commit also fixes that. --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
wmeddie
added a commit
to XpressAI/ve_llama.cpp
that referenced
this pull request
May 29, 2026
…aph (ggml-org#72) VEBP keeps token_embd as F16 (tied, so it's both the embedding and the lm_head). The VE has no F16 path, so GET_ROWS + that MUL_MAT were refused and the graph fragmented. Two changes make VEBP fully self-contained and let its prompt graph compile: 1. F16 -> BF16 on HBM upload (same 2-byte size, strides unchanged), served by the existing BF16 GET_ROWS / matvec / colmajor paths. get_rows/mul_mat supports + dispatch accept F16; the graph compiler maps src_type F16->BF16 and uploads the converted copy. Conversion uses the F16C row helpers in 1M chunks (a per-element call loop was ~10 s for the 621M-elem token_embd, charged to the first prompt eval). 2. Removed a stale guard that refused Q4_K/VEBP MUL_MAT when ne[1] != 1. It dated from when N>1 was refused entirely; now the codegen loops the matvec _inner over the n_tok columns, so it handles N>1. The guard was rejecting EVERY VEBP prompt graph (Qcur is VEBP, ne[1]=N) -> the prompt silently ran on the interpreter. (A new first-execute verbose log made this visible.) Also fixed N>1 codegen vectorization found while profiling (the .L showed it was NOT a table overflow — it was unvectorized loops): the MUL `src1[e % period]` modulo forced scalar code -> restored the nested broadcast form; added `restrict` to the element-wise / RMS_NORM / GLU pointers (void*-cast aliasing). Measured (GGML_VE_HBM=1, -fa on, -ctk/-ctv bf16, warm, run twice): Llama-3.2-3B prompt 57 -> 65, decode 48 -> 56 (vectorization fixes) Bonsai-VEBP prompt 9.8 -> 12.65 (1.29x, V.OP 12% -> 92%), decode 33.5 -> 38.7 All outputs token-for-token identical to the interpreter. VEBP's prompt gain is smaller than Llama's 16x because its per-token ternary matvec is compute-bound (V.OP already 92%), so the fork/join fusion saves a smaller fraction. Further gains need a batched ternary matmul (read the weight once across N columns) — a follow-up.
HEchternacht
pushed a commit
to HEchternacht/llama.cpp
that referenced
this pull request
Jul 18, 2026
ggml_vec_dot_q2_0_q8_0 gated its VNNI fast path on __AVX512VNNI__ && __AVX512VL__ with no AVX2/AVX-VNNI fallback, so x86 CPUs without AVX-512 silently took the scalar loop. This excludes all Intel 12th-14th gen consumer CPUs (Alder/Raptor Lake), where AVX-512 is fused off for the P/E hybrid design but AVX-VNNI is present. The fast-path body is already entirely 256-bit AVX2; the only AVX-512 dependency is the _mm256_dpbusd_epi32 intrinsic. AVX-VNNI exposes the identical operation as _mm256_dpbusd_avx_epi32, so alias the intrinsic and widen the guard. No algorithmic change, and no behavior change on AVX-512-VNNI hosts. Measured on Intel i5-13400 (Raptor Lake, AVX-VNNI, no AVX-512), 12 threads, CPU-only, same model and prompt (temperature=0): Ternary-Bonsai-8B Q2_0 decode: 2.17 -> 6.92 tok/s (3.2x) Ternary-Bonsai-8B Q2_0 prompt eval: 2.7 -> 8.6 tok/s (3.2x) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
My take at #70.
I tried using
blackfor formatting, but things are not in a great shape now, so I just tried to match the format on the lines I had to change.This looks correct to me, but I haven't been able to run the converted model yet, so I can't verify. I was able to convert it just fine, but got some tensor dimension mismatch later.