Repository navigation
llama : load per-layer embedding tables from a separate GGUF (--ple) - #152
Merged
Merged
Conversation
--ple FILE opens one more GGUF after the model files. Each tensor in it replaces the lookup table of the same name in the model, so a model can use a PLE / n-gram table of another precision (e.g. BF16 or Q8_0 instead of the IQ4_NL table it ships) without a rebuilt model file. The table of the model file is never read. Works for every load and lazy mode, including the qwen4exp on-direct reader, which reads the rows from the new file. - every key of the file must match the model in type and value, except split.* and general.* (general.architecture is compared); a file without such keys loads with a warning - a replaced tensor must keep its shape; only get_rows lookup tables can be replaced; a file without tensors is an error - the replaced byte ranges of the model are kept out of the mmap prefetch - the path is part of the model params in common, so a draft or prefill model made from a copy of the params does not get the file - vocab-only loads do not open the file - path_ple is the last member of llama_model_params Assisted-by: Claude Opus 5.5 Assisted-by: Claude Fable 5.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
--ple <file|none> takes several values like the other params, so tables of different precision can be compared in one run. Adds a ple output field. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
documentation
michal-zurkowski
marked this pull request as ready for review
October 7, 2026 18:31
Copies the PLE table of a model (any shard of a split model) to its own GGUF with the arch's ple.* keys, as a byte copy or converted to f32 / f16 / bf16 / q8_0 / q5_x / q4_x. --metadata-from takes the keys from another file, for a lone shard that has none. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michal-zurkowski
force-pushed
the
ple-sidecar
branch
from
October 7, 2026 18:41
3ba6fad to
3727eee
Compare
RowStream duck-typed the ndarray that GGUFWriter.write_tensor_data() declares, which ty rejects; the previous commit silenced that with an ignore. Let GGUFWriter write the header, keys and tensor infos only, and append the data section with its alignment from write_rows(). The output is byte-identical to before. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dzannotti
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds
--ple FILE: a GGUF whose tensors replace the lookup tables of the same name in the model. The model's own copy of a replaced table is never read.Why: Qwen3.8-Flash-Next (
qwen4exp) has a hashed n-gram table,per_layer_token_embd.weight, of 160 x 320,001,536 values (102.4 GB in BF16). Most GGUF quants ship it as IQ4_NL (28.8 GB), some as Q5_0 or Q8_0, whatever the precision of the rest of the model. On Strix Halo the table stays on disk (--lazy-mode on-direct) and only the rows of each batch are read, so a better table costs disk space, not RAM - but until now the only way to use one was a rebuilt model file (hand-patched split GGUFs). With--pleone table file serves every quant of the model.What it does:
llama_model_loader::load_ple(), called fromllama_model_loadright after the loader is built, adds the file as one more model file and points the matchingweights_mapentries at it. Everything downstream resolves tensors throughweights_map, so every load and lazy mode follows, including the qwen4exp on-disk row reader (llama_ple_disktakesml.fnames[idx]andoffs).split.*andgeneral.*(other thangeneral.architecture) must match the model in type and value; a file with none of them (e.g. a non-first shard of another split model) loads with a warningtoken_embdis refused because of its output use); a file without tensors is an errorllama_mmaptogether with the lazy ranges, so the mmap prefetch skips them.common: the path is a member ofcommon_params_model, so a draft or speculative-prefill model built from a copy of the params does not get the file; an MTP head that runs on the target weights shares the target's table.path_pleis the last member ofllama_model_params.llama-bench --ple <file|none>(multi-value, newpleoutput field).gguf-py/gguf/scripts/gguf_extract_ple.pywrites the table of a model (any shard) to its own GGUF with the arch'sple.*keys, as a byte copy or converted to f32/f16/bf16/q8_0/q5_x/q4_x.Tested with qwen4exp (UD-Q4_K_XL) and gemma4 E2B. gemma3n uses the same tensor and key but was not run.
Without
--plenothing changes: outputs are byte-identical to the merge-base (below).Commits:
4e5b6913dllama/common,c67218b4ellama-bench,3727eee22gguf-py,49cd270b7gguf-py fix for the ty type-check (writes the tables without faking an ndarray; output byte-identical).Measurements
This is a feature, not a speedup: no compute path changes. The tables show that neither the change nor the table precision moves speed. llama-bench
-p 2048 -n 128 -b 2048 -ub 2048 -r 4 -ngl 99 -fa on -ctk f16 -ctv f16 --load-mode dio --lazy-mode on-direct(ROCm also-d 0,12000); llama-bench feeds random tokens, the worst case for the table (every row is a cache miss). The three table files and the model shard that holds the embedded table were dropped from the page cache first. Run order: ROCm - baseline, change in two passes of opposite order, baseline again; Vulkan - baseline and change interleaved (baseline, change, baseline, change with all tables, baseline).Baseline (ROCm 10, merge-base, run first and last):
After (ROCm 10, 3ba6fad80, two passes in opposite order):
pass 1: none -> Q8_0 -> BF16
pass 2: BF16 -> Q8_0 -> none
The size column counts the table file (103.68 GiB with no
--ple, 127.52 GiB with Q8_0, 172.22 GiB with BF16). pp2048: baseline 1524.7 / 1527.5, change 1517.6 - 1522.6 for all tables in both passes (+- 34-44 each); pp2048 @ d12000 1400.6 - 1419.2 vs baseline 1409.8 / 1422.8; tg128 24.2 t/s and tg128 @ d12000 22.8 t/s everywhere. No difference outside the standard deviations.Vulkan, interleaved (same command without
-d):baseline, change (none), baseline, change (none, Q8_0, BF16), baseline
Vulkan pp2048: baseline 848.4 / 848.6 / 848.7, change 847.0 (none), then 847.6 / 846.6 / 847.5 (none / Q8_0 / BF16), +- 2-4 each; tg128 27.57-27.66 everywhere. A first, non-interleaved Vulkan run (baseline first) had shown the change 1.5 % lower; interleaved, the difference is gone, so that was run order.
Correctness:
Exact-logit harness (token ids, fnv64 of the logits of every prefill position, full final logits, 32 greedy decode steps with full logits), 4 repo corpora + the 4 concatenated 6x (16176 tokens),
-c 16384 -b 512 -ub 512:--ple, dio + on-direct--plewith a byte copy of its own IQ4_NL table--pleBF16 table--pleIQ4_NL with--lazy-mode on(mmap)--pleand--plewith its own table; resident, lazy, dio; lazy + mmap on CPU--ple--spec-type draft-mtpwith a shared MTP head, 6 prompts: no--plevs--pleIQ4_NL; hand-built BF16 file vs--pleBF16 (ROCm 10)Errors (all at load, before any weight is read; ROCm 10 build, CPU): architecture mismatch, changed key value, same value with another int type, shape mismatch, matmul weight in the file, tied
token_embdin the file, file without tensors, unused tensor, missing file; a valid file loads. Corrupted table with--check-tensors(both the read and the mapped path) (*).mmap prefetch (*): after an mmap load on CPU, 0.00-0.02 % of the replaced 28.8 GB range was in the page cache; with the
init_mappingshunk reverted, 46 % (13.2 GB) after 60 s and rising.Table quality on ROCm 10 (
llama-perplexity, 150 x 512, reference = UD-Q4_K_XL with the BF16 table, held-out corpus written after the model's cutoff, and wikitext-2 as a memorized-text control); every value matches an earlier run on ROCm develop-3821 to all printed digits:MTP draft acceptance, 24 prompts (*): BF16 65.2 %, Q8_0 64.8 %, IQ4_NL 66.0 %; all pairwise 95 % bootstrap intervals contain 0.
(*) measured on the previous base e3af5e8 with ROCm develop-3821 (+PM4 patch); the patch is byte-identical. Everything else was run on 4e0d917 / 3ba6fad80 with ROCm 10.0 and Vulkan.
test-backend-ops: not run, no ggml op changed.
Additional information
--lazy-mode on-direct,src/llama-ple-disk.cpp) is what makes a 54-102 GB table usable on a 128 GB unified-memory box, and all measurements are from here. The loader and gguf-py commits cherry-pick cleanly onto halo-box/llama.cpp master (d15f754, 2026-10-07) and compile there (CPU build, not run), so the change can be proposed there later; the llama-bench commit needs a small port.mmap+mlockwith CPU-mapped weights (-ngl 0,--cpu-moe):load_all_datagrows the lock from the start of the mapping, so the replaced range of the model file gets locked too. The same happens today for lazy tables. Not testable here (RLIMIT_MEMLOCK hard limit 8 MiB); GPU-offloaded weights do not take that path.--load-mode mmap(orauto), a model file that holds a replaced table is mapped withoutMAP_POPULATEand getsMADV_WILLNEEDon the rest instead, the same as a file with a lazy range today. For qwen4exp the table is lazy anyway, so nothing changes; for a model whose table is not lazy (gemma4 E2B) the first forward pass faults in the pages the prefetch did not reach yet.scripts/compare-llama-bench.pyignores the newplefield (as it doeslazy_mode): adding it as a key would make the script reject output of builds without it.Requirements
--plefile; fixed in the first commit) and asked for the stricter checks that are in it; the second found no further code defect and checked this text against the logs. Re-tested after rebasing onto current master with ROCm 10.0.--lazy-mode off, too large next to the weights); Windows (the mmap range code is shared, not run); CUDA, Metal and other backends; gemma3n and gemma4 E4B; contexts above 16K tokens in the logit comparison (llama-bench ran at depth 12000);scripts/compare-llama-bench.py; the test binaries (tests/) were not built locally, only llama-server, llama-bench, llama-perplexity and llama-completion; the halo-box/llama.cpp cherry-pick was compiled, not run.🤖 Generated with Claude Code