Repository navigation
model: add Qwen3.8-Flash-Next (qwen4exp) - #27742
Merged
Merged
Conversation
Adds the GGUF-side plumbing for HF model_type qwen4_exp: - MODEL_ARCH.QWEN4EXP plus tensors for the low-rank hyper-connection variant (hc_*_norm/down/up/inject) and the PLE n-gram hash embeddings. The DeepSeek-V4 hc_*_fn/base/scale tensors are a different parameterisation, so these are separate entries rather than reuse. - Reuses the existing indexer, per_layer_token_embd, SSM and compress_ratios keys unchanged. - conversion/qwen4exp.py inherits the Qwen3.5 linear-attention V-head reorder and interleaved mrope, concatenates the 128 PLE embedding shards, and splits index_qk_proj into separate indexer q/k tensors. The PLE hash multipliers reach ~2.4e13. prepare_tensors() casts every non-float dtype to float32 before modify_tensors() runs, and GGUF array writes infer INT32 from Python ints, so both paths are bypassed: the constants are read from the pre-cast lazy tensors and written as explicit UINT64 arrays. Additive only; no existing arch changes behaviour.
Adds LLM_ARCH_QWEN4EXP with its hparams and tensor loading. The graph comes in the next commit; this makes the model load and report correct metadata. - hyper-connections set n_embd_out_impl = hc_count * n_embd, so the residual stream is 4x wide and there is no output_norm: the final mixer's hc_norm is the last norm in the model. - registered as hybrid and given the same recurrent/attention memory filters as Qwen3-Next and Qwen3.5. - reuses the existing indexer, per_layer_token_embd, SSM and compress_ratios keys as-is. - the PLE table row count is read back from the file rather than recomputing the vocab padding rule. llama-model-loader gains UINT64 array support. That branch previously threw, so no existing caller changes behaviour; it is needed because the PLE hash multipliers do not fit in int32.
Implements the decode graph for Qwen3.8-Flash-Next: the hyper-connection residual stream, gated delta net layers, the MoE block with its gated shared expert, and dense full attention. The QSA indexer and the PLE n-gram embedding are not wired up yet and land in later commits. Hyper-connections are implemented here rather than shared with deepseek4.cpp. The two formulations agree on the [n_embd, hc, n_tokens] layout and little else: DeepSeek-V4 mixes with a full-rank projection and Sinkhorn-normalises it, whereas this model uses a low-rank down/silu/up sigmoid gate and collapses by a plain mean. Only the ~10 line stream mean is genuinely common, so sharing would mean touching DSV4's hot path and its three fused CUDA ops to reuse very little. What is reused is the substantive part: the LLM_KV_HYPER_CONNECTION_* keys, the n_embd_out_impl wide-residual support already in the loader, and the layout convention. Also allows a checkpoint to carry no PLE layers at all, which makes it possible to bring the graph up and validate it in stages. Validated against vLLM, the only working reference implementation. On a scaled-down model with an init scale large enough to give non-uniform logits, agreement with vLLM sits at the numerical noise floor: llama.cpp f32 against its own bf16 gives 84.3% top-1 agreement over 255 positions, and this graph against vLLM gives 85.1%. The comparison was calibrated by seeding three deliberate bugs (silu instead of sigmoid on the delta net gate, dropping the 1/hc scale in the mix, dropping the 2x in the combine); each drops top-1 to between 0% and 11%, an order of magnitude below the floor.
Adds the per-layer embedding: a custom I32 graph input hashes each token with its ngram_size-1 predecessors host-side and the result is a plain row gather over the shared table, the same shape gemma3n's per-layer embedding uses. The hash has to run on the host because the splitmix64-derived multipliers reach 2^45, so the products need 64-bit integers and an xor, neither of which ggml has. Predecessors that fall outside the ubatch come from a small per-sequence history on the model, mirroring the per-request ngram_context the reference carries. It is only trusted when contiguous with the incoming position, so a fresh prompt or a rewound cache falls back to EOS padding rather than hashing against stale tokens. The depthwise conv is written out as a sum of shifted, per-channel-scaled copies rather than through ggml_conv_1d_dw, which carries a correctness warning upstream. Verified two ways. The row indices match a transcription of the reference's tensor formulation exactly, 1024 of 1024 rows, including sequences with EOS tokens sprinkled through them to exercise the segment reset. Separately, with PLE placed on layer 0 so its input is just the token embedding, ple_embd and ple_gated_value match a PyTorch computation from the same checkpoint to every printed digit. End to end over 1023 scored positions the port sits the same distance from vLLM with PLE as without it, 6.3 points of top-1 against 6.0, so PLE costs no accuracy relative to the rest of the model. That common offset is vLLM's bf16 activations, which cannot be removed: its QSA kernel refuses float32. Two bugs found along the way, both caught by the row-index check. The history was read and updated in the same pass, so a token early in a ubatch could pick up an earlier token of that same ubatch as prior context; it is now snapshotted first. And an EOS token was cutting its own context, where the reference takes the last EOS strictly before the position, so a boundary only hides tokens from the positions after it. Known gap: the conv carries no state across ubatches, so it is exact only for a prefill that starts at position 0. Chunked prefill and decode need the conv state wired into the recurrent memory, and the conv branch itself is still numerically unverified because the fixture zeroes its weights.
The PLE depthwise conv was zero-padding on the left, which is only right for a prefill that starts at position 0. Decode and chunked prefill saw a truncated history for the first (kernel-1)*ngram_size positions of every ubatch. The PLE module sits on a layer that is also a delta-net layer, so both need a conv history in the same recurrent row. Rather than plumb a per-layer state size through build_rs and build_conv_state, the row is widened once and each convolution addresses its own slice through a local helper. n_embd_r() gains the extra span, which is zero for every other architecture because it is derived from ple_n_heads. Verified by feeding the same 1024 token sequence in chunks instead of one shot: at 64 tokens per decode the logits are bit-identical to the single-shot run, 1023 of 1023 top-1 and a maximum logprob deviation of exactly zero. At one token per decode they differ slightly, but the no-PLE model differs more under the same test (94.6% against 97.1%), so that is the usual gemv-versus- gemm accumulation difference and not the state. The conv branch is also no longer unverified. With non-zero conv weights the port sits 6.3 points of top-1 below the numerical floor, the same distance as with the weights zeroed and as the model with no PLE at all, so the branch adds no error of its own. test-llama-archs passes every existing architecture at 0.00e+00, including the delta-net models that share this code path.
build_rs writes into the state tensor in place, zeroing one row and copying the carried-over states, so calling it twice for the same layer let the second call clobber the first write-back. The PLE layer is also a delta-net layer, so that is exactly what happened: both convolutions gathered the same row. They now share a single gather per layer. The earlier claim that the conv state was carried correctly was tested on a fixture whose conv weights are zero, where the branch contributes nothing and chunking matches trivially. Re-running with non-zero conv weights showed the divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one boundary down to 90.2% at seven. With the shared gather it is bit-identical to the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum logprob deviation of exactly zero over 1023 positions. The delta-net-only model stays bit-identical too, so nothing regressed there. Also derive the delta-net conv channel count the way load_arch_tensors sizes wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r() only bounds the row and the convolution has to match the tensor feeding it. test-llama-archs previously aborted on this architecture and took every later architecture with it. qwen4exp is marked MoE-only, given the hyper-connection keys and an ssm_d_inner consistent with its tensor derivation, and skipped for now: the hyper-connection keys written by get_gguf_ctx are not reaching the synthesised file, which needs a separate look. The suite completes again, 124 architectures at 0.00e+00.
Groundwork for qwen4exp's QSA sparse attention. Its indexer needs a per-token key history for the full-attention layers, but a hybrid model cannot use llama_kv_cache_dsa: that class derives from llama_memory_i rather than llama_kv_cache, and llama_memory_hybrid constructs its attention cache directly. No existing architecture pairs recurrent state with a sparse indexer, so there was nothing to reuse wholesale. llama_memory_hybrid therefore gains a third, optional cache, shaped the same way llama_kv_cache_dsa shapes its lightning-indexer cache: a copy of hparams with n_head_kv forced to 1 and n_embd_head_k_full set to indexer_head_size. It is built only when a filter_idx callback is passed, which defaults to nullptr, so every existing architecture gets exactly what it got before. The per-sequence operations and the batch preparation forward to it under a null check, matching how the DSA cache prepares its two caches over the same ubatches. test-llama-archs passes all 124 architectures at 0.00e+00, including the 12 in the hybrid family that share this code. The qwen4exp fixtures are unchanged: same logits against vLLM, and chunked evaluation still bit-identical to single-shot.
The full-attention layers of this model do not attend to everything. An
indexer scores one mean-pooled key per block of compress_ratio tokens and
keeps a budget of the best blocks, plus the tail of tokens that do not yet
form a complete block. Below indexer_top_k + compress_ratio - 1 cached
tokens every block fits in the budget, so the result is exactly dense.
What is reused rather than rebuilt:
- the mask machinery. build_attn's DSA overload already turns a list of
token indices into a KQ mask via ggml_set_rows, so that block is lifted
out verbatim into build_attn_mask_top_k and shared with a new overload
on llm_graph_input_attn_kv. DSA's node sequence is unchanged; the new
overload exists because llama_kv_cache_dsa assumes MLA and cannot be
dropped into a hybrid model.
- the indexer key cache, which is the optional third cache added to
llama_memory_hybrid in the previous commit. It holds raw keys, because
pooling happens before the norm and the rotation.
The graph expands block scores rather than block indices: giving every
token of a block its block's score needs only a gather, where expanding
indices would need an integer multiply-add that ggml has no op for. Since
the budget is a whole number of blocks and a block's members tie exactly,
the cut still lands on a block boundary.
Everything that depends on cache layout is computed host-side in
set_input_qsa. Blocks are cuts of the position line rather than of the cell
array, so nothing assumes the cache is contiguous.
Measured on the tiny fixture against vLLM, comparing the selected token
indices directly rather than the logits:
below the budget selection identical, and 1024-token logits are
bit-identical to the pre-QSA dense path
above the budget mean jaccard 0.975
The direct index comparison is what made this correct. The reference
rectifies each head's dot product before summing over heads, which an
earlier reading of it had missed; on logits alone the resulting port looked
fine, because on a randomly initialised fixture the known-correct dense
path already disagrees with vLLM by more than the bug did. Comparing the
indices showed 0.794, and fixing the ReLU moved it to 0.975.
The indexer cache found its own slots, independently of the attention cache. Both are the same size and see the same ubatches, so in a straight-through prefill they agree, which is why every fixture and every single-shot parity run passed. They drift once the context is being rewritten between turns, and then the QSA top-k indices, which are applied against the attention mask, point at the wrong cells. The seven-turn chat test caught it on the third turn: llama-server aborted on the assertion that the two caches report the same n_kv. The cache is a side buffer addressed by the attention cache's cells, so it now takes that cache's slot layout instead of computing one. Applying that layout also marks its cells identically, so the two agree cell for cell by construction rather than by coincidence, and the assertion can no longer fire. Inert where the caches already agreed: test-llama-archs green at 126 archs and 0.00e+00, and the 4096-token tiny fixture is unchanged at max logit delta 0.0.
The old note guessed that the hyper-connection keys never reach the file. They do: dumping the gguf_context handed to llama_model_init_from_user shows both among its 67 KVs, and the loader still reports one missing.
The arch was skipped with a note guessing that the hyper-connection keys
never reached the synthesised file. They did. The suite builds a model, then
saves and reloads it, and llama_model_saver did not re-emit those keys, so
the failure was in the roundtrip leg rather than the first load. Three gaps,
all in shared code and all additive:
- add_kv_from_model wrote no hyper-connection, compress-ratio or PLE keys.
The PLE group only means anything whole, so it is written or omitted
together; the rest follow the file's existing style of writing every key
unconditionally, since an architecture that does not read one is
unaffected by a zero.
- the saver had no uint64 path at all, which the PLE hash constants need.
- add_tensors_from_model enumerates model-level tensors by hand and was
missing per_layer_tok_embd and the three final-mixer tensors.
Two smaller fixes on the qwen4exp side, both found by running the test:
- build_qsa_top_k divided by the compression ratio before asserting it was
non-zero, so a file without the key crashed instead of reporting.
- a layer with no compression ratio now falls back to dense attention,
which is what the model computes below the budget anyway. The test then
has to write a ratio to reach QSA at all, and an indexer key length no
narrower than n_rot, since the indexer ropes with the main attention's
rotary width.
Full suite: 126 archs, qwen4exp at 0.00e+00 with roundtrip OK. The tiny
fixture is unchanged, max logit delta 0.0 against the pre-QSA dense run.
The n-gram table arrives as 128 shards that were held in a dict and then torch.cat-ed, so the peak was the shards plus the concatenation: around 300 GB of RSS on the real checkpoint, which rules out machines that could otherwise convert this model. Each shard is now written straight into a memory-mapped file at its final row offset and dropped, so the resident set is one shard and the rest is the page cache's problem. The temporary file sits beside the output and is removed once the write finishes, including on failure. Shards other than the last must be uniform for direct placement, which is asserted rather than assumed, and a shard arriving before the stride is known is held instead of misplaced. Verified on the tiny fixture: the resulting GGUF is byte-identical to the one the concatenating path produced (md5 2d274efac91ad1e9a6007efb0687e597).
tensor_type_fallback demotes a tensor whose ncols is not a multiple of the target's block size, but its switch only enumerates the 256-block types. A target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls into default: and throws, even though the function already knows how to answer that case: the ncols check right below the switch resolves an unrepresentable shape to F16. Route those types into that check instead of throwing. Only paths that abort today change, so no quantization that currently succeeds is affected. Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but "failed to quantize model from ...", with no tensor name and no exception text, which made a quant recipe that had simply not pinned the tensor look like a corrupt model. It now names the tensor and continues.
per_layer_token_embd shares the TOKEN_EMBD category with token_embd.weight, so --token-embedding-type is returned for it before any --tensor-type pattern is consulted, and there is no way to give it a tier of its own. That grouping is fine as a default and stays the default. It is a poor fit for the size, though: on qwen4exp the table is 97.7 GiB of a 337.6 GiB BF16 file and about 46% of a 4-bit one, roughly eighty times token_embd.weight, and it is read by ggml_get_rows rather than a matmul so no imatrix ever covers it. Allow an explicit --tensor-type pattern to name it, and only it. Nothing changes unless such a pattern is passed, and token_embd.weight keeps the old precedence in either case. Measured on Qwen3.8-Flash-Next, Q4_K_M with an imatrix: the table lands at q8_0 (51.9 GiB, 113.5 GiB total) by following --token-embedding-type, and pinning it q4_1 gives 30.5 GiB for 92.1 GiB total, 19% off the file.
The per-tensor output buffer was sized `nelements * 4`, described as an upper bound. It is a very loose one: the output is at most 2 bytes per element (f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of it is never touched. The exact size is already known here, since it is what the quantization loop writes, what new_size sums to, and what the GGUF metadata is asserted against a few lines later. On a model whose largest tensor is a few GB none of this matters. On Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1. Measured on that model, VmHWM of a live llama-quantize was 485 GB per process. Three of them fit in 2 TB and five did not, which is what an OOM-killed quant ladder looks like. This removes about 150 GB of that. Byte-identical output, verified against the same binary built at the parent commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and without a PLE table present. Six cases, six matching md5s.
The PLE row indices are computed host-side from ubatch->token, and set_input
returned early when that was null. A multimodal ubatch is exactly that case:
the mtmd layer consumes the image placeholder ids and hands llama_decode
embeddings instead. The early return left the I32 index tensor uninitialised,
so ggml_get_rows indexed a 320 M row table with whatever the buffer happened to
contain, and aborted:
GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml_compute_forward_get_rows
mtmd_helper_decode_image_chunk -> llama_decode
Every image request crashed. Nothing caught it because the vision work had only
ever been verified by converting an mmproj, never by running one.
The reference computes the hash over input_ids, where those positions still
hold the image placeholder, so carry that id through as qwen4exp.ple.image_token_id
and hash it. The key is optional: a file converted before it existed falls back
to the PLE EOS token, which is defined and treats the image as a segment
boundary rather than crashing.
Verified end to end with llama-mtmd-cli, a Q4_K_M base and the F16 mmproj, on a
generated image with known content. The model names the red circle, the blue
square, the inverted green triangle and reads "UNSLOTH 42", each with the right
position.
set_input_qsa asserted n_stream == 1, so llama-server could not serve this model with more than one slot unless -kvu was passed. With a non-unified cache each sequence owns its own cells, and a cell index means a different token in each stream, so a single shared mapping is wrong. - cell_blk, blk_cells and bias gain a stream dimension. At n_stream == 1 these collapse to the shapes they had, so the unified path is unchanged. - Scoring is now batched over streams. ggml_mul_mat matches ne[2] on both operands, so stream s's queries only ever meet stream s's blocks; without this sequences would score against each other's context. - set_input_qsa loops per stream and resolves cells through v_cells[seq_to_stream[seq_id]], following set_input_kq_mask_impl, instead of hardcoding v_cells[0]. - llama_kv_cache_context::get_n_stream() is added, mirroring the ns that get_k and get_v already derive from the slot info. build_attn_mask_top_k needed no change: it already expects [n_top_k, n_batch, 1, n_stream], so the top-k result is reshaped to meet it. set_input_qsa has exactly one caller, so the blast radius is qwen4exp only. Validation, UD-Q4_K_XL on one B200: - unified cache unchanged within noise: 1802.9/68.85 -> 1807.2/69.11 t/s at batch 1, 2262.5/192.43 -> 2270.1/193.75 at batch 4. - non-unified now runs at npl 1, 4, 16 where it previously aborted, and is 22% faster than the -kvu workaround at batch 16 (1205 vs 984 t/s total), since per-stream cells avoid the cross-sequence masking a unified cache pays for. - no cross-stream contamination: four concurrent sequences each carrying a distinct secret all recall their own and no other, on both cache modes. - test-llama-archs green on qwen4exp, deepseek2, gemma3n, qwen3next, llama. Note on testing: comparing concurrent output against solo output exactly is not a valid check. It failed 0/4 with no bug present, and the unified-cache control failed the same way, because batch composition changes the floating-point reduction order and near-tied tokens flip. The contamination test above is what the exit code gates on.
github-actions
Bot
added
model
This comment was marked as resolved.
This comment was marked as resolved.
Closed
ggerganov
marked this pull request as ready for review
August 26, 2026 14:01
ggerganov
requested review from
CISC,
JohannesGaessler and
ggerganov
as code owners
August 26, 2026 14:01
Contributor
Author
|
I'm adding MTP - still WIP - everything else works well (including vision) I copied Gemma-3N's PLE approach so it's 1 large table - it can be offloaded to RAM or Disk via mmap |
Collaborator
|
Previous work on ngram embeddings: #19167 |
This was referenced Aug 31, 2026
3 tasks done
This was referenced Sep 3, 2026
|
Related work, in case it is useful to people following the qwen4exp / PLE table support: we built a small tool that edits the PLE (Engram) table post-hoc to add a fact without touching the weights. The trigger's last three tokens address 16 rows; we optimize the 8 trigram-keyed rows by gradient through a CPU replica of the frozen model (routing refreshed every step) and ship them as an overlay that a fork of llama.cpp substitutes at gather time ( How this was built: the code, the experiments and the write-up were produced in a Claude Code session (Anthropic's Claude, model Fable 5.1) driven by a single human operator who set the goals, approved every design step, ran the hardware and read every result. We are not after stars: we want the method checked, broken and improved. |
1 task
4 of 7 tasks
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Off-topic comments and AI slop/spam in this thread will be deleted. Keep the discussion focused.
Draft. Adds support for
Qwen3.8-Flash-Next(HFmodel_type: qwen4_exp,Qwen4ExpForConditionalGeneration): converter, text graph, sparse attention, vision, and three quantizer fixes the model needs. Opening as a draft because the weights are not public yet, so the accuracy numbers below cannot be reproduced from outside at the moment.Architecture
build_moe_ffn, unchangedggml_rope_multiwithLLAMA_ROPE_TYPE_IMROPE, unchangedhc_count = 4, low rank)qwen4exp.cpp;deepseek4.cppis untouchedset_input, thenggml_get_rowsllama_memory_hybridgit diff master --stat -- ggml/is empty: no new ggml op, and no change to any existing one.Correctness, against the reference implementation
test-llama-archs -a qwen4expTwo caveats on that table, since neither is obvious from the numbers. The bit-identical row is measured on BF16 and F32; a quantized model is not bit identical across the boundary, and UD-IQ1_S shows a max logit delta of 2.84e-03. And
test-llama-archsis a weaker check than it looks for this architecture: its synthesised model carries no PLE tensors, so the PLEset_inputnever runs, and it is insensitive to the GDN fused-QKV segmentation convention (three different segmentations, one correct and two deliberately wrong, all returned 8.44e-08). It is a plumbing and roundtrip check, not a correctness oracle.The bit-identical result below the budget is the useful one: QSA is dense by construction under
indexer_top_k + compress_ratio - 1cached tokens, so it is a free oracle on the whole selection and masking path.Effect on other architectures
Six shared files contain deletions rather than pure additions:
llama-quant.cpp: three fixes, each byte-exact against the parent commit.tensor_type_fallbackhad no case for 32-block types, so a tensor whose ncols is not divisible by 32 aborted instead of falling back;--tensor-typecould not nameper_layer_token_embdbecause--token-embedding-typereturned first; and theworkbuffer was sizednelements * 4as an upper bound on a value already known exactly, which is 150 GB per process on a model with a 51.2 G element tensor. Six A/B cases, six matching md5s.llama-graph.cpp:build_attn_mask_top_klifted verbatim out of the DSAbuild_attnoverload so both share it. The node construction sequence is identical after splicing the helper back into its call site, anddeepseek32andglm-dsaboth pass at 0.00e+00 on CPU.llama-memory-hybrid.cpp: the newfilter_idxdefaults tonullptrand every use is guarded.llama-model-loader.cpp: adds aGGUF_TYPE_UINT64case to a type check that previously threw.llama-hparams.cpp,llama-model.cpp: new fields and new switch cases.A shared hyper-connection base with
deepseek4was considered and rejected: the two formulations agree on the residual layout and nothing else (full-rank projection with Sinkhorn normalisation there, low-rank silu gate and a mean collapse here), so factoring it out would touch DSV4's hot path to share about ten lines.Note for reviewers
This branch was developed against a snapshot and rebased onto current master. Two rebase conflicts were resolved by hand, both additive:
test-llama-archs.cppwhere master had added aLLM_ARCH_DEEPSEEK4condition to a line this port also edits, andllama-model-saver.cppwhere master reworked the DSV4 key block. Worth a close look, since the local build was against the pre-rebase tree.AI Usage
Used Claude and Local Models for testing, iteration and code design - manual verification of model / PR usage