[bong-water-water-bong]

Adds docs/zyphra-models.md — the Zyphra family (Zamba2, Zamba v1, BlackMamba) — and a pointer to it from docs/models.md. Refs #1036
(closed, so nothing is being re-opened by this).

It exists because two changes made a Zamba2 GGUF the size of its checkpoint, and neither is visible from a single model file: the shared
transformer block is stored once per shared block and aliased per layer (#203), and each layer's rank-128 MLP adapter is stored as a
low-rank pair and applied in the graph as base + B*(A*x) instead of being baked into a full FFN matrix (#218). What the page records:

• Sizes, measured: 7B 16,875,503,904 → 14,722,123,360 B (−2.005 GiB), 771 → 786 tensors; 2.7B 6,009,822,848 →
  5,328,903,360 B (−0.63 GiB), 520 → 531. Both forms load from one binary, so a GGUF written before the change keeps working.
• Why the forms are not bit-identical (KLD 0.000067 at 2.7B, 0.000095 at 7B; f32-merged-at-conversion vs pair-at-run-time) and therefore
  why a quantized arm must be re-measured, not inherited — which is exactly what forced the pick decision below.
• The picks under the family bar (mean KLD ≤ 0.03 AND same-top-1 ≥ 95 %), including the decision this page exists to make durable:
  Zamba2-7B-Instruct moves Q5_K_M → Q6_K (5829 MiB, KLD 0.010430, top-1 96.593 %), with Q5_K_M kept visible as the smaller file
  that no longer clears the bar (4938 MiB, 0.019383, 94.792 %, −0.21 points). The reason is stated: the adapter form moved this
  model's top-1 down 0.52 points against 0.025 of headroom, while it moved the 2.7B's up 0.05 — the same change, opposite reaction. The
  32-chunk rows are marked, because those were decided on the longer run.
• The stale-quantizer trap, which is the one that costs a silent 21 %: a llama-quantize older than the F16 default quantizes the
  rank-128 pair, and since the block formats need input_size ≥ 256 while the float path only needs % 4, the pair's B matmul leaves
  HRX on every hybrid layer — 9 of 3051 nodes, pp512 463.66 vs 589.47 tok/s, no error. The flag is
  --tensor-type ffn_gate_up_adapter=f16 (+31.8 MiB).
• Layout asserts worth making before trusting any PPL (ffn_up ×2, zamba2.ffn_gate_up_adapter_rank = 128, 9+9 / 13+13 adapters), and
  the current -ngl 99 placement caveat (engine hrx: token-embedding GET_ROWS leaves the device at -ngl 99 after the host-pointer weight import (#226/#228), 1 node + 1 split per graph #1316: the tok-embd GET_ROWS is off HRX at the current base, base-level, both forms).

On the two items this was asked to cover

1. "Update the registry/recipe pick." There is no engine-side pick to update: I checked config/recipes.json (llama-server settings per
   model/route, no Zamba2 entry), models/pins.json (pins published repos' files; no Zamba2-7B/2.7B entry), and
   registry/{architectures,census,check_models.tsv,checked,significant,support_pins}.json, and grepped the tree for pick/sweet_spot/
   default_quant/recommended — nothing names a per-model quantization. The pick has only ever lived in PUBLISH-READY.md, which
   already carries the decision (Zamba2-7B-Instruct | Q6_K | sweet spot + smallest usable | 5829 MiB | 0.010430 | 96.593 % with
   ~~Q5_K_M~~ withdrawn … 94.792 %), so I did not edit another lane's working doc to restate it. This page is where the engine now
   records it, which is the piece that was missing.
2. The family page: docs/model-families/ does not exist in the engine (docs/ is flat), so the page is docs/zyphra-models.md to
   match the repo, linked from docs/models.md. The archived 1bit-MONSTER monorepo was not touched.

tools/copyright.py --check → clean (exit 0). Docs-only; no code, no pin, no behaviour.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1036 - Partially compliant Compliant requirements: The PR introduces documentation that explains the fix for the shared attention block duplication issue It documents the size reduction achieved by the fix (from 20.45 GiB to ~15 GB) It explains the two changes that fixed the issue: shared block stored once per block and per-layer MLP adapter stored as low-rank pair It documents the file sizes and tensor counts for both forms (merged and adapter) It explains why the forms are not bit-identical and why quantized arms must be re-measured Non-compliant requirements: The PR does not contain the actual code changes to fix the shared attention block duplication issue in conversion/zamba2.py The PR does not include the implementation of the fix for the per-layer MLP adapter storage Requires further human verification: The actual implementation of the fix in conversion/zamba2.py needs to be verified by a developer The size reduction from 20.45 GiB to ~15 GB needs to be confirmed through testing The PPL and KLD values need to be validated against the original implementation 1316 - Partially compliant Compliant requirements: The PR documents the HRX placement issue and its impact on token-embedding GET_ROWS It mentions that the issue affects both forms equally and is tracked as engine hrx: token-embedding GET_ROWS leaves the device at -ngl 99 after the host-pointer weight import (#226/#228), 1 node + 1 split per graph #1316 It notes that the issue is related to the host-pointer weight import and GET_ROWS matcher Non-compliant requirements: The PR does not contain the actual code changes to fix the HRX placement regression Requires further human verification: The actual fix for the HRX placement issue needs to be implemented and verified The fix for the GET_ROWS matcher needs to be implemented and tested

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Documentation Gap The PR only adds documentation about the fixes for issues #1036 and #1316 but does not include the actual implementation of those fixes in the codebase. The documentation explains the problems and solutions but lacks the concrete code changes needed to resolve the issues. <!-- Copyright 2026 bong-water-water-bong SPDX-License-Identifier: Apache-2.0 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # The Zyphra family: Zamba2, Zamba v1 and BlackMamba The three Zyphra architectures the engine carries — `zamba2`, `zamba` (v1, with its MoE) and `blackmamba` (Mamba-1 with alternating MoE) — what their files weigh at the current pin, which quantization to publish for each, and the two things that will silently go wrong if you convert or quantize them with an older tree. The architecture work is in [models.md](models.md) rows and in the fork; this page is the file-level record. Engine issue: **#1036** (the Zamba2 sharing work) and **#991** (HRX placement classes). ## File sizes: a Zamba2 GGUF is now the size of its checkpoint Zamba2 checkpoints store the transformer block **once per shared block** (two of them at 2.7B and 7B) and hold each hybrid layer's MLP difference in a rank-128 adapter pair. Converting used to expand both — the block once per hybrid layer, and each layer's adapter merged into a full FFN matrix — so a 7B BF16 GGUF came out at **20.45 GiB** against a 14.82 GB checkpoint. Two changes fixed that, and both are in the pin: | change | fork PR | what it stops | |---|---|---| | shared block stored once per block, aliased per layer | [#203](https://github.com/1bit-MONSTER/llama.cpp/pull/203) | 13 copies of a set that has 2 distinct values | | per-layer MLP adapter stored as a low-rank pair, applied in the graph as `base + B*(A*x)` | [#218](https://github.com/1bit-MONSTER/llama.cpp/pull/218) | 13 copies of a 102.76 M-param matrix (205 MB each at bf16) where 2 plus a 4.13 M pair per layer do | | model | merged form (old) | adapter form (current) | change | |---|---|---|---| | Zamba2-2.7B-instruct | 6,009,822,848 B (5.60 GiB), 520 tensors | **5,328,903,360 B (4.96 GiB)**, 531 tensors | **−680,919,488 B (−0.63 GiB)** | | Zamba2-7B-Instruct | 16,875,503,904 B (15.72 GiB), 771 tensors | **14,722,123,360 B (13.71 GiB)**, 786 tensors | **−2,153,380,544 B (−2.005 GiB)** | **Both forms load from one binary.** A GGUF written before these changes keeps taking the merged path — the loader requires `ffn_up` per layer when the file carries no `zamba2.ffn_gate_up_adapter_rank` key, and reads the low-rank pair when it does. Nothing published before the change stops working. The two forms are **not bit-identical**, and that is expected: the merged file has `up @ down` computed in f32 at conversion time and rounded to bf16, while the adapter path evaluates the pair at run time. Measured against the merged form's own logits (CPU, 16 × 512): | model | mean KLD | same top-p | PPL | |---|---|---|---| | Zamba2-2.7B-instruct | **0.000067** | 99.510 % | 16.795417 (merged 16.7937) | | Zamba2-7B-Instruct | **0.000095** | 99.755 % | 8.499977 (merged 8.5028) | Both are two to three orders of magnitude below the family bar below, so a quantized arm still has to be **re-measured** rather than inherited from the merged form — see the next section, where one arm changed sides. ## What to publish per model The family bar is **mean KLD ≤ 0.03 AND same-top-1 ≥ 95 %**, and each model gets a sweet spot, a max-quality and a smallest-usable pick. The Zamba2 rows are the adapter form; the others are unchanged. | model | arm | role | size | KLD | top-1 | |---|---|---|---|---|---| | Zamba2-2.7B-instruct | **Q6_K** | sweet spot | 2119 MiB | 0.007244 † | 95.711 % † | | | Q8_0 | max quality | 2728 MiB | 0.001242 | 98.235 % | | | Q4_K_M-imx | smallest usable | 1511 MiB | 0.043205 | 90.858 % | | Zamba2-7B-Instruct | **Q6_K** | sweet spot **and** smallest usable | **5829 MiB** | 0.010430 | 96.593 % | | | Q8_0 | max quality | 7515 MiB | 0.001669 | 98.603 % | | | ~~Q5_K_M~~ | **withdrawn** | ~~4938 MiB~~ | 0.019383 | **94.792 %** — misses the bar by 0.21 points | | Zamba-7B-v1 | **Q6_K** | sweet spot and smallest usable | 6943 MiB | 0.002685 | 97.16 % | | | Q8_0 | max quality | 7484 MiB | 0.000962 | 97.97 % | | BlackMamba-2.8B | **Q8_0** | sweet spot and max quality | 2831 MiB | 0.001312 | 97.99 % | | | IQ4_XS-imx | smallest usable | 1542 MiB | 0.026519 | 91.01 % | † 32-chunk run (32 × 512 = 16384 tokens), because the 16-chunk run sat too close to the bar to decide; every other row is a 16-chunk run. Mark the long rows when quoting them. **Why the 7B moves up a tier.** The adapter form perturbs top-1 in proportion to how close an arm already sat to the bar, and the two Zamba2 models reacted differently to the same change: the 2.7B's top-1 moved **up** 0.05 points, the 7B's **down** 0.52. `Q5_K_M` was clearing the 95 % bar by 0.025 points on the merged form, so it no longer clears it (94.792 %), and `Q6_K` — 5829 MiB, still 11-12 % smaller than the merged `Q5_K_M` — is the smallest arm that does. `Q5_K_M` is listed above as withdrawn rather than deleted because it is the smaller file for anyone who accepts the lower top-1 deliberately. ## Two things that go wrong with an older tree **1. A quantizer that predates the F16 default will drop the adapter off HRX.** The adapter pair stays at F16 by default, because its `B` contraction has `input_size = rank = 128`: the float weight path admits that (`input_size ≥ 4 && % 4 == 0`), the block formats do not (their block layout needs `≥ 256`). Quantize the pair with an older `llama-quantize` and that matmul leaves the device on every hybrid layer — measured on a 2.7B `Q5_K_M` as **9 of 3051 nodes on the CPU and pp512 463.66 vs 589.47 tok/s (0.79x, -21 %)**, with no error to tell you. If your tree predates the default, pass it explicitly: ``` llama-quantize --tensor-type ffn_gate_up_adapter=f16 <in.gguf> <out.gguf> Q6_K ``` Cost of the F16 pair: **+31.8 MiB** on an 1804.93 MiB 2.7B file. An explicit `--tensor-type` still wins; `--pure` still means pure. **2. Assert the layout before you trust a PPL.** A pair written transposed (`{rank, n_embd}` instead of `{n_embd, rank}`) is refused by the loader with `check_tensor_dims`, but only after a conversion; the cheap check is on the container: `ffn_up` **×2** (not one per hybrid layer), `zamba2.ffn_gate_up_adapter_rank = 128`, adapters **9 + 9** at 2.7B and **13 + 13** at 7B, and the byte sizes in the table above. ## HRX placement, if you are quoting a number The adapter path adds two `MUL_MAT`s and an `ADD` per hybrid layer and **all of them are placed on HRX** — `+27` nodes at 2.7B (9 × 3), with the same off-device residue as the merged form, and `GGML_HRX_STRICT=1` exits 0 on both forms at prefill and decode. There is no throughput cost to report: pp512 441.98 → 440.57 and tg64 22.87 → 22.60 on one binary with both forms (0.997 / 0.988). One caveat for anyone running `-ngl 99` today: the token-embedding `GET_ROWS` is currently **not** on HRX (`1 of 3024` nodes, `GET_ROWS x 1`, 2 graph splits) after the host-pointer weight import landed — that is base-level and affects **both** forms equally, tracked as engine **#1316**, not in this family's changes

[bong-water-water-bong]

VERDICT: READY at 677af4e21db9726d7665b9c376cc332fc8740d28. Docs only, and every figure I could hold it to checks out.

• Sizes: 16,875,503,904 → 14,722,123,360 B = −2.0055 GiB (7B), and 6,009,822,848 → 5,328,903,360 B = −0.6342 GiB (2.7B). On disk, ~/models/zyphra-orig/ref/Zamba2-2.7B-instruct-bf16.gguf is exactly 6,009,822,848 B and Zamba2-7B-Instruct-bf16-adapter.gguf exactly 14,722,123,360 B, so two of the four numbers are files, not transcriptions.
• The picks match the record they cite: models/zyphra-orig/quant/PUBLISH-READY.md:45 has Q6_K, 5829 MiB, KLD 0.010430, top-1 96.593 %, and :47 has Q5_K_M withdrawn at 4938 MiB, 0.019383, 94.792 %, "misses top-1 by 0.21 points" (95 − 94.792 = 0.208 ✔). The decision not to restate it inside another lane's working doc is right.
• The stale-quantizer cost: pp512 463.66 / 589.47 = 0.787, i.e. the "silent 21 %" ✔. --tensor-type ffn_gate_up_adapter=f16 as the remedy follows from the block formats' input_size ≥ 256 versus the rank-128 pair.
• Every relative link in docs/zyphra-models.md resolves. The docs/models.md pointer is one paragraph. copyright.py --check rc 0.
• Refs #1036 on a closed issue, with nothing re-opened, is the right form.

Note: the #1316 caveat (tok-embd GET_ROWS off HRX at -ngl 99) is accurate for main today. Engine #1344 shows that's by design (LLAMA_INPUT_LAYER_ON_DEV), so when #1344 lands with its rebased pin, this line can say "by design, opt-in to move it" rather than reading as a defect.