convert.py: Fix loading safetensors and ggml format on Windows by comex · Pull Request #991 · ggml-org/llama.cpp

comex · 2023-04-15T02:01:42Z

Calling mmap.mmap on Windows apparently resets the file offset of the raw file object (and makes the BufferedReader return a negative file offset). For safetensors, avoid using the file offset after calling mmap. For GGML format, explicitly save and restore the offset.

Fixes #966.

Calling `mmap.mmap` on Windows apparently resets the file offset of the raw file object (and makes the BufferedReader return a *negative* file offset). For safetensors, avoid using the file offset after calling mmap. For GGML format, explicitly save and restore the offset. Fixes ggml-org#966.

…ama.cpp (ggml-org#991) See ggml-org#990. This change makes the logits_to_logprobs function equivalent to the version in the llama.cpp repository. It uses numpy so it's much faster than the previous version.

…org#991) Calling `mmap.mmap` on Windows apparently resets the file offset of the raw file object (and makes the BufferedReader return a *negative* file offset). For safetensors, avoid using the file offset after calling mmap. For GGML format, explicitly save and restore the offset. Fixes ggml-org#966.

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…org#991) Calling `mmap.mmap` on Windows apparently resets the file offset of the raw file object (and makes the BufferedReader return a *negative* file offset). For safetensors, avoid using the file offset after calling mmap. For GGML format, explicitly save and restore the offset. Fixes ggml-org#966.

…gine ggml-org#991) (#197) An HRX build prints "device HRX0" and then runs part of the graph on the CPU without saying so. The engine admits a dense MUL_MAT only when input_size % 256 == 0 (eight gate sites in dispatch-registration/common/ dispatch-mul-mat.cpp, stated by common_is_supported_dense_input_size()), so a model whose hidden size is not a multiple of 256 loses its matmuls silently while the SSM/scan/norm side stays on HRX: measured with GGML_HRX_STRICT=1, Zamba-7B-v1 (n_embd 3712) puts 383 of 5113 nodes on the CPU and BlackMamba-2.8B (1472) 145 of 1179, while Zamba2-2.7B (2560) is fully claimed. The placement observer already counted the off nodes for GGML_HRX_REPORT and GGML_HRX_STRICT; it now also counts the off matmuls whose input_size is not a multiple of 256 and, in the default configuration, prints one WARN per process on the first graph that loses a node: placement_observer: placement: 383 of 5113 nodes on the CPU (MUL_MAT x 281, CONT x 89, UNARY x 13) - 3712 is not a multiple of 256, so 281 matmuls have no HRX dense schedule (input_size % 256 != 0) GGML_HRX_REPORT=0 silences it and a value that is not a number counts as unset, so a typo cannot hide the line (an explicitly numeric <=0 does silence it, as before); GGML_HRX_REPORT=1 keeps the pre-existing per-graph report instead of printing both; GGML_HRX_STRICT=1 is unchanged. WARN rather than INFO because a placement fallback is a degraded state of the same class as the residency guardrail, and it should survive a front-end that keeps warnings and drops info. Note llama-bench without -v installs a null log callback through llama_log_set -> ggml_log_set, so no ggml line of any severity reaches the terminal there; use -v, or llama-server/llama-cli, which print it as usual. No behaviour change: placement, kernels and the split are untouched. The scan is bounded to the first 16 graph evaluations, which is where the prefill graph and the first decode steps are built. Scanning every graph for the rest of the run would put a per-node backend lookup plus a vector allocation in the default hot path for a log line, and for a fully claimed model the scan would never terminate itself. GGML_HRX_STRICT=1 (aborts on the first off node) and GGML_HRX_REPORT>=2 (every graph) remain the exhaustive probes. Measured on gfx1151 under gpu-hold (llama-server, default verbosity, HRX0, -ngl 99): Zamba-7B-v1 prints the line above once per process; with GGML_HRX_REPORT=0 it prints nothing; with an unparseable GGML_HRX_REPORT it still prints; Zamba2-2.7B prints nothing; the greedy continuation of "The capital of France is" is unchanged from the pre-change reference in every arm. GGML_HRX_STRICT=1 still aborts from inside the observer at the same node_14 MUL_MAT f32 ne=[14848,512,1,1] with the same 383/5113 counts. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>

…256 (engine ggml-org#991) The dense MUL_MAT admission required input_size % 256 == 0, and F16/BF16/F32 weights fall into the block-quantised arm of that predicate, so a model whose hidden size is not a multiple of 256 lost every dense matmul to the CPU while the rest of it stayed on the device (Zamba-7B-v1 n_embd 3712: 383 of 5113 nodes; BlackMamba-2.8B 1472: 145 of 1179). The refusal was policy, not the kernel's contract: the wmma kernel declares multiple_of(input_size, 32) and its core already rounds the contraction up to whole 256-blocks and masks both operands past input_size, so a K tail contributes zeros. Only element-wise formats can be unaligned at all - block formats are laid out in 256-value blocks, and a quantised tensor with ne[0] % 256 != 0 cannot exist in ggml - so the relaxation is one predicate about the element-wise formats. The relaxation is an explicit flag on the shared matcher, passed only by the three callers that dispatch the plain %32 kernel (plain, fused-unary and decode). The fused motifs that genuinely need 256 - mul_mat_postops, mul_mat_id_f16_f16, llm.attention_qkv, q6_K packed/shortlist, symmetric i8 - keep the old predicate: had the shared one been relaxed, a fused match would have compiled K=3712 against a kernel whose declared constraint forbids it. With the flag a fused match simply does not engage and falls through to the plain matmul, so the node stays on the device and only the fusion is lost. gfx1151: Zamba-7B-v1 383 -> 254 of 5113 nodes off (MUL_MAT 281 -> 165, UNARY 13 -> 0, CONT 89 unchanged), splits 384 -> 179; Zamba2-2.7B (2560) unchanged at 0 off; test-backend-ops -o MUL_MAT 404/404 and -o MUL_MAT_ID 852/852 on HRX0 vs CPU; greedy output byte-identical to the pre-change run. Not closed: the remaining nodes are the K % 32 != 0 class (ne=[264,512,2,1] etc.) and the %256 fused/MoE-id motifs.

…256 (engine ggml-org#991) (#199) The dense MUL_MAT admission required input_size % 256 == 0, and F16/BF16/F32 weights fall into the block-quantised arm of that predicate, so a model whose hidden size is not a multiple of 256 lost every dense matmul to the CPU while the rest of it stayed on the device (Zamba-7B-v1 n_embd 3712: 383 of 5113 nodes; BlackMamba-2.8B 1472: 145 of 1179). The refusal was policy, not the kernel's contract: the wmma kernel declares multiple_of(input_size, 32) and its core already rounds the contraction up to whole 256-blocks and masks both operands past input_size, so a K tail contributes zeros. Only element-wise formats can be unaligned at all - block formats are laid out in 256-value blocks, and a quantised tensor with ne[0] % 256 != 0 cannot exist in ggml - so the relaxation is one predicate about the element-wise formats. The relaxation is an explicit flag on the shared matcher, passed only by the three callers that dispatch the plain %32 kernel (plain, fused-unary and decode). The fused motifs that genuinely need 256 - mul_mat_postops, mul_mat_id_f16_f16, llm.attention_qkv, q6_K packed/shortlist, symmetric i8 - keep the old predicate: had the shared one been relaxed, a fused match would have compiled K=3712 against a kernel whose declared constraint forbids it. With the flag a fused match simply does not engage and falls through to the plain matmul, so the node stays on the device and only the fusion is lost. gfx1151: Zamba-7B-v1 383 -> 254 of 5113 nodes off (MUL_MAT 281 -> 165, UNARY 13 -> 0, CONT 89 unchanged), splits 384 -> 179; Zamba2-2.7B (2560) unchanged at 0 off; test-backend-ops -o MUL_MAT 404/404 and -o MUL_MAT_ID 852/852 on HRX0 vs CPU; greedy output byte-identical to the pre-change run. Not closed: the remaining nodes are the K % 32 != 0 class (ne=[264,512,2,1] etc.) and the %256 fused/MoE-id motifs. Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com>

…AT_ID (engine ggml-org#991) (#204) BlackMamba-2.8B loses 90 of 1179 nodes to the CPU and 72 of them are one MoE block pattern: MUL_MAT_ID x 54, GLU x 18. Its experts are [1472, 3872, 8] and [3872, 1472, 8], so the contractions are 1472 and 3872 -- both multiples of 32, neither a multiple of 256. common_mul_mat_id_supported_dense_input_size() forwarded to the dense table, whose default branch is % 256, and F16/BF16/F32 are not among its listed cases -- so element-wise expert weights inherited the block-quant default and were refused. The refusal is policy, not hardware: the three matchers that consult this predicate bind five kernels and all five declare or assume multiple_of(32), and the wmma core already pads the contraction (padded_input_size = input_size + 255) with a valid_k gate and masked activation loads. Route the predicate through common_mul_mat_supported_plain_dense_input_size(), the same element-wise/block split the plain dense path took in #199, so only F16/BF16/F32 verdicts change. Adds three test-backend-ops cases at the real shapes next to the existing gpt-oss k=2880 ones. Co-authored-by: agent <agent@local>

…ml-org#991) (#207) A MUL_MAT whose activation carries batch planes against an unbatched weight is the same 2-D contraction repeated per plane, and it is how the SSM/Mamba ports write their small projections. Every format-agnostic matcher refused it in common_match_mul_mat_any_format on common_is_2d(*input), before any size rule, so those nodes fell back to the CPU and split the graph. This is not a kernel gap: the two kernels involved (ggml_mul_mat_f32_f32_wmma for prefill, ..._decode_wave64 for token 1) already compute a 2-D tile from a buffer base, so the planes are bound one dispatch at a time. dispatch-gated-delta-net.cpp already binds { id, t * nb[2], byte_count - t * nb[2] } per token, which is the same mechanism. - common_match_mul_mat_any_format takes allow_batched: the activation and output may carry planes when the output's planes match the activation's. ggml pairs the operands' planes when both carry them and broadcasts when only the activation does, so the weight may be one matrix shared by every plane or one matrix per plane (with the same (dim2, dim3) factorisation, since pairing follows that index order). plane_count and batched_weight ride on the match. - common_plane_offset maps a plane index to its byte offset in a contiguous value. - build_batched_mul_mat_dispatch emits one dispatch per plane: activation, weight and output bindings offset to the plane, the token count is the per-plane one. - Two registrations, common.mul_mat.f32_f32_wmma_batched (82) and ..._decode_batched (62), one above each plain dense registration they mirror. The K contract is unchanged: still the plain element-wise predicate, so K % 32 != 0 stays on the CPU until the padded tile lands. Test cases are the shapes the port actually emits (m=264,n=8,k=3712 and m=464,n=8,k=256 with a shared weight; m=264,n=8/1,k=3712 with a BF16 matrix per plane; plus the unaligned k=232 rows that must stay refused). Co-authored-by: agent <agent@local>

…(engine ggml-org#991) The element-wise weight readers take four values at a time and the activation load is masked with `vector.mask.range [input_k to input_size)`, so a contraction only has to be a multiple of 4 for every packet to stay inside its weight row. The 32-value packet (and the 256 floor) the kernels declared was a contract nobody's arithmetic needed, and it is what leaves the Zyphra ports' dt_proj on the CPU: BlackMamba ssm_dt.weight is [92, 2944] and Zamba v1's is [232, 3712]. - common_mul_mat_supported_plain_dense_input_size: element-wise formats now need input_size >= 4 && % 4 == 0. Block formats are untouched and keep their %256/%128/%64/%32 from common_mul_mat_supported_dense_input_size(). - mul_mat_f32_f32_wmma's ggml.mul_mat.input_size config and the assume in mul_mat_f32_f32_wmma_core and both entry points of mul_mat_f32_f32_decode: range 256 -> 4, multiple_of 32 -> 4. Weakening the contract is safe for every caller: the dispatcher still gates each format at its own granularity, and a block-format contraction is always %256, hence also %4. K % 4 != 0 stays refused on purpose: the last packet of such a row would read up to three values past the row, which is correct by accident only where the following row exists.

…(engine ggml-org#991) (#208) The element-wise weight readers take four values at a time and the activation load is masked with `vector.mask.range [input_k to input_size)`, so a contraction only has to be a multiple of 4 for every packet to stay inside its weight row. The 32-value packet (and the 256 floor) the kernels declared was a contract nobody's arithmetic needed, and it is what leaves the Zyphra ports' dt_proj on the CPU: BlackMamba ssm_dt.weight is [92, 2944] and Zamba v1's is [232, 3712]. - common_mul_mat_supported_plain_dense_input_size: element-wise formats now need input_size >= 4 && % 4 == 0. Block formats are untouched and keep their %256/%128/%64/%32 from common_mul_mat_supported_dense_input_size(). - mul_mat_f32_f32_wmma's ggml.mul_mat.input_size config and the assume in mul_mat_f32_f32_wmma_core and both entry points of mul_mat_f32_f32_decode: range 256 -> 4, multiple_of 32 -> 4. Weakening the contract is safe for every caller: the dispatcher still gates each format at its own granularity, and a block-format contraction is always %256, hence also %4. K % 4 != 0 stays refused on purpose: the last packet of such a row would read up to three values past the row, which is correct by accident only where the following row exists. Co-authored-by: agent <agent@local>

…es (engine ggml-org#991) The first version of the padded-tile tests used n=8, and that was not enough: BlackMamba's prefill ubatch runs these contractions at 512 tokens. The predicate itself is fine there (both cases are OK on the device against the CPU reference), so the residue that GGML_HRX_STRICT=1 reports at 512 tokens is a different gate -- the model's dt_proj activation is a non-contiguous view at that ubatch size, which the matcher refuses before any size rule. Keeping the coverage anyway: it pins the token-count half of the contract, and it is what turned an assumption into a measurement.

…the CPU (engine ggml-org#991) (#210) * tests: pin the padded shapes at the token count the prefill ubatch uses (engine ggml-org#991) The first version of the padded-tile tests used n=8, and that was not enough: BlackMamba's prefill ubatch runs these contractions at 512 tokens. The predicate itself is fine there (both cases are OK on the device against the CPU reference), so the residue that GGML_HRX_STRICT=1 reports at 512 tokens is a different gate -- the model's dt_proj activation is a non-contiguous view at that ubatch size, which the matcher refuses before any size rule. Keeping the coverage anyway: it pins the token-count half of the contract, and it is what turned an assumption into a measurement. * hrx: densify a row-padded activation once instead of falling back to the CPU (engine ggml-org#991) The SSM ports hand their small projections a column slice of a wider tensor: BlackMamba's `dt_proj` contracts 92 of the 264 columns of `x_proj`, so its activation is a view whose rows are dense but gapped. The plain dense matcher refused it on `!input->contiguous`, which put all 18 `dt_proj` nodes of the prefill ubatch on the CPU (GGML_HRX_STRICT=1 aborts on `node_31`, 18 of 1179 nodes off, 37 graph splits). The kernels bind a packed 2-D tile, so the view cannot be bound directly -- but it can be copied once. The copy path already exists for `GGML_OP_CPY` (`ggml_copy_f32` / `ggml_copy_strided_source_f32`, which takes the source extents and strides as config), so this exports that as `append_f32_densify_dispatch` and uses it from the dense matmul builder: - `common_match_mul_mat_any_format` takes `allow_row_padded_activation`; the input may then be a row-padded F32 view (`nb[0] == sizeof(float)`), and the match records `strided_activation`. - `build_mul_mat_dispatch` allocates a packed transient for exactly that node, emits the densify dispatch, and binds the transient as the activation. One copy of K*tokens floats per node per ubatch (188 KiB here), not a per-layer fallback. - The plain and unary dense matchers opt in; everything else keeps requiring a contiguous activation. Test cases build the model's shape directly (`test_mul_mat_row_padded_activation`: dense weight, activation view of a 264-wide backing at K=92 and K=232, at 512 and 8 tokens). --------- Co-authored-by: agent <agent@local>

…kernel (engine ggml-org#991) (#213) Zamba-7B-v1's attention has 464-wide heads (29 x 16) and its 13 FLASH_ATTN_EXT nodes were the only nodes left off HRX: `13 of 5048 nodes off (FLASH_ATTN_EXT x 13)`, 27 splits, and `GGML_HRX_STRICT=1` aborts on `fattn-2`. 464 is not a multiple of 64, which is what the decode-split kernel declared on `value_head_size` (`range(64, 512), multiple_of(64)`), and the HRX matcher mirrored that declaration. The kernel does not need a 64-wide tile: it publishes a head through `ceil(value_head_size / 128)` output tiles, stores 4-wide vectors at `output_tile * 128 + lane_output_channel`, and clamps every one of those stores with `lane_output_channel < value_head_size`. Its real requirements are a 16-value WMMA fragment (`value_head_size % 16`) and a whole 4-wide channel group (`% 4`, which 16 implies). The 64 block belongs to the *prefill* op, which divides by it (`output_tile_count = value_head_size / 64`) and publishes whole 64-wide tiles with no tail path. So this widens the decode side only: - `flash_attention_decode_split_f32_f16_wmma.loom`: `multiple_of(value_head_size, 64)` -> `16` in the config declaration and in all seven `index.assume`s that repeat it. - `dispatch-flash-attention.cpp`: `kDecodeValueHeadSizeBlock = 16` and `is_supported_decode_value_head_size()`, used by the decode-split matcher alone; the prefill matcher keeps `kPrefillValueHeadSizeBlock = 64`. - The decode predicate's upper bound becomes `kDecodeMaxValueHeadSize = 512`, the ops' own declared value range, instead of the wider Q/K cap (576): a contraction the JIT would reject must never be claimed, because a rejected claim is a hard `failed to prepare command` while declining is a CPU fallback. No test case is added: `test-backend-ops` cannot reach this matcher. Its `FLASH_ATTN_EXT` cases at 464 (and at the old widths 64/192) are all refused by `match_decode_split_flash_attention_f32_f16` at its `has_query_layout` gate, which requires the permuted `[tokens][heads][head_size]` layout a model graph produces while the suite builds contiguous `[head_size, tokens, heads]` tensors; the suite cases that do pass at 96/192 are claimed by a different registration (`common.attention_strided_f32_f16`, V % 32). A temporary probe above that gate printed `dsplit-refused: stride-layout` for hsv=464/96/80/192 in `support` mode, so the device evidence for this change is model-level (KLD against the same nodes on the CPU), not a suite result. Nothing else changes: `qk_head_size` keeps its 16 block, the decode matcher's other gates (query length < 16, head counts, KV capacity, the 4096-byte partial alignment) are untouched, and no instruction in any kernel changed. Co-authored-by: agent <agent@local>

…ane (engine ggml-org#991) Zamba2-2.7B's prefill collapsed to 0.065 of its pp512 throughput at -p 4096, and the cause was a kernel selection cliff rather than the model or the graph: `common.attention_strided_f32_f16` picks its kernel by context length, and past 2048 keys the rows kernel can no longer keep its score row in workgroup memory, so every FLASH_ATTN_EXT node fell to the strided kernel, whose workgroup has one lane per value dimension and each of those lanes recomputes the whole q.k dot product for every key - nkv*dv*d instead of nkv*(d + dv), which is 80x the arithmetic at d = dv = 160 and worse at 224. Measured on one binary, Zamba2-2.7B BF16, `-p 512,2048,2560,4096`: default matcher pp512 395.38 pp2048 270.59 (0.684) pp2560 72.19 (0.183) pp4096 25.69 (0.065) ONEBIT_HRX_ATTN_PER_LANE=1 (strided) pp512 87.48 The second row is the point: at 512 tokens, with the same shapes and the same arithmetic, the strided kernel costs 78% of the throughput, so the kernel and not the context length is the cost. The cliff is exactly at the 2048-key switch (0.684 -> 0.183 between neighbouring lengths). This adds `ggml_attention_rows_blocked_f32_f16`: the rows kernel walking the keys in blocks of 2048 with the flash-attention online softmax - running max and sum carried across blocks, a dv-wide output row in workgroup memory rescaled by exp(old_max - new_max) per block, 1 KB query row + 8 KB score row + 1 KB output row. Keys are padded to whole blocks by clamping the key index and forcing the padded score to the lowest f32, so their probability is exactly zero in both the sum and the value pass and no load ever leaves the buffer. `ggml_attention_rows_f32_f16` is untouched: 2048 keys or fewer keeps the old kernel, so nothing below the switch changes and the A/B above is attributable. Why clamping instead of `min(nkv - block_start, 2048)` as the loop bound: loom cannot invert a subtraction, so `block_start + slot < nkv` was not provable and the JIT refused the kernel outright with SUBRANGE/024 on the key, mask and value loads - a hard `failed to prepare command`, not a CPU fallback. `index.min %j0, %last_key` states the bound as a fact about a min result, which is what the compiler can use. Found with ggml-hrx-compile-kernel, which compiles a single specialization on the CPU from the text loom source, so the diagnosis took seconds instead of a GPU ticket; verified there at kv 2560/3000/4096/5000 before rebuilding. Cases: six new FLASH_ATTN_EXT cases at kv 2560 and 4096 (hsv 64, 160 and 256; 160 is Zamba2's own head size). The suite's longest case was exactly 2048 - the last length that still took the rows kernel - which is why no gate ever saw this. On gfx1151 all six pass against the CPU reference, the chosen specialization is deterministic (1024 workgroups x 64 lanes, 10256 B of workgroup memory), and the model then measures pp2560 72.19 -> 190.93 and pp4096 25.69 -> 131.28 with the <= 2048 path unchanged. The remaining decline with context is the honest O(nkv) attention cost, not a cliff.

…ane (engine ggml-org#991) (#214) Zamba2-2.7B's prefill collapsed to 0.065 of its pp512 throughput at -p 4096, and the cause was a kernel selection cliff rather than the model or the graph: `common.attention_strided_f32_f16` picks its kernel by context length, and past 2048 keys the rows kernel can no longer keep its score row in workgroup memory, so every FLASH_ATTN_EXT node fell to the strided kernel, whose workgroup has one lane per value dimension and each of those lanes recomputes the whole q.k dot product for every key - nkv*dv*d instead of nkv*(d + dv), which is 80x the arithmetic at d = dv = 160 and worse at 224. Measured on one binary, Zamba2-2.7B BF16, `-p 512,2048,2560,4096`: default matcher pp512 395.38 pp2048 270.59 (0.684) pp2560 72.19 (0.183) pp4096 25.69 (0.065) ONEBIT_HRX_ATTN_PER_LANE=1 (strided) pp512 87.48 The second row is the point: at 512 tokens, with the same shapes and the same arithmetic, the strided kernel costs 78% of the throughput, so the kernel and not the context length is the cost. The cliff is exactly at the 2048-key switch (0.684 -> 0.183 between neighbouring lengths). This adds `ggml_attention_rows_blocked_f32_f16`: the rows kernel walking the keys in blocks of 2048 with the flash-attention online softmax - running max and sum carried across blocks, a dv-wide output row in workgroup memory rescaled by exp(old_max - new_max) per block, 1 KB query row + 8 KB score row + 1 KB output row. Keys are padded to whole blocks by clamping the key index and forcing the padded score to the lowest f32, so their probability is exactly zero in both the sum and the value pass and no load ever leaves the buffer. `ggml_attention_rows_f32_f16` is untouched: 2048 keys or fewer keeps the old kernel, so nothing below the switch changes and the A/B above is attributable. Why clamping instead of `min(nkv - block_start, 2048)` as the loop bound: loom cannot invert a subtraction, so `block_start + slot < nkv` was not provable and the JIT refused the kernel outright with SUBRANGE/024 on the key, mask and value loads - a hard `failed to prepare command`, not a CPU fallback. `index.min %j0, %last_key` states the bound as a fact about a min result, which is what the compiler can use. Found with ggml-hrx-compile-kernel, which compiles a single specialization on the CPU from the text loom source, so the diagnosis took seconds instead of a GPU ticket; verified there at kv 2560/3000/4096/5000 before rebuilding. Cases: six new FLASH_ATTN_EXT cases at kv 2560 and 4096 (hsv 64, 160 and 256; 160 is Zamba2's own head size). The suite's longest case was exactly 2048 - the last length that still took the rows kernel - which is why no gate ever saw this. On gfx1151 all six pass against the CPU reference, the chosen specialization is deterministic (1024 workgroups x 64 lanes, 10256 B of workgroup memory), and the model then measures pp2560 72.19 -> 190.93 and pp4096 25.69 -> 131.28 with the <= 2048 path unchanged. The remaining decline with context is the honest O(nkv) attention cost, not a cliff. Co-authored-by: agent <agent@local>

ggerganov approved these changes Apr 15, 2023

View reviewed changes

prusnak merged commit 74f5899 into ggml-org:master Apr 15, 2023

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

Make gguf-py stuff work with numpy 2.0 (ggml-org#991)

60227f4

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

bong-water-water-bong mentioned this pull request Nov 4, 2026

hrx: say once, at load, when a model's nodes fall back to the CPU (engine #991) 1bit-MONSTER/llama.cpp#197

Merged

bong-water-water-bong mentioned this pull request Nov 4, 2026

hrx: admit element-wise dense MUL_MAT at input_size % 32, not only % 256 (engine #991) 1bit-MONSTER/llama.cpp#199

Merged

This was referenced Nov 5, 2026

model: Zamba v1 (Zyphra/Zamba-7B-v1) 1bit-MONSTER/llama.cpp#184

Merged

hrx: let element-wise expert weights use the 32-value packet in MUL_MAT_ID (engine #991) 1bit-MONSTER/llama.cpp#204

Merged

bong-water-water-bong mentioned this pull request Nov 5, 2026

hrx: run a batched dense MUL_MAT as one dispatch per plane (engine #991) 1bit-MONSTER/llama.cpp#207

Merged

bong-water-water-bong mentioned this pull request Nov 5, 2026

hrx: admit element-wise dense MUL_MAT contractions at input_size % 4 (engine #991) 1bit-MONSTER/llama.cpp#208

Merged

bong-water-water-bong mentioned this pull request Nov 5, 2026

hrx: densify a row-padded activation once instead of falling back to the CPU (engine #991) 1bit-MONSTER/llama.cpp#210

Merged

bong-water-water-bong mentioned this pull request Nov 6, 2026

hrx: admit a 464-wide value head in the decode-split flash-attention kernel (engine #991) 1bit-MONSTER/llama.cpp#213

Merged

bong-water-water-bong mentioned this pull request Nov 6, 2026

hrx: score each key once above 2048 keys, instead of on every value lane (engine #991) 1bit-MONSTER/llama.cpp#214

Merged

Navigation Menu

Repository navigation

convert.py: Fix loading safetensors and ggml format on Windows - #991

convert.py: Fix loading safetensors and ggml format on Windows#991
prusnak merged 1 commit into
ggml-org:masterfrom
comex:fix-safetensors-on-windows

comex commented Apr 15, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

comex commented Apr 15, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants