Repository navigation
Conversation
…loaded them and never read them) On the SYCL port STRATA_HC_Q8=1 did nothing: native_dense kept a Q8_0 copy only when the GGUF stored the projections as Q8_0 (the ISTA files store BF16, so nothing was copied), and fused_gr_read_multi had no Q8_0 read - it ignored q8_down / q8_up and read the BF16 copies either way. Now the loader quantizes BF16 projections (type 30) to Q8_0 at load (ggml's rounding, on the host; a GGUF's own Q8_0 blocks as they are) in the read's split layout (quants row-major, then the fp16 scales; fused_gr_q8_0_split_host), and fused_gr_read_multi runs a Q8_0 read when the pointers are set: two kernels, one row per 16-lane sub-group lane, the norm and the partial reduction folded into the down kernel (last-arriving work-group finishes a row group; no co-residency assumed), K split over the sub-groups in the up kernel. Inject rows stay BF16; the BF16 copies stay in the arena for the prompt path. With the flag unset the pointers are null and the read is the BF16 one as before. Not bitwise the BF16 read (Q8_0-rounded weights, another summation order), as STRATA_HC_Q8 is on CUDA. gr_q8_parity --selftest: the read against an FP64 reference on its Q8_0 bytes (T 1..8, pending write, inject rows), R_out bitwise the BF16 read's, graph replays bitwise, the fixed-stride fallback bitwise. docs/INTEL.md has the entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
hotschmoe
changed the title
|
/reopen Rebased on 0.2.0-beta ( |
|
Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review. |
d97e9e1 to
f1cdc43
Compare
|
Tested on 2x Arc Pro B70 (CachyOS, kernel 7.2.11, xe) with Swift 1.5 Flash-Next IQ2_XS: layer split 26, every expert in VRAM, int8 KV, MTP spec 4, 262144 context. The image is v0.2.0-beta (3d1fb48) plus our SYCL patches, including #2804 and #2808. Each arm ran on a fresh engine, two rounds. Greedy, 5 fixed prompts, decode tok/s:
That is +7.6% (+5.5% to +10.1% per prompt). The second round matched within 0.6 tok/s. As the PR says, the output changes. Code, maths and edit gave identical text. Prose diverged at character 124 and story at character 60, at the same place in both rounds, and both still read fine. The story prompt's MTP acceptance dropped from 65% to 57%; it is still faster. VRAM went up by 425 MiB on card 0 and 361 MiB on card 1 (24,539 to 24,964 and 22,854 to 23,215 MiB), consistent with the BF16 copies staying resident. On our Swift IQ3_S profile (every expert resident) that left 31 experts out of VRAM at With #2802 and #3248 also on: 136.7 tok/s mean (+11.2%). No xe resets or page faults, and the kernel suite passes (39 tests; the 1 failure is We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine. |
Problem and result
On the SYCL (Intel) build
STRATA_HC_Q8=1did nothing. The loader kept a Q8_0 copy of the hyper-connection projections only when the GGUF stored them as Q8_0 (the ISTA files store BF16, so nothing was copied), and the port'sfused_gr_read_multihad no Q8_0 read: it ignoredq8_down/q8_upand read the BF16 copies whatever the flag said (fused_gr.dp.cpp: "the Q8_0 reads (S23 STRATA_HC_Q8) are not built"). Still the case on 0.2.0-beta.With this change the flag does on SYCL what it does on CUDA: the hyper-connection down / up projections (the layers' and the final mixer's) are read as Q8_0 in the verify window, from the GGUF's own Q8_0 blocks or from its BF16 values quantized at load. Flag unset: nothing changes.
Result on 0.2.0-beta, two B70s, one stream: verify window 39.4 / 40.0 -> 36.8 / 36.9 ms (-7%), decode 70.6 / 73.9 -> 82.4 / 87.8 tok/s (part of it a different draft acceptance of the changed output). Prompt reading unchanged. Costs 0.63 GiB of VRAM (below).
Related discussion
Bug report #2809 (closed in the 0.2.0 tracker reset). #2812 (
STRATA_GR_FUSED=1) is stacked on this branch.Approach
sycl/src/kernels/cuda/fused_gr.dp.cpp,namespace gr_q8: two kernels. The down kernel stages a 512-column slab ofR' * w_normfor the T rows with the slab's sums of squares, each lane holds 4 rows' quants over its sub-group's 32 columns, the 16 sub-groups add their partials in a fixed order, and the last work-group of a row group to arrive (an atomic counter, no waiting, so it is safe inside a SYCL graph) applies rs per stream and resets the counter. The up kernel splits K = 320 over five sub-groups summed in a fixed order, then the default read's epilogue. One guarded line infused_gr_read_multi; null pointers (the default) skip it. Rows not at the callers' fixed strides fall back to the BF16 read with one stderr line. Scratch per queue, allocated outside any capture.sycl/src/core/native_dense.cpp: with the flag, hyper-connection tensors of GGUF type 8 (Q8_0) and type 30 (BF16, quantized on the host with the file'sq8_0_of, ggml's rounding) are uploaded in the split layout, shape-checked, another stage's layers skipped in a layer split.sycl/include/strata/kernels/fused_gr_q8.hpp(new),sycl/src/kernels/gr_q8_parity.cpp(new, in the--selftestloop),docs/INTEL.md.Not changed:
src/,include/(CUDA / HIP / Mac untouched), the default SYCL read,STRATA_GR_V3,STRATA_HC_SPLIT.Validation
Setup: 0.2.0-beta
maine88ee39 against this branch rebased on it. 2x Intel Arc Pro B70 32 GB, kernel 7.1, Compute Runtime 26.22.38646.4 at run time; built with the repository dev image (sycl/tools/Dockerfileon the currentllama-sycl-b70:stable, IGC 2.41.5; an older cached base with IGC 2.36.3 cannot build 0.2.0, see #3219), AOTbmg-g31. Model Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS,--layer-split 30 --prefill 8192 --max-context 32768 --vram-reserve-mib 512 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts --expert-cache auto, one slot, greedy, two fixed coding prompts with 512-token replies, a fresh engine per arm, then a fresh 4K prompt. Coherence check and per-card health after every stop.Passed:
gr_q8_parity --selftest(0.2.0-beta + this branch, head f1cdc43, one B70):gr_q8_parity: pass (0 failing of 33 cases)(T = 1..8 x pending write x inject rows; against an FP64 reference from the same Q8_0 bytes within 1e-4; R_out bitwise; graph replay bitwise; Q8_0mixeddiffers from BF16 by the weights' rounding, which proves the Q8_0 bytes are read; fallback bitwise).--max-context 32768 --vram-reserve-mib 512(decode tok/s, tokens per window, ms per window):STRATA_HC_Q8=1--max-context 65536; this branch unset gave the same SHAs at 65536 and at 32768). Flag set: coherent, text differs (Q8_0 weights). Coherence true and per-card health passed after every stop.Failed:
--max-context 65536with the default 1,024 MiB reserve the engine refuses to start with the flag (REFUSED: 310 experts are neither in VRAM nor mirrored), the same as on 0.1.x: the 0.63 GiB of Q8_0 copies come out of the expert cache. This is the VRAM limit below, not a crash.Not run: the teacher-forced NLL on 0.2.0 (on 0.1.x, 4,762 tokens: 3.1974 unset vs 3.1699 set, no degradation measured, not claimed as an improvement); kernel timings on 0.2.0 (unchanged code; on 0.1.x BF16 -> Q8_0 read T = 1 / 4 / 6: 38.7 -> 24.7, 50.8 -> 29.3, 62.0 -> 34.4 us); two streams; CUDA/HIP (not touched);
ctestas a whole.Limits
add_hc_skip/hc_bf16_view) is not ported here.STRATA_HC_Q8_INJECThas no effect on SYCL: the read takes the inject rows from BF16.docs/INTEL.mdsection quotes the 0.1.x measurement (main 296b59d); I can swap in the 0.2.0 numbers.🤖 Generated with Claude Code