Repository navigation
Conversation
…GEMMs read (bitwise) With STRATA_BF16_TC the prompt path converted xn16 (T x 10240) from BF16 to FP16 before each of its two GEMMs (down and inject), in 16M-element slices: ~2.8 s of a 130K prompt read on an RTX 2080 Ti. The four xn16 writers now store gemm.cu's conversion of the BF16 value (bf_to_hf_conv: the same clamp and rounding as bf16_to_f16_kernel), and Gemm::bf16(..., x_f16) skips the conversion on the tensor-core path (same slices, same GEMMs) and converts back to BF16 on any other. Only when the stage runs BF16 on FP16 tensor cores, without the BF16X2 low parts or a padded xn16 stride; the MTP draft's own read keeps BF16. STRATA_XN_F16=0 turns it off. First logits after a 98K prompt: bitwise identical; prompt read 1431 -> 1473 tok/s; a 4K run writes the same tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xrip
added a commit
to xrip/Strata
that referenced
this pull request
Oct 12, 2026
…mpt read, first logits byte-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Niko1221
pushed a commit
that referenced
this pull request
Oct 17, 2026
…-core GEMMs read (bitwise) With STRATA_BF16_TC the prompt path converted xn16 (T x 10240) from BF16 to FP16 before each of its two GEMMs (down and inject), in 16M-element slices: ~2.8 s of a 130K prompt read on an RTX 2080 Ti. The four xn16 writers now store gemm.cu's conversion of the BF16 value (bf_to_hf_conv: the same clamp and rounding as bf16_to_f16_kernel), and Gemm::bf16(..., x_f16) skips the conversion on the tensor-core path (same slices, same GEMMs) and converts back to BF16 on any other. Only when the stage runs BF16 on FP16 tensor cores, without the BF16X2 low parts or a padded xn16 stride; the MTP draft's own read keeps BF16. STRATA_XN_F16=0 turns it off. First logits after a 98K prompt: bitwise identical; prompt read 1431 -> 1473 tok/s; a 4K run writes the same tokens.
Niko1221
added a commit
that referenced
this pull request
Oct 17, 2026
… parameters (the shared headers changed; SYCL never uses the FP16 image) The SYCL build of the sm_75 series failed with 'out-of-line definition of bf16 does not match any declaration' (sycl/src/prefill/gemm.dp.cpp); the four gr_* writers would have failed at link. Gemm::bf16_converts_x() answers false.
Owner
|
Thanks rafatxf. Ported with your authorship. It only does something with STRATA_BF16_TC=1 on a 7.5 card, so it is opt-in and we could not measure it. One thing for future patches: the SYCL tree has its own copies of Gemm::bf16 and the gr_* writers, so a header signature change breaks the SYCL build until those are updated. We added that fix on our side. |
xrip
added a commit
to xrip/Strata
that referenced
this pull request
Oct 17, 2026
Conflicts: upstream took the Turing kernels this fork had cherry-picked (Niko1221#1821 Niko1221#1825 Niko1221#2069-Niko1221#2072 Niko1221#2081 Niko1221#2091 Niko1221#2093), now opt-in with STRATA_SM75_PERF=1: their versions kept (native_mmvq, qsa_select, qsa_prompt_attn, prefill, gemm, kernels). Upstream's sm_75 table replaces ours (STRATA_SM75_TABLE gone; STRATA_MMVQ_IL_ROWS can set rows). conversation_cache.hpp: the disk tier's on_evict on upstream's serial-aware evict_oldest. generate.cpp: the disk tier's read-back before Niko1221#1164's borrow (a disk restore never borrows), --conversation-cache-park-gain beside the disk options. CMakeLists: prefill_mmq_bench kept beside upstream's new tests. Tokenizer: our long-text cache beside max_special_len. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
imanu86
added a commit
to imanu86/Strata-2080Ti
that referenced
this pull request
Oct 18, 2026
…l dequant a 16 B come nel Daily Patch 14 della pila (ora 11), ridotta: il dequant con store da 16 B e' gia' in upstream (Niko1221#2081, dietro STRATA_SM75_PERF=1 / STRATA_TC_SM75); qui sul fork resta acceso di default su una 7.5 (STRATA_TC_SM75=0 lo spegne). STRATA_SELECT_SIMT acceso di default sotto sm_80 (=0 spento, =2 tile grandi): upstream non l'ha preso (PR Niko1221#2080). hc16 NON portato: upstream ha un equivalente piu' largo, STRATA_BF16_TC=1 con STRATA_XN_F16 (Niko1221#655/Niko1221#2091); resta da confrontare in tempo di prompt e KL prima di cambiare il Daily (vedi docs/UPSYNC_MAIN.md). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Author
|
Ported in 0.1.45, thanks! Turing numbers for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The follow-up I mentioned in #655. With
STRATA_BF16_TC=1the prompt path convertsxn16(T × 10240) from BF16 to FP16 before each of its two GEMMs (the down and inject projections), in 16M-element slices: on a 130K prompt read that was ~2.8 s ofbf16_to_f16_kernel(nsys), 3.2 % of the read.Now the four
xn16writers (gr_norm,gr_norm_rs,gr_write_norm_rs,gr_write_cvec_norm_rs) can storegemm.cu's conversion of the BF16 value directly (bf_to_hf_conv: the same clamp of finite values beyond ±65504 and the same rounding asbf16_to_f16_kernel), andGemm::bf16(..., x_f16)skips the conversion on the tensor-core path - the same slices, so the same GEMMs. Any other path (one output row, no room for the buffers) converts the image back to BF16 a slice at a time (exact: every value is a BF16 that fits FP16, or was clamped as the tensor-core path clamps it).It is on only where it changes nothing: the stage runs BF16 on FP16 tensor cores (
Gemm::bf16_converts_x()), without the BF16X2 low parts and with the unpaddedxn16stride; the MTP draft's own read keeps BF16.STRATA_XN_F16=0turns it off.Setup
RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, driver 595, SM 1650 MHz, 240 W;
v0.1.43. Model Q2_0,--resident-experts,--max-context 524288,--kv int8 --kv-resident 32768,STRATA_BF16_TC=1.Results
STRATA_XN_F16=0=1First logits after the 98K prompt (
STRATA_DUMP_FIRST_LOGITS): bitwise identical. A 4K one-shot run writes the same tokens.🤖 Generated with Claude Code