Skip to content

qsa_prompt_attn: Q4_0 KV on tensor cores (mode 4) - long prompts with --kv q4_0 ~19% faster - #452

Closed
architectds wants to merge 1 commit into
Niko1221:mainfrom
architectds:pr/q4-prompt-attn
Closed

qsa_prompt_attn: Q4_0 KV on tensor cores (mode 4) - long prompts with --kv q4_0 ~19% faster#452
architectds wants to merge 1 commit into
Niko1221:mainfrom
architectds:pr/q4-prompt-attn

Conversation

@architectds

@architectds architectds commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What

The tensor-core prompt attention (D-1, qsa_prompt_attn_batch) handles FP16, INT8 and K8V4 KV but refuses Q4_0. With
--kv q4_0, every prompt's QSA attention therefore ran on the FP32 per-query kernel (qsa_decode_attn_batch, 32
queries at a time). This adds a mode for Q4_0 K and V.

  • Mode 4 is mode 1 (int8) with a scale per 32 values. Each q4_0 block's 16 code bytes become 32 exact int8 codes
    (code - 8) at gather, and its fp16 scale goes into the per-group scales: 8 groups of 32 per row, where int8 has 4
    of 64. The scales are applied in FP32 as int8's are: q.k per 32-dim group, and for p.v each group's scale is
    folded into p (two groups per warp).
  • Signed scales. ggml's q4_0 scale is d = max / -8, so about half of a real pool's blocks have a negative scale.
    The p.v fold is bounded by the chunk's largest |scale|.
  • Everything else is mode 1's: q and p split into hi + lo halves, the online softmax. The caller's rotation of q
    and of the output is unchanged.
  • A/B switch. STRATA_PROMPT_ATTN_Q4=0 restores the old kernel.
  • Test. qsa_prompt_attn_parity gets Q4_0 cases (32K and 1.5K contexts) against FP64 and the old kernel. Their
    scales are signed and spread over two decades, as a pool's are.

What it speeds up, and what it does not

  • Prompts with --kv q4_0: +18.5% at 8K tokens, +18.8% at 16K, +19.9% at 30K. With the old kernel the attention
    was 21-22% of a long prompt's GPU time on this machine.
  • Other KV formats: unchanged. Their dispatch is untouched, and the parity test's int8 and fp16 cases pass with
    the same numbers as before.
  • Decode: unchanged. This is the prompt path's batch attention only.
  • HIP: unchanged. The tensor-core kernels are CUDA-only, and HIP keeps the old kernel.

Measured

Setup:

  • RTX 5070 Ti 16 GB on PCIe 3.0 x16 (X370), Ryzen 9 5900XT, DDR4-2133, Windows 11, CUDA 13.0, built on v0.1.34.
  • Qwen3.8-Flash-Next IQ3_XXS, --kv q4_0 --kv-resident 32768, text-only args (4,560 expert slots), --prefill auto
    (8,192-token chunks).

Kernel (qsa_prompt_attn_parity 32768 2048 5):

Case Old kernel Mode 4 Speed vs FP64 (old / new) New vs old
Q4_0, 32K context, 2,048 queries 32.4 ms 8.6 ms 3.76x 4.8e-6 / 6.2e-6 1.6e-6 of the output scale
Q4_0, 1.5K context, 1,500 queries 9.96 ms 2.62 ms 3.79x 6.4e-6 / 5.1e-6 1.3e-6 of the output scale

The int8 and fp16 cases still pass.

Prompts:

  • One binary, the old kernel (STRATA_PROMPT_ATTN_Q4=0) and mode 4 in alternating engine runs (old, new, old, new).
  • The display was off: on this card the desktop compositor otherwise takes up to a third of the GPU.
  • Every prompt has its own first line and corpus offset.
  • The binary also held three independent prompt-path changes (prefill: a group's expert gathers in one launch (bit-identical, long prompts +4%) #439's batched gather and two others), switched off in
    both arms by their own variables, so the arms are v0.1.34 and v0.1.34 + this commit.
Prompt tokens Old kernel (tok/s, run 1 / run 2) Mode 4 (tok/s, run 1 / run 2) Change (mean)
~8K 1,579 / 1,610 1,970 / 1,805 +18.5%
~16K 1,599 / 1,556 1,729 / 2,047 +18.8%
~30K 1,574 / 1,596 1,962 / 1,839 +19.9%

Quality (not bitwise, as D-1 is not):

  • Needles: tools/needle_bench.py's haystack and needle, driven through the engine. 8K, 16K and 32K at depths
    10/30/50/70/90: 15 of 15 with the old kernel, 15 of 15 with mode 4.
  • Greedy continuations part from the old kernel's after a few tokens, as the int8 tensor-core kernel's do: the
    model amplifies any FP32-level change.
  • A bug the tests caught. An earlier draft bounded the p.v fold by the largest positive scale, as int8's are
    bounded. On real K/V that overflowed FP16 and the model returned NaN (needles 0 of 15). The parity test had passed
    it, because its synthetic scales were all positive. The test's scales are now signed, and it fails on that bound
    (new vs old 0.93 of the output scale).

Not tested here

  • HIP: the kernels are CUDA-only, and HIP keeps the old kernel as before.
  • Turing (sm_75): mode 4 is in the v1 template kernel, which runs there through pairs of m16n8k8 MMAs. Not run.
  • A layer split, Linux, and packs other than IQ3_XXS (the attention does not depend on the pack).

Developed with an AI coding assistant; all numbers measured on the machine above.

🤖 Generated with Claude Code

@architectds @claude
qsa_prompt_attn: Q4_0 K and V on tensor cores (mode 4) - long prompts…
e71abf8 
… with --kv q4_0 no longer take the old kernel

The tensor-core prompt attention (D-1) refused Q4_0 pools, so --kv q4_0 read every prompt through the FP32 per-query
kernel. Mode 4 is mode 1 (int8) with a scale per 32 values: each q4_0 block's 16 code bytes become 32 exact int8
codes (code - 8) at gather, its fp16 scale goes to the per-group scales, which are applied in FP32 as int8's are
(q.k per 32-dim group; p.v folds each group's scale into p, two groups per warp). The p.v fold is bounded by the
chunk's largest |scale|: q4_0 scales are signed (ggml's d = max / -8). The caller's rotation of q and of the output
is unchanged. STRATA_PROMPT_ATTN_Q4=0: the old kernel (A/B).

qsa_prompt_attn_parity gets Q4_0 cases with signed scales spread over two decades, as a pool holds them (32K and
1.5K contexts), against FP64 and the old kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@architectds

architectds commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Next-token comparison at long prompts. This is the method of bench/results/2026-09-27-cache-parity, applied at
the one position a prompt path decides: the next token after the prompt.

Setup:

  • 24 raw-text prompts (12 code, 12 prose) of 4K, 8K, 12K and 16K tokens, each at its own corpus offset.
  • Each prompt read with the old kernel and with mode 4.
  • The logits after the prompt compared.
Old kernel vs mode 4
Top-1 next token the same 24 of 24
KL(old ‖ new) median 1.95e-3, mean 2.2e-2, max 0.17
Log-likelihood of the actual next token new - old = +0.026 ± 0.046 nats (mean ± standard error): no difference

For scale: on 60 shorter prompts (200-1,000 tokens), the engine's own two GPU prompt paths (MMQ against
STRATA_PREFILL_MMQ=0) differ by median KL 2.2e-3 and max 0.15, with top-1 60 of 60.

Build: a v0.1.33-based build with this same kernel, plus a serve-mode STRATA_DUMP_FIRST_LOGITS hook for reading
the logits. The hook is not in this PR. Same machine as above, display off.

architectds pushed a commit to architectds/Strata that referenced this pull request Oct 2, 2026
Chen Bao @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
88ae736 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
architectds pushed a commit to architectds/Strata that referenced this pull request Oct 2, 2026
Chen Bao @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
cabc091 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
architectds pushed a commit to architectds/Strata that referenced this pull request Oct 2, 2026
Chen Bao @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
4509adf 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
architectds added a commit to architectds/Strata that referenced this pull request Oct 3, 2026
@architectds @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
5f5834d 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
architectds added a commit to architectds/Strata that referenced this pull request Oct 3, 2026
@architectds @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
99e8b36 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Niko1221 added a commit that referenced this pull request Oct 3, 2026
@Niko1221 @claude
qsa_prompt_attn: Q4_0 mode 4 on sm_80+ only (#452 follow-up)
79670a6 
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Niko1221 added a commit that referenced this pull request Oct 3, 2026
@Niko1221 @claude
Engine 0.1.38: faster prompts (#372 #374 #413 #452), Q5_0 experts on …
5123c97 
…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
@Niko1221

Niko1221 commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Released in 0.1.38: --kv q4_0 prompts take mode 4 on sm_80 and newer. RTX 20 (sm_75) keeps the old kernel until a Turing parity run shows mode 4 there; a run of qsa_prompt_attn_parity on an RTX 20 card would let us enable it. Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

@Niko1221 Niko1221 closed this Oct 3, 2026
architectds added a commit to architectds/Strata that referenced this pull request Oct 3, 2026
@architectds @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
278bb3e 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 99e8b36)
architectds added a commit to architectds/Strata that referenced this pull request Oct 3, 2026
@architectds @claude
README: the fork on v0.1.38 - its three PRs are in Strata itself now (N…
05c0f36 
…iko1221#452 Niko1221#453, Niko1221#439 as Niko1221#372)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
architectds added a commit to architectds/Strata that referenced this pull request Oct 3, 2026
@architectds @claude
Images on demand: the expert cache lends the image encoder its VRAM w…
704d32d 
…hile it is loaded (--vision-on-demand; engine --lendable-cache, LEND / RECLAIM)

The engine: --lendable-cache makes the expert cache 64 MiB pieces of one reserved address range (CUDA virtual memory,
looked up through cudaGetDriverEntryPointByVersion, so nothing new is linked).  --serve takes two lines between
requests: LEND <MiB> marks the experts in the cache's last MiB as CPU misses, uploads the residency table and unmaps
and frees those pieces (another process can have the VRAM); RECLAIM maps new pieces at the same addresses, refills
every lent slot from the arena (first and last read back; all with STRATA_VERIFY_RECLAIM=1) and marks them resident
again.  While the loan is out the adaptive tier leaves the lent experts alone and the prompt path lays its buffers out
below the loan: the startup chunk if it fits there, else the usual sizes below it (12288 -> 8192, not 6144).  RECLAIM
is refused while the VRAM is not free yet (the borrower still exiting): under WDDM an over-full card pages instead of
failing.

The server: --vision-on-demand ("vision_on_demand": true in the config; it adds --lendable-cache) starts the image
encoder with the first picture the cache does not hold, after a LEND of --vision-lend-mib ("vision_lend_mib", default
1400), keeps it while pictures keep coming, and unloads it --vision-idle seconds ("vision_idle_s", default 600) after
the last one - between requests only - then RECLAIMs.  An engine restart or unload ends the loan; an engine that cannot
lend keeps the encoder resident as before.  With --expert-cache auto the reserve is the one without images: the engine
measures the VRAM that is free once its cache is written, before any encoder is there.

Measured on an RTX 5070 Ti (IQ3_XXS, 400K context, 0.1.34 + Niko1221#439 Niko1221#452 Niko1221#453,
bench/results/2026-10-01-vision-on-demand): decode 59.8 tok/s with the encoder resident, 66.0 on demand before any
picture, 55.4 while it is loaded, 65.2 after it unloads; the first picture 8.3 s to the answer; LEND 0-16 ms, RECLAIM
0.9-1.15 s with every slot verified; the engine's VRAM the same after 5 cycles; with --pcie-frac 0 (byte-identical
repeats) greedy replies after RECLAIM are the replies before LEND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 99e8b36)
rwkeyes added a commit to rwkeyes/Strata_Dirigo that referenced this pull request Oct 3, 2026
@rwkeyes
Merge tag v0.1.38 into the AVX1/retune fork
fce1e32 
Upstream v0.1.38: faster prompts (one gather per expert group Niko1221#372, the first chunk's PLE rows
beside layer 0 Niko1221#374, DeltaNet three heads per thread Niko1221#413), --kv q4_0 prompts on tensor cores
Niko1221#452, Q5_0 experts on the GPU Niko1221#473, IQ4_XS on AVX-2 Niko1221#415, unbuffered expert loading on Windows
Niko1221#357/Niko1221#362, the --peer-device second-GPU expert tier Niko1221#531, a 6 GB card starts Niko1221#496, the Host
header / DNS-rebinding checks, PR batch.

Our side: the AVX1 floor (STRATA_ISA_FLOOR, cpu_avx1_ok, the relaxed gate, the scalar router
fallback), the AVX1 router kernel (kq_avx1.cpp), the retune patch + setup hooks.
@architectds
architectds deleted the pr/q4-prompt-attn branch October 4, 2026 22:52
Niko1221 added a commit that referenced this pull request Oct 6, 2026
@Niko1221
qsa_prompt_attn: Q4_0 mode 4 on sm_80+ only (#452 follow-up)
482e22b 
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.
Niko1221 added a commit that referenced this pull request Oct 6, 2026
@Niko1221
Engine 0.1.38: faster prompts (#372 #374 #413 #452), Q5_0 experts on …
cb5cfb7 
…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221
qsa_prompt_attn: Q4_0 mode 4 on sm_80+ only (Niko1221#452 follow-up)
4762b37 
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221
Engine 0.1.38: faster prompts (Niko1221#372 Niko1221#374 Niko1221#413 N…
209de46 
…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221 @claude
qsa_prompt_attn: Q4_0 mode 4 on sm_80+ only (Niko1221#452 follow-up)
0dc6c1d 
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221 @claude
Engine 0.1.38: faster prompts (Niko1221#372 Niko1221#374 Niko1221#413 N…
38bdc08 
…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
@afanty2021 afanty2021 mentioned this pull request Oct 15, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@architectds @Niko1221