[matrix9neonebuchadnezzar2199-sketch]

Same Windows machine and engine, so the two sizes can be compared without an engine change.

Hardware

RX 7900 XT (20464 MiB, gfx1100), Ryzen 7 9800X3D, 95.6 GiB RAM, Windows 11 Pro build 26300. Ready-made HIP engine 0.1.40.2 at e8ca9af. Model files are on a separate NVMe.

Model

Qwen3.8-Flash-Next GSQ-RCO, IQ3_S and IQ3_XXS, one loaded at a time. Vision off.

Configurations

Three fresh runs each of a ~100-token prompt and a 38,184-token synthetic log, with max_tokens set to 256. Rates come from last_timings, not from dividing generated tokens by the whole request. cache_n was 0 on every run.

• IQ3_S: context 131072, KV q4_0, expert cache auto (5512 slots, 10.48 GiB). Long prompt 1015.8 tok/s median (1015.0–1016.3), decode 24.3 tok/s median (21.4–25.4).
• IQ3_XXS: context 262144, KV int8, expert cache auto (6376 slots, 10.31 GiB). Long prompt 1400.1 tok/s median (1399.7–1400.3), decode 72.3 tok/s median (59.4–73.1).

Report: bench/results/2026-10-07-community-rx-7900-xt/

Limitations

One machine. No TTFT, no needle sweep, and no VRAM sample during generation. The short-prompt tok/s figure is overhead, not prefill speed. On IQ3_S the counted long-prompt decode (24.3 tok/s) was slower than its warm-up (51.7); the cause was not isolated. On IQ3_XXS one long run hit the 256-token cap and the visible answer was cut off at amber-keel-2. This does not replace the existing RX 7900 XTX report; that machine used different settings.

[matrix9neonebuchadnezzar2199-sketch]

Follow-up on the same PC: I ran the same prompts again with the v0.1.40 Windows HIP build (strata.exe only; same packs and the same arguments). Rows are in the report folder as runs-iq3_s-0140 and runs-iq3_xxs-0140.

IQ3_XXS, long prompt, median (range):

• 0.1.40: prompt 1384.9 tok/s (1384.1–1386.7), decode 66.0 (53.6–67.5)
• 0.1.40.2: prompt 1400.1 (1399.7–1400.3), decode 72.3 (59.4–73.1)
  The decode ranges overlap. Not a clear difference.

IQ3_S, long prompt, median (range):

• 0.1.40: prompt 1357.8 tok/s (1355.5–1361.0), decode 54.1 (46.4–57.1)
• 0.1.40.2: prompt 1015.8 (1015.0–1016.3), decode 24.3 (21.4–25.4)
  These ranges do not overlap. On 0.1.40 the counted decode stayed near its warm-up (46.8). On 0.1.40.2 the warm-up was 51.7 and the three counted runs fell to 21.4–25.4.

Expert cache was auto on both, and it did not pick the same size for IQ3_S:

• 0.1.40: 4255 slots, 8.12 GiB, 2170 MiB VRAM free after load
• 0.1.40.2: 5512 slots, 10.48 GiB, 323 MiB VRAM free after load
  So this is not a pure engine comparison. I did not hold the cache size fixed.

[matrix9neonebuchadnezzar2199-sketch]

The IQ3_S slowdown was checked one hypothesis at a time.

1. The first 0.1.40 versus 0.1.40.2 pair used the same prompts and the same arguments, with expert cache left on auto. Auto did not pick the same cache: 4,255 slots / 8.12 GiB / 2,170 MiB free on 0.1.40, against 5,512 slots / 10.48 GiB / 323 MiB free on 0.1.40.2. Long-prompt ranges did not overlap. On 0.1.40.2 the warm-up decode was 51.7 tok/s and the three counted runs fell to 21.4–25.4. IQ3_XXS on the same two engines did not show that drop: its long-prompt decode ranges overlap. So the engine was not uniformly slower, and that IQ3_S pair was not a pure engine comparison.

2. The 323 MiB start matches the shape in docs/DETAILS.md: on Windows an oversized expert cache can page instead of failing, the hit rate stays high, and the tell is a small "MiB of VRAM free with everything loaded". That 0.1.40.2 IQ3_S log also printed the WDDM verify-window warning. Hit rate on the slow runs stayed about 71–89%, so the drop is not cache misses. The same warning and about 328 MiB free on IQ3_XXS 0.1.40.2 did not collapse decode, so low free VRAM alone was not enough to settle the cause.

3. The check was to give 0.1.40.2 IQ3_S the same headroom as the 0.1.40 start, without pinning a slot count. A fixed --expert-cache does not shrink after the slots are written, so an oversized fixed value would page again. --vram-reserve-mib was raised from 700 to 2600 and the cache stayed on auto, so it could still shrink. 2600 was chosen so the free figure could land near 2,170 MiB.

It did. Auto settled at 4,299 slots (8.20 GiB) with 2,224 MiB free. The little-room warning from the 323 MiB start was absent.

Long-prompt medians of 3 counted runs. Every cache_n is 0. Percents on the last row are against the 0.1.40 medians.

	prompt tok/s	decode tok/s
0.1.40, 2,170 MiB free	1,357.8 (1,355.5–1,361.0)	54.1 (46.4–57.1)
0.1.40.2, reserve 700, 323 MiB free	1,015.8 (1,015.0–1,016.3)	24.3 (21.4–25.4)
0.1.40.2, reserve 2600, 2,224 MiB free	1,355.9 (1,354.1–1,358.7), −0.1%	55.5 (44.9–59.6), +2.6%

The reserve-2600 ranges overlap the 0.1.40 ranges. Warm-up decode was 46.0 tok/s, and the three counted runs stayed at 44.9–59.6. They did not fall into the 21 tok/s band. All three quoted amber-keel-2904 and finished with stop.

On 0.1.40.2 alone, reserve 700 to 2600 is +33.5% on the long prompt (1,015.8 to 1,355.9) and +128% on decode (24.3 to 55.5).

With the cache size and free VRAM held next to the 0.1.40 start, this 0.1.40.2 IQ3_S run matches that 0.1.40 run. The earlier drop tracked the auto cache that left 323 MiB free.

Rows: bench/results/2026-10-07-community-rx-7900-xt/runs-iq3_s-01402-reserve.json

[Niko1221]

Thank you, this is the most useful kind of report: same PC, same packs, one variable changed at a time.

What we found: the auto sizing formula (free VRAM minus the reserve, divided by the slot size) and the check after the slots are written are the same in 0.1.40 and 0.1.40.2, so we cannot point to one commit that made auto take more for the same reserve. We could not reproduce it on an NVIDIA card either: on our RTX 5070 with Q2_0, 0.1.40 and 0.1.40.2 pick the same slot count and leave the same VRAM free (408 vs 406 MiB), so the difference you saw is in how much VRAM was free before the cache was sized and what was allocated after it on your Windows HIP setup, which we cannot see from here. What your numbers do show is that the default 700 MiB is not enough headroom for this card under WDDM: 0.1.40 happened to end with 2,170 MiB free, 0.1.40.2 with 323 MiB, and with --vram-reserve-mib 2600 auto settled at 4,299 slots (8.20 GiB) with 2,224 MiB free, long prompt 1,015.8 -> 1,355.9 tok/s (+33.5%) and decode 24.3 -> 55.5 tok/s (+128%), inside the 0.1.40 ranges.

So 0.1.40.3 restores that headroom by default: on Windows, with --expert-cache auto and no --vram-reserve-mib given, a card of 16 GiB or more keeps 2,560 MiB free after the cache (the engine says so in its log). A reserve you give is used as given, and smaller cards are unchanged. The cache size does not change the output bits. We could not test this on a Windows AMD card (we have none); the logic is plain C++ that does not depend on the GPU vendor, and on our NVIDIA PC the 12 GB card is below the 16 GiB threshold and behaves exactly as before.

Could you try the 0.1.40.3 HIP zip with no reserve setting and paste the expert cache auto and MiB of VRAM free with everything loaded lines, plus the decode rate? We expect roughly 4,300 slots and about 2,200 MiB free. Thanks again for the benchmark folder.

[matrix9neonebuchadnezzar2199-sketch]

Follow-up on engine 0.1.40.3, same PC, same IQ3_S pack, same prompts, and the same arguments as the slow 0.1.40.2 run: expert cache auto, no explicit --vram-reserve-mib. /v1/status reported engine 0.1.40.3.

The start log applied the #1376 floor: on a card this size, auto keeps 2,560 MiB free after the cache. It reserved 2,560 MiB from 13.74 GiB free (proposed 4,445 slots), shrank once because only 1,267 MiB was free once the slots were written, and settled at 5,097 slots (9.69 GiB). With everything loaded, 2,176 MiB was free. That is next to the 0.1.40 start (4,255 slots, 8.12 GiB, 2,170 MiB free) and the manual reserve-2600 run on 0.1.40.2 (4,299 slots, 8.20 GiB, 2,224 MiB free). The little-room warning from the 323 MiB start was absent.

Long-prompt medians of 3 counted runs. Every cache_n is 0. Percents on the last row are against the 0.1.40 medians.

	prompt tok/s	decode tok/s
0.1.40, 2,170 MiB free	1,357.8 (1,355.5 to 1,361.0)	54.1 (46.4 to 57.1)
0.1.40.2, auto, 323 MiB free	1,015.8 (1,015.0 to 1,016.3)	24.3 (21.4 to 25.4)
0.1.40.3, auto, 2,176 MiB free	1,375.8 (1,374.1 to 1,377.1), +1.3%	56.6 (50.8 to 61.9), +4.6%

The 0.1.40.3 prompt range does not overlap the 0.1.40 prompt range. The decode ranges do. Warm-up decode was 50.5 tok/s, and the three counted runs stayed at 50.8 to 61.9. They did not fall into the 21 tok/s band. All three quoted amber-keel-2904 and finished with stop.

Against the slow 0.1.40.2 auto run, the long prompt is +35.4% (1,015.8 to 1,375.8) and decode is +133% (24.3 to 56.6).

With auto left to choose the cache, 0.1.40.3 left about 2 GiB free, and its long-prompt medians are within 5% of the 0.1.40 run.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.