You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for the report. I couldn't reproduce it on the cards I have: on an RTX 4070 laptop (sm_89) with Qwen3.6 UD-Q3_K_M, poly.5, poly.6 and 0.1.45-poly.1 all write at 72-81 tok/s. A T4 is sm_75, which neither my laptop nor my 5090 is, so I need a bit more from yours:
Which size (UD-Q2_K_XL, UD-Q3_K_M or UD-Q4_K_M), and your strata-qwen36-<size>.json. In particular: is "parallel" in it? Each batch slot takes VRAM from the expert cache as its conversation grows, which can slow a single chat on a 16 GB card.
From the engine log (strata-qwen36-<size>.log) of each version, the line loaded in ... s; N of 10496 routed experts in VRAM. If 0.1.45 holds fewer experts, that's most likely the cause.
How you measured the 70 and 55. The speed in the chat page moves a lot with the question, so the same command on both versions would help: python polystrata.py benchmark --model <model name> (or BENCH.bat / ./bench.sh).
One thing worth a try on a T4: STRATA_SM75_PERF=1 in the environment (or in the config's "env"). It turns on the Strata project's Turing kernels, which 0.1.45 added and leaves off by default.
If you can, also try poly.6's engine: download strata-poly-windows-x64-04beb8305ea6c2ff.zip from the 0.1.44.1-poly.6 release (or build that tag on Linux) and put it in engine-poly/ instead, keeping a copy of the current one. That tells me whether the drop came with poly.6's parallel requests or with the 0.1.45 merge.
I ran benchmarks for different versions, and the results are roughly the same. It seems something strange is going on in the web UI, as it was showing varying speeds.
# v0.1.44.1-poly.5
## log
strata-poly: loaded in 35.7 s; 6043 of 10496 routed experts in VRAM (10.7 GiB), 14 CPU threads
## bench
### without STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 74.20 (68.70 - 77.40, n=9)
chat first token (s) 0.31 (0.30 - 0.41, n=9)
4K prompt tok/s 804.00 (795.80 - 818.50, n=3)
4K first token (s) 5.01 (4.92 - 5.06, n=3)
16K prompt tok/s 849.10 (847.60 - 864.10, n=3)
16K first token (s) 18.93 (18.60 - 18.97, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (338.00 - 341.00, n=3)
agent turn first token (s) 5.54 (3.24 - 6.04, n=15)
agent decode tok/s 55.80 (49.90 - 63.40, n=15)
1 clients total tok/s 56.16 (55.90 - 59.25, n=3)
### with STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 72.80 (67.80 - 78.00, n=9)
chat first token (s) 0.30 (0.30 - 0.34, n=9)
4K prompt tok/s 811.70 (810.50 - 820.70, n=3)
4K first token (s) 4.96 (4.91 - 4.97, n=3)
16K prompt tok/s 861.10 (856.10 - 876.50, n=3)
16K first token (s) 18.67 (18.34 - 18.78, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (341.00 - 341.00, n=3)
agent turn first token (s) 5.49 (3.20 - 5.88, n=15)
agent decode tok/s 57.10 (52.50 - 64.70, n=15)
1 clients total tok/s 54.16 (53.71 - 57.35, n=3)
# v0.1.44.1-poly.6
## log
strata-poly: loaded in 6.7 s; 6043 of 10496 routed experts in VRAM (10.7 GiB), 14 CPU threads
## bench
### without STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 73.20 (68.90 - 77.20, n=9)
chat first token (s) 0.34 (0.30 - 0.38, n=9)
4K prompt tok/s 790.70 (790.60 - 819.70, n=3)
4K first token (s) 5.09 (4.91 - 5.09, n=3)
16K prompt tok/s 845.90 (840.00 - 861.30, n=3)
16K first token (s) 19.00 (18.66 - 19.14, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (341.00 - 341.00, n=3)
agent turn first token (s) 5.61 (3.27 - 5.98, n=15)
agent decode tok/s 55.40 (49.70 - 65.90, n=15)
1 clients total tok/s 55.94 (53.97 - 59.44, n=3)
### with STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 72.50 (66.80 - 78.50, n=9)
chat first token (s) 0.31 (0.30 - 0.34, n=9)
4K prompt tok/s 814.50 (809.90 - 823.60, n=3)
4K first token (s) 4.94 (4.89 - 4.97, n=3)
16K prompt tok/s 861.10 (855.00 - 877.20, n=3)
16K first token (s) 18.67 (18.32 - 18.80, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (341.00 - 342.00, n=3)
agent turn first token (s) 5.49 (3.21 - 5.88, n=15)
agent decode tok/s 56.80 (52.20 - 65.30, n=15)
1 clients total tok/s 54.80 (53.11 - 58.54, n=3)
### v0.1.45
## log
strata-poly: loaded in 5.5 s; 6043 of 10496 routed experts in VRAM (10.7 GiB), 14 CPU threads
## bench
### without STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 74.90 (66.40 - 80.00, n=9)
chat first token (s) 0.32 (0.31 - 0.36, n=9)
4K prompt tok/s 802.60 (794.40 - 817.10, n=3)
4K first token (s) 5.01 (4.93 - 5.07, n=3)
16K prompt tok/s 848.00 (847.80 - 866.90, n=3)
16K first token (s) 18.96 (18.54 - 18.96, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (341.00 - 342.00, n=3)
agent turn first token (s) 5.56 (3.29 - 5.95, n=15)
agent decode tok/s 57.40 (49.70 - 64.70, n=15)
1 clients total tok/s 55.93 (55.83 - 58.91, n=3)
### with STRATA_SM75_PERF=1
Results (median, range over the runs):
chat decode tok/s 75.70 (68.50 - 79.40, n=9)
chat first token (s) 0.32 (0.30 - 0.38, n=9)
4K prompt tok/s 803.40 (793.20 - 816.70, n=3)
4K first token (s) 5.01 (4.93 - 5.07, n=3)
16K prompt tok/s 850.70 (846.60 - 869.60, n=3)
16K first token (s) 18.90 (18.48 - 18.99, n=3)
agent cache hit % 68.83 (68.83 - 68.83, n=3)
agent re-read tokens/session 341.00 (341.00 - 341.00, n=3)
agent turn first token (s) 5.60 (3.30 - 5.97, n=15)
agent decode tok/s 56.40 (49.50 - 65.20, n=15)
1 clients total tok/s 55.34 (53.74 - 58.16, n=3)
Thanks for running all of them, @inpos. That settles it for the T4: every version gives the same numbers within the ranges (chat decode 73-76 tok/s, 4K prompts 790-815, 16K 845-861), and STRATA_SM75_PERF=1 makes no difference on your card. The 55 you saw matches the benchmark's agent lines (55-57 tok/s), where each answer comes after a long conversation, so the chat page was most likely showing that kind of request.
@cinetube, the same benchmark on your machine would tell whether you're seeing the same thing: python polystrata.py benchmark --model <model name> (or BENCH.bat / ./bench.sh), plus your GPU, OS, the Qwen3.6 size, and the loaded in ... s; N of 10496 routed experts in VRAM line from strata-qwen36-<size>.log. I'll keep this open until then.
Activity
VecSzn commentedon Oct 18, 2026
Thanks for the report. I couldn't reproduce it on the cards I have: on an RTX 4070 laptop (sm_89) with Qwen3.6 UD-Q3_K_M, poly.5, poly.6 and 0.1.45-poly.1 all write at 72-81 tok/s. A T4 is sm_75, which neither my laptop nor my 5090 is, so I need a bit more from yours:
strata-qwen36-<size>.json. In particular: is"parallel"in it? Each batch slot takes VRAM from the expert cache as its conversation grows, which can slow a single chat on a 16 GB card.strata-qwen36-<size>.log) of each version, the lineloaded in ... s; N of 10496 routed experts in VRAM. If 0.1.45 holds fewer experts, that's most likely the cause.python polystrata.py benchmark --model <model name>(orBENCH.bat/./bench.sh).STRATA_SM75_PERF=1in the environment (or in the config's"env"). It turns on the Strata project's Turing kernels, which 0.1.45 added and leaves off by default.If you can, also try poly.6's engine: download
strata-poly-windows-x64-04beb8305ea6c2ff.zipfrom the 0.1.44.1-poly.6 release (or build that tag on Linux) and put it inengine-poly/instead, keeping a copy of the current one. That tells me whether the drop came with poly.6's parallel requests or with the 0.1.45 merge.The PCIe share by the copy kernel for every pack (simplexai-labs#31, s…
cinetube commentedon Oct 19, 2026
I am also getting poor performance. around 55 tps.
inpos commentedon Oct 19, 2026
I ran benchmarks for different versions, and the results are roughly the same. It seems something strange is going on in the web UI, as it was showing varying speeds.
inpos commentedon Oct 19, 2026
Yeah. I have same speed in agent over api in 0.1.55.
inpos commentedon Oct 19, 2026
I've tested agent over api in version 0.1.44.1-poly.5 and have the same result - 55 tok/s.
VecSzn commentedon Oct 19, 2026
Thanks for running all of them, @inpos. That settles it for the T4: every version gives the same numbers within the ranges (chat decode 73-76 tok/s, 4K prompts 790-815, 16K 845-861), and
STRATA_SM75_PERF=1makes no difference on your card. The 55 you saw matches the benchmark's agent lines (55-57 tok/s), where each answer comes after a long conversation, so the chat page was most likely showing that kind of request.@cinetube, the same benchmark on your machine would tell whether you're seeing the same thing:
python polystrata.py benchmark --model <model name>(orBENCH.bat/./bench.sh), plus your GPU, OS, the Qwen3.6 size, and theloaded in ... s; N of 10496 routed experts in VRAMline fromstrata-qwen36-<size>.log. I'll keep this open until then.