[Engine] Performance degradation among other issues on V0.2.1 #3526

Activity

1. yusheng227507 commented on Oct 21, 2026

   

   yusheng227507

   on Oct 21, 2026

   More actions

   No regression on a homogeneous 2x RTX 2080 Ti box (engine 0.1.45.3 -> 0.1.46.3), and the crash message has changed

   Reporting a data point that may narrow this down: on a box with two identical NVIDIA cards, going from 0.1.45.3 to 0.1.46.3 changed nothing measurable, and the answers did not change either. So the slowdown in this report looks specific to the setup rather than to 0.2.1 in general.

   Setup: 2x RTX 2080 Ti 22 GB (sm_75), NVLink NV2, PCIe 3.0 x8 host link (6.5 GB/s), driver 595.99.02, CUDA 12.8, Strata source checkout, engine 0.1.45.3 -> 0.1.46.3 (both compiled from source for sm_75, -DCMAKE_CUDA_ARCHITECTURES=75), i7-10700K, 94.1 GiB. Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, 262144 context, kv int8, --kv-resident 32768, --spec 4 --spec-min-p 0.5, --layer-split 26, --vram-reserve-mib 1500, 3 slots, MTP on, STRATA_SM75_PERF=1 + STRATA_BF16_TC=1 + STRATA_PA_V2T=1 in both versions.

   Speed (tools/strata_bench.py, 3 runs per test, medians, same bench and same settings on both engines):

   Test	engine 0.1.45.3	engine 0.1.46.3
   chat decode	73.8 tok/s	75.9 tok/s
   4K prompt	930.4 tok/s	930.9 tok/s
   16K prompt	1625.2 tok/s	1625.3 tok/s
   16K TTFT	9.91 s	9.91 s
   agent decode	76.9 tok/s	78.2 tok/s

   Answers (an 82-question suite: arithmetic, facts, code that is really executed, tool calls, long context, hallucination checks): 71/82 on both engines, and 78 of the 82 answers are byte-identical - the four that differ are worded differently with the same verdict.

   About the crash: the error in this report reads "the fused layout's MoE buffers are too small for layer 45". The current engine says more:

   prefill: the fused layout's MoE buffers are too small for layer 45's expert path at a chunk of N tokens
   

   so the chunk size is now part of the message. That backstop fires when a layer takes the MMQ or the FP16 path at a full chunk, which is a prefill-layout sizing drift (src/prefill/prefill.cpp, #583 / #954) - it is not tied to the quant or the card class as such, but a mixed card pair (RTX 5080 + RTX 2080) is exactly the kind of setup where different layers take different paths. I have not seen it on a homogeneous pair.

   If the report is still reproducible on 0.1.46.3, the new message (with the chunk size) and a STRATA_VERIFY_PROFILE=1 line would point at the layer that drifts; and the prefill speed in the report (1200 -> 500 tok/s) is worth comparing against the same 4K/16K prompt on a single card pair, since a mixed pair also splits the layers unevenly.

   For reference, the standard-protocol runs of the same box on 0.1.46.3 are in PR #3535.
2. Thenenea commented on Oct 21, 2026

   

   Thenenea

   on Oct 21, 2026

   More actions

   Dual Tesla P4 with I9 13900K 16t (P cores) and 42Gb DDR4 3200 allocated to VM. Performance decreased on 0.1.46 versus 0.1.45. In both cases, custom image built with the architecture specific requirements. Longer report attached.

   onyx_0.1.46-sm61_comparison (2).pdf

   Summary:

   Short prompt (seed)

   • 0.1.45-sm61: decode 26.8 t/s (21.3–32.7), TTFT 8.38 s, prefill 50.3 t/s, pass 2/8
   • 0.1.46-sm61: decode 23.1 t/s (17.4–31.1), TTFT 6.90 s, prefill 42.6 t/s, pass 4/20
   • same engine, 3 boots of 0.1.46: decode spread only 0.5% — the engine is stable boot to boot
   • 0.1.46 vs 0.1.45: decode −14.0% lower, TTFT −17.7%, prefill −15.1%

   Long prompt (endurance)

   • 0.1.45: decode 34.5 t/s (34.4–34.6), prefill 96.7 t/s, pass 0/6
   • 0.1.46: decode 31.2 t/s (26.2–34.3), prefill 108.0 t/s, pass 0/6
   • same engine, 2 boots: spread 3.8%
   • 0.1.46 vs 0.1.45: decode −9.7% lower, prefill +11.7% higher

   MTP and reuse

   • 0.1.45: 2,861 of 3,312 drafts accepted (86.4%)
   • 0.1.46: 5,975 of 7,291 accepted (82.0%)
3. ashdahor2-design commented on Oct 24, 2026

   

   ashdahor2-design

   on Oct 24, 2026

   More actions

   Hi Strata developers!

   I would like to add my own findings to this performance regression report. I am experiencing a substantial generation speed decrease after upgrading Strata, along with an issue where the desktop application does not preserve the selected prediction mode.

   System configuration

   • Operating system: Windows 11
   • CPU: Intel Core i7-14700KF
   • GPU: NVIDIA GeForce RTX 5060 Ti (Blackwell), 16 GB VRAM
   • System memory: 128 GB DDR4 RAM
   • Model: Qwen3.8-Flash-Next (Unsloth)
   • Quantizations: UD-Q4_K_XL and IQ3_S
   • Previous desktop version: Strata 0.2.2
   • Current desktop version: Strata 0.2.3
   • Current engine version: 0.1.48
   • Workload: Extended autonomous software development, including reading files, generating code, running commands and executing tests

   1. Significant performance degradation

   Before the update, I observed the following generation speeds:

   UD-Q4_K_XL:

   • Previous performance: approximately 35–50 tokens/s
   • Current performance: approximately 15–21 tokens/s
   • Recent extended coding session: approximately 20.1 tokens/s reported at the end of a 73-minute task

   IQ3_S:

   • Previous observed range: 36.9–52 tokens/s
   • Previous typical performance: approximately 48–52 tokens/s
   • Current performance: approximately 36–39 tokens/s

   The slowdown is especially severe with UD-Q4_K_XL.

   I previously had a very good experience running this quantization on my hardware. Now it performs considerably slower during similar software development workflows.

   I have also tested 256K and 512K context configurations and did not observe a meaningful performance difference between these two settings in my tests.

   Unfortunately, I no longer have screenshots showing the older performance measurements. However, I can provide screenshots of the current speeds and diagnostic information.

   These numbers are observations from real coding sessions, not a controlled benchmark using identical prompts on both versions. I do not want to claim that a regression has been conclusively proven, but the difference is significant enough to investigate.

   2. Prediction setting repeatedly resets to Auto

   I have reproduced another issue in Strata 0.2.3.

   Steps to reproduce:

   1. Open the desktop application's model settings.
   2. Change the prediction setting to "Off".
   3. Load the model.
   4. Reopen the model settings.
   5. The prediction setting has returned to "Auto".

   This happens repeatedly, even when I explicitly select "Off".

   I also tried manually selecting MTP, but the setting did not reliably persist in the desktop interface.

   As an alternative, I installed Strata Terminal and successfully used the following command to request a launch without MTP:

   strata.cmd launch --family unsloth --model ud-q4_k_xl --context 512k --drafting off

   The Terminal accepts the configuration, but I cannot reliably preserve the equivalent "Off" selection in the desktop UI.

   I would appreciate clarification on whether the desktop UI is only displaying an incorrect saved value or whether the actual inference configuration is also being overridden.

   3. Stability improved after the update

   There is also a positive change worth mentioning.

   On Strata 0.2.2, I repeatedly experienced autonomous agent sessions being interrupted by a four-minute inactivity timeout.

   After updating to 0.2.3, my agents have been able to work for over an hour and complete substantial coding tasks without those interruptions.

   This improvement is very welcome. I would like to keep the new stability improvements while restoring the previous generation performance.

   4. Request for investigation

   Could you please investigate the following?

   • Is there a possible performance regression affecting the RTX 5060 Ti 16 GB, particularly with UD-Q4_K_XL?
   • Have there been changes in expert caching, CPU/GPU memory placement, or prediction behavior that could explain the slowdown?
   • Why does the desktop application repeatedly reset the prediction setting to Auto?
   • Is there a recommended configuration for this specific GPU and 128 GB RAM setup that could restore the previous generation speed?
   • Which diagnostic logs or benchmark results would be most useful to identify the cause?

   I can provide current screenshots, diagnostic logs and additional test results if needed.

   I really appreciate the work being done on Strata. Running such a large model locally on consumer hardware is impressive, and I would love to help identify and resolve these issues.

   Thank you for your time and continued development!
4. ashdahor2-design commented on Oct 24, 2026

   

   ashdahor2-design

   on Oct 24, 2026

   More actions

   Additional finding: NVFP4 became faster, while UD-Q4_K_XL and IQ3_S became slower

   Hi again!

   I noticed another important detail in my performance measurements that may help narrow down the cause of the regression.

   On the same Windows 11 PC with an RTX 5060 Ti 16 GB and 128 GB RAM, different quantizations appear to behave very differently after the Strata update.

   Here are my observed results:

   Quantization	Strata 0.2.2	Strata 0.2.3
   UD-Q4_K_XL	Approximately 35–50 tokens/s	Approximately 15–21 tokens/s
   IQ3_S	Typically 48–52 tokens/s, minimum observed 36.9	Approximately 36–39 tokens/s
   NVFP4	Approximately 17–22 tokens/s	Approximately 24–28 tokens/s

   The interesting part is that NVFP4 actually became faster, while the other two quantizations became slower.

   My latest NVFP4 coding session is running at approximately 24–26 tokens/s.

   This makes me wonder whether the issue is related to quantization-specific inference paths, GPU expert caching, memory placement, or speculative decoding behavior rather than a general performance regression affecting all models.

   Please note that these are real-world observations from different coding workloads, not controlled benchmarks. The context sizes and prediction configurations were not identical across all runs, so I cannot attribute these differences exclusively to the Strata update.

   I also suspect that MTP may not always be functioning as expected, even when the desktop interface displays Auto or MTP. This is only a hypothesis, not a confirmed finding.

   Could you please check whether changes between versions 0.2.2 and 0.2.3 might affect UD-Q4_K_XL, IQ3_S, and NVFP4 differently?

   I thought this additional comparison might be useful for identifying the root cause.

   Thank you again!
5. Niko1221 commented on Oct 26, 2026

   

   Niko1221

   on Oct 26, 2026

   Owner

   More actions

   Thanks. This report mixes several problems, which makes it hard to act on. Please retest on 0.2.6 and, if it still happens, open one issue per problem with: the exact engine start line, the engine log from start to the failure, 'strata --version', the layer split used, and for speed the STRATA_DECODE_TIMING=1 line. For the MoE-buffer crash the log line with the chunk size and STRATA_VERIFY_PROFILE=1 output would show which layer takes a different path on the 5080+2080 pair. For the settings that the app does not keep (vision off, prediction mode, GPU choice) a separate issue with the app version is best. We could not reproduce a slowdown on a two-identical-card setup.
6. Niko1221

   closed this as not plannedon Oct 26, 2026
7. gusmanb commented on Oct 26, 2026

   

   gusmanb

   on Oct 26, 2026

   Author

   More actions

   @Niko1221 Thanks for the reply. I will not open any new bug, I'm really happy to report that with 1.50 everything works as expected and even better, my prefill has gone from 1200 t/s up to 2.925 t/s, a massive speed up and the generation went back to the normal ~65 t/s.

   Just a comment, having an old version with the new ones seems to mess up things a lot, I had living 1.46 with 2.6.0 (terminal only) in different folders and 2.6.0 was changing files from 1.46 even when there was no link between the installations. I did a full cleanup of everything except the model files and now all is working properly.

   Thanks :)