Repository navigation
Strata serves Qwen3.8-Flash-Next with strange behavior - really bad results #2685
Description
Strata serves Qwen3.8-Flash-Next dumber than llama.cpp: tool-call loops + engine nondeterminism
Hardware: RTX 5070 12 GB, 60 GB RAM (only one 46 GB instance fits at a time), driver 595.99.02, CUDA 13.2.
Pack: IQ3_S (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:IQ3_S), Strata native pack.
Reference server: llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:IQ3_S -c 110000 -np 1 -fa on -ctk q8_0 -ctv q8_0 -b 4096 -ub 1024 --load-mode none --no-mmproj --port 8080 --cpu-moe --moe-cache-mib 2048
1. Observed behavior (the complaint)
Same model, same prompts, two servers:
- Via Strata: tool calls fail most of the time (malformed args, hallucinated tools:
Bash/exec/WebFetch/registry/search, OpenCode-style args likefile_path,limit/offset
that don't exist in the offered schema). Failures trigger agent retry loops. Also "dummy loops":
very short thinks followed by the same command over and over (list_filesx3-x5,ls -lax5). - Via llama.cpp (
llama serve, same quant): noticeably better reasoning, working tool calls. - Result: Strata not usable as daily driver.
2. Issues found AND fixed (config)
2a. Greedy-by-default sampling. strata-iq3_s.json had no sampling block, and
serve/server.py only forwards temperature>0 / top_p<1 etc, so the engine ran greedy
(temperature 0). Greedy decoding is the classic cause of repetition loops.
Fix applied to strata-iq3_s.json (backup strata-iq3_s.json.pre-fix.bak):
sampling {temperature 0.6, top_p 0.95, top_k 20}.
2b. tool_call_recovery off. Absent from config = False (server.py:3102,6864;
docs/DETAILS.md:845 quotes 92.9% vs 98.3% tool-call success on 1462 turns).
Fix applied: tool_call_recovery true.
Parser proof (GPU-free): synthetic 5-shape check 1/5 recovered without the flag, 5/5 with it;
serve.test_server.ToolCallRecovery* 12/12 pass.
3. Agentic evals (controlled, same harness)
Harness (evidence/dailydriver*.py): agentic loop over a slide-generator repo
(/home/gustavo/Testes/lol/slide_test: ppt.js, deck_claude.js, styles.css, pdf_text.txt,
binary pptx/pdf), tools list_files/read_file/grep, full call trace, keyword scoring.
- v1 (5 runs/arm, max_tokens 1024): 0/10 summaries, mostly
finish=length(harness flaw: 1024
starves xhigh thinking). - v2 (max_tokens 4096, 8 turns, 5 runs/arm): old-greedy arm loops 4/5, fixed arm loops 2/5 +
hallucinations 2/5. Both arms 0/10 task completions. - Same harness vs llama.cpp (arm C, port 8083): 5/5 clean, 14-15/15 valid calls, zero loops.
- v3 (fixed config, temp 0.6, 3 runs): 0/3 usable — same disease (list_files loops, binary reads,
hallucinatedwrite_file). Config fix did NOT cure it. - Ruled out prompt rendering: Strata
chat_template.jinjabyte-identical to llama.cpp/props
template (183 lines, empty diff); Strata-rendered vs llama-tokenized prompt ids IDENTICAL
(516/516 turn 1, 562/562 turn 2, tools block included). Difference is engine-side (logits)
and/or sampling RNG (harness sendsseed; Strata seed-honoring untested). - Single-request continuation test: Strata 1/6 full reads + Bash hallucination + partials;
llama 6/6 correct progress. - Near-greedy (temp 0.01, seeds 111x2+222): llama 3/3 bit-identical; Strata nondeterministic
with the same seed. <-- this redirected the whole investigation to the engine.
4. Engine nondeterminism hunt (the deep part)
Method (evidence/run_determ.sh): canonical engine args (pack iq3_s, native+ple GGUFs,
expert-profile.bin, cache auto, prefill auto, spec 4, mtp, max-ctx 110000, kv int8, pcie-frac 0,
spec-min-p 0.75, vram-reserve 896) + --tokens-file prompt609.txt (609 ids) + --max-new 24 +
--window-hashes. Two fresh processes, compare hashes. Baseline: DIFFER from prefill-tail
window 0 (residual hash 2df94610 vs 8860bf2c, argmax 9764 vs 40; 20 vs 26 windows).
Still differed after each of (one factor at a time): MTP removal, --pool-workers 1, reasoning
budget, --prompt-cache 0, STRATA_ROUTE_TAIL_SKIP=0, --no-coupled-draft,
STRATA_PREFILL_CPU_SHARE=0, --sync-every-layer, --adapt-every 0,
--expert-cache-cpu-order, --ple-io ram, --no-host-worker, STRATA_PREFILL_ISSUER=0,
STRATA_PF_STEP_SYNC=1, --pcie-frac 0 (partial: window-0 argmax identical, diverges token 3).
--dump-routing: records 0-2 identical, record 3 differs. --dump-logits writes empty files
(broken); --dump-residual all zeros; --dump-layers OOMs (~1.2 GB dump buffers).
Per-layer ground truth via the research hook STRATA_DBG_FEAT (+_LAYERS, prefill.cpp:2948;
parser evidence/featparse.py): layers 0 AND 1 bit-identical (ids, weights, hidden states),
layer 2+ differ hugely (~1.5M/1.55M values, magnitudes ~0.9-1.7). Onset varies run to run
(layer 1 vs 2). Layer-1 MoE fp32 input bit-identical. Decisive: STRATA_PREFILL_MMQ=0 runs are
bit-IDENTICAL (hashes byte-equal, streamed/resident counts exactly equal) => the divergence
is inside the MMQ expert-compute path; attention/routing/tokenizer/server/sampling cleared.
Cleared by experiment: same-process back-to-back greedy server requests DIFFER (live race, not
just process state); VRAM-poison P0-vs-P1-vs-P0b all differ (not uninitialized memory);
CUDA_LAUNCH_BLOCKING=1 still diverges at layer 2 (not a GPU overlap race; prefill completes,
then hangs at verify capture — known issue #1341); CPU-share=0 still diverges (pool truly idle:
PoolHold gating, cpu_maybe); CLB+SHARE=0 still diverges (both jointly cleared);
stream_all-forced path still diverges (STAGE-ring cleared); tiny prompts T=2,4,8 all diverge
(shape theory dead); near-all-streamed (--expert-cache 32 + SHARE=0) still diverges;
no-MTP (--spec 2, +0 MiB for the draft head) still diverges (draft cleared).
Cleared by audit: staging-slot protocol, upload rings (all single-stream m.cs), host grouping
tables (per-layer full sync), stager + file sources (thread-safe libaio/OVERLAPPED),
static residency (host_res built once at startup; full 48x512 table verified identical),
no float atomics anywhere, combine (same call both modes), vanilla unmodified llama.cpp MMQ
headers (third_party git-clean), deterministic J-loop / stream-K / launch selection.
5. The paradox (where the hunt stands)
Custom uncommitted debug patch STRATA_DBG_MOE_TENSORS (prefill.cpp ~4082) dumps layer-1
Xq/GU/Dm/bo + gathered blobs + bounds + cnt/off + residency + row map. Across runs, EVERY
gate/up-product input is bit-identical (activations, all 16 group blob slots, device bounds
nbound=957, counts, offsets, static residency, ids 0..N) yet gate/up products differ on ALL
6080 rows (resident and streamed experts alike). Standalone replay harness
(evidence/replay.cpp): the kernel is deterministic in isolation (15/15 identical) — but its
bits don't match the engine's (fidelity gap; the pack's actual gate/up type value is unverified).
compute-sanitizer synccheck hung at the #1341 verify capture and was killed (inconclusive).
6. What persists / next steps
- Root cause UNKNOWN. Air-tight paradox: identical inputs + deterministic kernels + different
outputs is logically impossible, so one verified claim must be false (fidelity gap in what the
engine actually launches) or the sm_120 kernel build misbehaves. Next: close the replay
fidelity gap (actualgu_type, launch params), then per-group launch-param dump. - Workaround with evidence:
STRATA_PREFILL_MMQ=0is the only flag that gave bit-identical
runs. Agent-quality validation (3x repo test, fixed config + MMQ=0) was running when this was
written — seeevidence/dailydriver4.jsonl/dailydriver4.log, results appended below. - Still owed regardless: greedy temp-0 3x test (workaround check), Strata seed-honoring check.
Appendix A. Validation: MMQ=0 as daily-driver workaround
Apples-to-apples with v3 (B-fixed config, temp 0.6, same repo/task/harness, 3 runs), plus
STRATA_PREFILL_MMQ=0 in the server environment (evidence/dailydriver4.py).
Results (runs completed 2026-10-15)
| run | turns | tool calls | executed | loop | empty turns | think chars | SUMMARY | kw/10 |
|---|---|---|---|---|---|---|---|---|
| 0 | 8 | 14 | 14 | no | 0 | 1785 | none (still exploring at turn cap) | 0 |
| 1 | 8 | 14 | 14 | no | 0 | 1625 | none (still exploring at turn cap) | 0 |
| 2 | 8 | 13 | 13 | no | 0 | 1740 | none (still exploring at turn cap) | 0 |
All 41 tool calls executed (no parse failures, no hallucinations: no Bash/exec/
WebFetch/registry, no OpenCode-style file_path/limit/offset args). Every run did
productive exploration — list_files, then read_file of all four real repo files
(deck_claude.js, ppt.js, pdf_text.txt, styles.css) plus targeted greps
(writeFile, SLIDE \d+, FONT_*, MASTER_, ...). Zero dummy loops, zero empty turns.
Verdict: MMQ=0 cures the loop/hallucination disease (v3 with MMQ on: 0/3, dummy
list_files loops + hallucinated tools/args). But task completion is still 0/3 by the
harness metric: no run produced a SUMMARY within the 8-turn cap (all were still gathering).
Caveats: (a) server stdout/stderr went to DEVNULL, so no prefill-timing numbers and MMQ=0
is confirmed only via env inheritance (the same mechanism as the bit-identical NOMMQ engine
runs); (b) 8 turns may be too tight — the MMQ=0 runs spent their turns reading instead of
looping, so a longer cap or a nudge-to-summarize might complete; (c) determinism does not
imply quality — this validates stability, not that summaries are good.
Appendix B. Evidence files in this repo
evidence/run_determ.sh— engine A/B harness (canonical BASE args)evidence/prompt609.txt— the 609-token prompt (comma-separated ids)evidence/featparse.py— FEAT dump parser (proves layer-0/1 identical, 2+ differ)evidence/dailydriver2.py,dailydriver3.py,dailydriver4.py— agentic-loop harnesses
(+.jsonltraces,.logoutputs)evidence/replay.cpp— standalone MMQ-kernel replay (15/15 deterministic in isolation)evidence/armB.json— fixed server config used by the harnessesevidence/NOMMQa.txt,evidence/NOMMQb.txt— the byte-identical MMQ=0 window hashesevidence/W1.txt,evidence/W2.txt— baseline differing window hashesevidence/sp_req.py,evidence/poison.py— same-process probe, VRAM-poison tool
dailydriver4.log
TINY8b.txt
TINY8a.txt
N2.txt
N1.txt
W2.txt
W1.txt
NOMMQb.txt
NOMMQa.txt
armB.json
replay.cpp
poison.py
sp_req.py
prompt609.txt
dailydriver4.py
dailydriver3.py
dailydriver2.py
featparse.py
run_determ.sh
Activity
LeGeRyChEeSe commentedon Oct 15, 2026
A data point from a different card, in case it helps narrow this down: on an RTX 4070 SUPER (sm_89, Windows 11, driver 617.42, CUDA 13.1 build of v0.1.44, IQ2_XS), the MMQ path is bit-reproducible once the prefill CPU share is off.
Setup: the engine started directly with my server's arguments (
--expert-cache auto --prefill auto --spec 4 --mtp ... --kv int8 --kv-resident 32768 --pcie-frac 0.55 --ple-io direct), a 965-token public prompt (the start of Strata's README), greedy,--max-new 32,--window-hashes, one fresh process per run:STRATA_PREFILL_MMQ=0STRATA_PREFILL_CPU_SHARE=0STRATA_PREFILL_MMQ=0+STRATA_PREFILL_CPU_SHARE=0So here the run-to-run variation comes from the CPU share (on by default for chunks under 1,024 tokens, and its split is timed per layer), not from MMQ;
MMQ=0alone does not make the runs identical on this card. You found the opposite on sm_120 (CPU_SHARE=0still diverges,MMQ=0fixes it), which fits your guess that the difference is specific to the sm_120 build or kernels.Quality on this card, for what it's worth: a small public edit bench (30 mutated HumanEval solutions, the model reads the file, edits it and runs the tests through tool calls, 3 seeds, T 0.6) passes 90/90 with the default MMQ path, no malformed or invented tool calls.
s1arsky commentedon Oct 16, 2026
Third card, third pattern: RTX 3090 (sm_86), Linux, v0.1.43.1, IQ3_S — the prompt side is bit-reproducible here, and none of the MMQ / CPU-share / prompt-cache switches touch the divergence.
Env (observed): RTX 3090, 64 GB RAM, Fedora-based Linux, engine built from source (
BUILD.json: source:local, archs:[86], host compiler pinned gcc-15), same pack (ISTA-DASLab … GSQ-RCO … IQ3_S). Server run with--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <draft> --max-context 262144 --kv int8 --vram-reserve-mib 1074 --conversation-cache-mib 6144 --conversation-cache-slots 1+STRATA_ADAPT_OVERLAP=1(this env var appears in neither of your two posts — my all-off arm below includes it off, so it is covered once either way).Method (server-level, simpler than yours): 3 byte-identical requests against the running OpenAI endpoint (
temperature: 0, seed: 4242, reasoning_effort: low, fixed ~1036-token prompt), sha256 over the full assistant message (content + reasoning). Depth ladder =max_tokens8/16/32/64/128/512. Every arm is live-verified: env arms from the engine process's own/procenviron, arg arms from its spawn cmdline. I burned ~4 batches early on because stamps never reached the spawned process — read state from the process, not the config file, and skip my mistake.Observed ("3 variants" = 3 different hashes at 512 tokens):
STRATA_PREFILL_MMQ=0alone (your decisive flag)--prompt-cache 0(log line reads0 checkpoints)STRATA_ROUTE_TAIL_SKIP=0STRATA_ROUTE_TAIL_SKIP=0+--spec 2MMQ=0+CPU_SHARE=0+ADAPT_OVERLAP=0(combined; all three live in engine environ)What it looks like from here: divergence onset sits between the 65th and 128th generated token — at a 1036-token prompt the prompt result is reproducible through at least 64 greedy steps, so your window-0 paradox does not reproduce on this Ampere card. What I do have is decode-side amplification that survives every switch that fixed or changed the picture on your sm_120 and on the sm_89 comment. One wrinkle for your attractor question: only a few output hashes recur across separate batches and engines — attractors, not uniform noise — consistent with near-tie argmax flips amplifying sub-ULP decode-path differences (hypothesis, mechanism unproven here).
No agent-quality collapse alongside it: heavy agentic use at temp 1.0 (config defaults per the model card), tool calls parse, no loop disease, decode 85–100 tok/s — so the determinism failure here looks benign so far, which is itself a data point about how the sm_120 failure might be prefill-specific.
Not tested here (please don't over-read my negatives): engine-level
--tokens-file+--window-hashes,--pool-workers/--no-host-worker/--sync-every-layer/--ple-io rametc. (your matrix targets a prefill failure that looks absent on this box),CUDA_LAUNCH_BLOCKING,CPU_SHARE=0alone, and v0.1.44.x — I am on v0.1.43.1.Offer: I can run the engine-level
--window-hashesladder here (server stopped, engine direct) on 0.1.43.1 and/or after bumping to v0.1.44.1, and/or paste the exact probe script — say the word if an sm_86 window-hash trace helps triangulate.serve: a request with no sampling gets the model card's (thinking 1.0…
web: the page sends no sampling until the user changes it, so the ser…
William-AIdev commentedon Oct 17, 2026
RTX 5090: tool-calling failures with the CUDA 13.2.1 build, MMQ workaround, and CUDA 13.2.2 follow-up
I encountered intermittent tool-calling failures while running Qwen3.8-Flash-Next IQ3_S on an RTX 5090. Disabling MMQ mitigated the failures on my older build. After rebuilding the same source with CUDA 13.2.2, neither MMQ setting reproduced those tool-calling failures in the tests below. I now use the new build with MMQ enabled and a 256K context limit.
Environment
296b59d2f73c8f04a805df29c2fa97655adace78.STRATA_IQ_MT_MIN=1was retained in the MMQ comparisons. Requests used temperature 0 and disabled thinking.1. Old build: MMQ-off mitigated the tool-calling failures
With MMQ enabled, the model sometimes called tools that were not offered, or returned prose instead of making the requested tool call. Setting
STRATA_IQ_MT_MIN=1alone did not reliably resolve this.A six-start comparison alternated the two MMQ settings. Each setting had three independent engine starts, with 40 tool-call checks per start. Checks covered the tool name, required JSON arguments, and expected city/date values.
A separate small multi-turn comparison completed 3/6 tasks with MMQ enabled and 6/6 with MMQ disabled (one start per setting).
I therefore used
STRATA_PREFILL_MMQ=0as a workaround. These observations support mitigation of this particular failure pattern, not a claim that disabling MMQ fixes every generation error. Repeated requests within a start are clustered observations, not independent startup trials.2. CUDA 13.2.2 build: the original tool-calling failures were not reproduced with either setting
After rebuilding the same source with CUDA 13.2.2, all three numerical test entries passed without skips:
iq_parity_fixtures,iq_parity, andprefill_fused_iq_test.I then ran two rounds of end-to-end comparison. Each round used four independent starts ordered MMQ off, on, on, off, keeping the 262,144-token context limit and other settings unchanged within the round.
Neither setting reproduced the earlier invalid-tool-name / prose-instead-of-call failures in these probes. No corrective retries were used.
The one incomplete task was a different failure: the model correctly called the weather tool, but its final answer shortened the synthetic verification code
WX-SH-7319toWX-SH-7. The response ended withfinish_reason=stop, at 42 completion tokens against a 384-token limit. Its cause is unknown; it should not be omitted or treated as evidence that all tasks are now reliable.3. MMQ performance comparison on the new build
In the latest round, each setting had six measurements per input length, split across two independent starts. Inputs were identical across settings, and all measured prefill requests reported zero reused tokens. Values below are median prefill times.
Coarse NVIDIA telemetry during the 65K MMQ-on prefill showed PCIe Gen5 x8 and receive throughput around 30 GB/s. This suggests that transfer bandwidth may be a material constraint, but there was no x16 comparison or profiler trace establishing the bottleneck.
An earlier new-build comparison showed nearly identical MMQ-on/off timings. The reason for that discrepancy has not been established; the latest result should not be presented as universally reproducible.
4. Official benchmark with the selected daily-use configuration
I also ran the official
tools/strata_bench.pysuite with three runs per test and--long, following the official calibration. The suite completed successfully. This section is a speed benchmark, not a correctness evaluation.Selected configuration (machine-specific paths omitted):
CPU expert workers remained at the engine default of 7. Model, pack, tokenizer, MTP, and expert-profile paths are omitted for privacy.
All fresh-input tests reported zero cache reuse. Agent follow-up cache hit rate was approximately 68.6%; median historical tokens reread per session was 74. Speed uses engine timings; time to first token is measured by the HTTP client. Model loading is excluded. Concurrency was not tested, and the maximum input in this official suite was about 32K, not a full 256K.
Calibration selected the parameters above, but the full benchmark was essentially unchanged from the previous configuration (
pcie-frac=0,spec-min-p=0.70, default adaptive tier): short-chat decode 157.6 vs 154.9 token/s, agent decode 179.0 vs 178.5, and 32K prefill 6,493.0 vs 6,482.7. I do not interpret the calibration result as a demonstrated overall speedup. The MMQ A/B table and this official benchmark use different prompt workloads and should not be compared as if they were the same test.Interpretation and limitations
The progression on this machine was:
This does not establish the original root cause. The compiler and CUDA libraries changed together, the machine underwent GPU fault recovery between historical phases, and the old build was not retested in a controlled failing state against the new build. There was no direct GPU kernel tracing. The old and new experimental campaigns also did not all use the same context capacity. These are bounded test results, not proof of deterministic output or general agent reliability.
Dhrhciebcy commentedon Oct 17, 2026
Same.
Since I updated Strata to v0.1.44, I’ve had the impression that Qwen3.8-Flash-Next IQ3_S has become noticeably dumber; it’s been inconsistent with its own answers—to the point where it would sometimes even notice this itself and issue a correction in the following message (always if the correction it claimed to have made was actually real...). I rolled back to v0.1.41 now, and its answers seem much more consistent; however, I’m finding hard to trust the model again after what happened, so I’m not sure if I’ll update Strata again—at least not until this issue maybe is officially closed and resolved.
For context, I’m running this on a system with an RTX 5090 mobile (24 GB VRAM) and 64 GB of RAM, NVIDIA Driver Version: 591.74, CUDA Version: 13.1
I really hope the developers manage to fix this issue because, apart from that, Strata has literally changed my life. Really a wonderful project!
merge upstream v0.1.45 into feature/sampling-presets-1129
setup, serve: the model card's sampling presets, opt-in (Niko1221#1129)
Merge upstream main (477ac13, 0.1.45): the server, setup, docs and up…
serve: sampling のない要求にモデルカードの値、貪欲は頼んだときだけ（Niko1221#2685、upstream a8091f4
web: サンプリングの設定を変えるまで画面は値を送らず、サーバーのモデルカードの既定を使う（Niko1221#2685、upstream e…
serve: a request with no sampling gets the model card's (thinking 1.0…
web: the page sends no sampling until the user changes it, so the ser…
Niko1221 commentedon Oct 19, 2026
Thanks for taking the time to report this.
We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.
serve: temperature 0 stays greedy - no model-card penalties on a gree…
feat(server): model-card sampling defaults, several API keys, toleran…
SONNET-SAMPLING-Temperatura 0 esplicita resta greedy puro senza sched…