Strata v0.1.43 returns an empty final response when a specific string appears in the model's thinking output

I encountered a bug in Strata v0.1.43. When the model's thinking output reaches the point where it would produce the string </tool_call>, the thinking output often appears to be truncated immediately before that string. After the client returns an empty final assistant response.

The client does not crash, and the chat session remains usable. The client returns to the user input state, and the chat can continue if the user sends another message.

I initially thought an MCP server was involved, but the issue can be reproduced without an MCP server.

A prompt for reproduction (IQ3_S):

This is an experiment to find bugs in the client.
Please output the following string verbatim: </tool_call>.

Expected:
The client should display the requested string as the final response.

Actual:
The final assistant response is empty. The chat returns to the user input state.

Log during reproduction:

[strata] reading the prompt: 70 of 75 tokens, 4 s so far
[strata] thinking: 35 of max 262061 tokens, 47.7 tok/s, 5 s
[strata] done: 39 tokens in 5 s (48.1 tok/s) (stop, cancel=False), expert cache 45.5% hit (+29.2% of the routed experts over PCIe)

Environment:

Strata: 0.1.43 (IQ3_S)
OS: Windows 11 Pro
CPU: AMD Ryzen 7 5700X
GPU: AMD Radeon RX 9060 XT (16 GB)
RAM: 64 GB DDR4

Reproduction rate:
Consistently reproduced after 10+ attempts.

Additional notes:

I thought [Engine] serve: a tool call whose </tool_call> is missing ends at its own </function> #1881 might be related, so I tried implementing the changes, but the issue still occurs.

please let me know how this does
#1886

and if it doesn't do what you want please give me logs and tell me what you want </tool_call>. to do instead. i.e show failure and give a manual fixed example and i should be able to handle the regression from there

I tried applying #1886, but the situation remained unchanged.

Here’s what I did:

Launched Strata using run-iq3_s.bat
Entered This is an experiment to find bugs in the client. Please output the following string verbatim: </tool_call>. into the “Ask anything...” area in the Web UI

Result:
No response from the model. The expected response is </tool_call>.

This is the logs from startup:

[strata] 2026-10-12 12:52:50 engine started: C:\Users\kntroh\root\kntroh\LLM\Strata\engine\strata.exe --serve --pack C:\Users\kntroh\root\kntroh\LLM\Strata-data\packs\iq3_s --native C:\Users\kntroh\root\kntroh\LLM\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf C:\Users\kntroh\root\kntroh\LLM\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile C:\Users\kntroh\root\kntroh\LLM\Strata\data\expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp C:\Users\kntroh\root\kntroh\LLM\Strata-data\mtp\rt --max-context 262144 --kv int8 --control-vector-scaled C:\Users\kntroh\root\kntroh\LLM\Strata\data\experimental-speed-projection\Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer
strata: gfx12 (RDNA4): 5 exact speed switches on by default (STRATA_GFX12_DEFAULTS=0 turns them off; a switch you set is kept): GDN_HEAD GDN_PP GDN_CONVL2 GDN_NOY CVEC_FUSE
strata generate: memory guard: limit 63.9 GiB (source: meminfo), current 8.6 GiB, reclaimable cache credited 0.0 GiB; available 55.3 GiB (MemAvailable 55.3 GiB), floor 2560 MiB
strata generate: PCIe probe: 28.4 GB/s host->device (best of 26.3 28.4 28.4 28.4) -> pcie_frac 0.55 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: C:\Users\kntroh\root\kntroh\LLM\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.3 s)
strata generate: 1466 MiB of weights loaded from C:\Users\kntroh\root\kntroh\LLM\Strata-data\packs\iq3_s in 0.3 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 5.1 s
strata generate: control vector mode = project, dir = per-layer, layers 4..44 (41 steered)
strata generate: PLE direct read probe: 165314 rows/s (fine)
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile C:\Users\kntroh\root\kntroh\LLM\Strata\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: PLE on, table 320001536 rows of C:\Users\kntroh\root\kntroh\LLM\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata mtp: draft layer loaded, 1098 MiB of VRAM (experts 675, dense 111), files read in 0.25 s (3088 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes, in 1.0 s
strata generate: GPU 0: AMD Radeon RX 9060 XT (gfx1200)
strata generate: hint: STRATA_HIP_WMMA=1 reads prompts about 30% faster on this card with --kv int8 (prompt attention on the matrix cores); it is off by default because the output bits change (last-place rounding). Nothing was enabled.
strata generate: HIP runtime C:\Users\kntroh\root\kntroh\LLM\Strata\engine\amdhip64_7.dll
strata generate: expert arena read unbuffered (2 of 16 probe reads from the file cache; 7.4 GiB available, 51.1 GiB of files)
strata generate: expert arena: locked 1302 MiB via working-set minimum + VirtualLock; cudaHostRegister of the whole arena FAILED (invalid argument); 47 slices pinned (45 GiB); large pages refused for 50295996416 B (GetLargePageMinimum=2097152, VirtualAlloc error 1314); using 4 KB pages
strata generate: loaded 46.84 GiB at 5.40 GiB/s
strata generate: CPU pool tasks/phase: 24 (automatic), participating threads: 8
strata generate: 7 pool workers on logical processors 2,4,6,8,10,12,14, the host thread on 0 (draining too) (--host-core first)
strata aux cpus: not available on this system
strata: Windows budgets 15447 of this card's 16304 MiB for this process; free VRAM is counted within that (STRATA_WDDM_BUDGET=0: off)
strata generate: expert cache auto: Windows keeps 2560 MiB free after the cache on a card this size (the default 700 MiB left 323 MiB on a 20 GB card and Windows paged, #1376); --vram-reserve-mib N sets it
strata generate: expert cache auto: 6.26 GiB free, 2560 MiB reserved (+218 MiB for the draft head) -> 1430 slots
strata generate: only 1696 MiB free once the slots are written (reserve 2560 MiB); shrinking the expert cache
strata generate: expert cache 1367 slots, 2.64 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 1367 of 1367 slots from the profile in 0.4 s (6718 MB/s); slot 0 verified
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.43)
strata generate: token graph hit path: 1367 resident experts, decided on the device
strata serve: Windows commit: 28.3 GiB left of a 109.9 GiB limit (RAM + page file)
strata serve: prompt chunk auto: 4608 tokens, a 199-slot ring
strata serve: the prompt path borrows 1216 CUDA0 cache slots (2.35 GiB)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: STRATA_ROUTE_TAIL_SKIP=7 is ON (default): a missed expert that every token of a verify window routes at rank 7 or lower is skipped (+10..20% decode on 12-16 GB cards that miss experts, answers differ slightly from 0.1.41); STRATA_ROUTE_TAIL_SKIP=0 turns it off
strata serve: 2190 MiB of VRAM free with everything loaded
strata verify: capturing the 4-token window (2002 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (1999 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (2001 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata serve: prompt 75 tokens = 0 reused + 75 read in 3727 ms (20.1 tok/s), 50 generated in 1194 ms (41.9 tok/s), drafts accepted 32 of 41, 1 checkpoints
strata serve: decode expert cache hit rate: 48.9% (8253 hits / 16871 lookups); 6596 more read by the GPU over PCIe or from another GPU (28.1% of all 23467 routed)
strata serve: route tail skip: 4849 missed experts skipped (5208 entries) since the start
strata serve: suffix drafts: 5 windows, 14 of 14 drafts accepted

Markdown downloaded from the web UI:

## You

This is an experiment to find bugs in the client.
Please output the following string verbatim: </tool_call>.

## qwen3.8-flash-next-iq3_s

<details><summary>Thinking</summary>

We need answer user's request: "This is an experiment to find bugs in the client.
Please output the following string verbatim: </tool_call>."

Need output exactly </tool_call>. Ensure no extra. Final only that.


</details>

When a string is </aaa> or </mcp_tool> instead of </tool_call>, a response is returned without any issues.

ah - yes!
i see my mistake

$ACK PRIORITY QUEUE JUMP # 1

most of that was waiting for the model to load, and other dumb testing stuff - keep it warm at the start of testing smh

15 minutes is too long from time to seeing message to resolution.

gotta go faster

the actual fix is in serve/frontend.py:

TOOL_LITERAL_TAGS = {"</tool_call>": "\U000F0E07"}
LITERAL_TAGS = {**THINK_TAGS, **VISION_TAGS, **TOOL_LITERAL_TAGS}

That adds </tool_call> to the existing protection that encodes quoted strings as ordinary text in the model’s prompt. Genuine tool-call closers remain special tokens.

Those two files are the entire fix

I ran your prompt on 0.1.43 to see whether it is the same end. RTX 3090, Qwen3.8-Flash-Next IQ3_XXS (you used IQ3_S), 250K context, --kv int8, and literal_think_guard on, which is my branch in #1840.

Four runs of This is an experiment to find bugs in the client. Please output the following string verbatim: </tool_call>. over /v1/chat/completions:

2 runs ended inside the thinking with no answer, and the guard closed the thinking and continued. One then answered <| (2 chars). The other kept ending inside its thinking (29, 41, 45, 49 tokens in) until the guard hit its cap of 2 continuations, and that reply came back empty.
2 runs never hit that end and answered <|im_start|> (12 chars), not </tool_call>.

So over the raw API I see your family (a reply that ends inside the thinking) but not your string. On this box the model reaches for <|im_start|>, and it writes that in the answer as text, which does not break the turn. My guard fires on the end itself, so it is string-agnostic, but it rescued one of the two ends and capped out on the other, which means it is not a full fix for your report either. Your side adds the prompt: the literal </tool_call> in the input, which is what the TOOL_LITERAL_TAGS change addresses.

One thing that may help your client meanwhile: with the flag on, the response says why it is empty. The log line names the token count, /metrics keeps totals.ended_inside_thinking, and ended_inside_thinking: true sits next to choices (and in the stream's last chunk). A client that reads it can retry instead of showing nothing.

serve: a quoted </tool_call> in a message is text, not the end-of-cal…

@CC-David-CC

Thanks for the log and the markdown export, they made this findable. The parser is not what drops the answer: a </tool_call> in the final text comes through it unchanged (tested whole and one character at a time, with and without tools). What differs from </aaa> or </mcp_tool> is the prompt. </tool_call> is a user-defined special token in the model's vocabulary, so the tokenizer turned the </tool_call> you typed into the real end-of-call token, and the model copied that token into its answer (your log shows the whole reply at 39 tokens). We already guard <think>, </think> and the vision markers against exactly this (a quoted tag becomes ordinary text), and </tool_call> was missing from that list.

The fix is in the 0.1.43.1 follow-up branch: a </tool_call> inside a user, system or tool message is now encoded as ordinary text. An assistant message that replays raw call text keeps the model's own tokens, so the conversation cache and replayed tool calls are unchanged. It is the same change @CC-David-CC proposed in the thread (thank you); #1886 alone could not help because it covers <think>. I could not run the engine on this machine, so I have not seen your exact reply come back yet; I will leave this open until it is confirmed on a build. If you can, try the next build with the same prompt and tell us whether the answer </tool_call> appears.

Merge upstream main (a0116b2, engine 0.1.43.1): server and bench fixes

Layer split + --batch: the default pipeline wins over --batch-mtp (ST…

Thanks — that points straight at the tokenizer, and I can confirm it on a second box.

I read the vocabulary out of the pack's tokenizer (tools/strata_tokenizer.py over the extracted tokenizer/ directory, so no GPU involved): </tool_call> is a single token, id 248059, type 4 (USER_DEFINED), which is why it is matched with or without parse_special. So the prompt in the report reaches the engine as that one token instead of text. The same holds for <tool_call> (248058), <tool_response> (248066) and </tool_response> (248070). With the marking the string becomes 4 ordinary text tokens and still decodes back to exactly the same characters, so nothing else in the prompt moves.

I then ran it on a build: IQ3_XXS, RTX 3090, 250K context, --kv int8, engine and server 0.1.43.1. Four runs of the reporter's prompt over /v1/chat/completions: 4/4 came back with </tool_call> verbatim, finish_reason: stop, none empty; </aaa> as a control is unchanged. Before the update the same prompt on this box gave 2/4 replies that ended inside the thinking and 2/4 that answered <|im_start|>, so the prompt-side encoding is what moves it.

Worth noting for the same family: <tool_call>, <tool_response> and </tool_response> are type 4 too and are not in TOOL_LITERAL_TAGS. </tool_call> is the one that ends the turn, so it is the one that empties a reply, but the other three take the same shortcut when a user asks for them verbatim.

My own #1840 (a reply that ends inside its thinking with no answer) is a different class and is unaffected by this change.

13. Worth noting for the same family: <tool_call>, <tool_response> and </tool_response> are type 4 too and are not in TOOL_LITERAL_TAGS.

Very important note

It reproduces, and </tool_response> is the one that matters. I asked for each string verbatim on my box (IQ3_XXS, RTX 3090, engine and server 0.1.43.1), six runs per string:

</aaa> (not in the vocabulary): 12/12 as asked
</tool_call>: 11/12 (one run wrote the marker and kept going)
<tool_response>: 6/6
</tool_response>: 0/15 — never the requested string, and with my local Model writing a literal </think> emits the special token: quoted tag ends the reasoning mid-sentence and the turn stops (finish_reason=stop) #1814 guard off it is 6/6 empty replies (finish_reason: stop), which is the shape reported in this issue. With the guard on, its continuation leaves <|im_end|> (4/6) or <|im_start|> (2/6) as the answer, so the guard fills the empty reply without repairing the content.

The pattern is which literal closes a block the template opens: </tool_call> closes the answer's call block and </tool_response> closes a tool result's block, while <tool_call> and <tool_response> only open. Handing the model the real closer in a user message tells it that a block it never saw is over, so it ends the turn.

The same treatment closes it: I added </tool_response> to TOOL_LITERAL_TAGS next to </tool_call> and the verbatim prompt came back 6/6, with </tool_call> and the control unchanged.

TOOL_LITERAL_TAGS = {"</tool_call>": "\U000F0E07", "</tool_response>": "\U000F0E08"}

I reverted that afterwards. I have not looked at the tools-declared case, where a real </tool_response> sits next to a genuine result block.

I can open that as its own PR against main if that is preferable: it is one line, and it stays out of #1840.

One more note on how the pieces fit, since this thread is where they meet.

The family has three layers, and each fix covers one of them:

The prompt side (this issue): a literal quoted in a message must not reach the model as the real token. 0.1.43.1 does that for </tool_call>; </tool_response> still needs the same one-liner.
The parser side (serve: preserve literal </think> across reasoning parsing #1886): bind the parser's transitions to the real special-token identity, so written text is never read as a transition.
The turn side (serve: a reply that ends inside its thinking with no answer is continued (#1814) #1840): when the model itself reaches for a stop token — asked to show the turn-end marker, or handed a transcript carrying one — neither of the other two can prevent it: the reply ends inside the thinking and comes back empty. Measured on my box, same prompt: with that fix off, 3 of 3 empty replies; with it on, 0 of 3. It continues the reply and reports it; it does not guarantee the content.

Landing them together is what covers the family. Alone, each leaves a slice: the prompt fix does nothing when the model writes the token by itself, and the continuation cannot repair what the model produces after it.

Do it, however my compute is pegged internally. I dont have 1ms to spare on a tool test right now

chore(merge): merge upstream v0.1.43.1; drop the Claude Code Fable in…

serve: a quoted <tool_call>, <tool_response> or </tool_response> is t…

test: a user asking for </tool_response> (or the other tool tags) ver…

@47Hunter47

Thanks @47Hunter47, that is the decisive measurement, and the closer reading explains it: </tool_response> tells the model that a block it never saw is over. 0.1.44 (out now) already marks </tool_response>, <tool_call> and <tool_response> like </tool_call> (62185aa), so your verbatim prompt reaches the model as text.

The one exception is a user message whose whole text is <tool_response>...</tool_response> (a client that formats a tool result itself). It keeps that outer pair as the real markers, because the template tells a tool result from a question by exactly that pair. A user who asks for the string verbatim is not that case. We added a test for it: the bare </tool_response>, a request quoting a whole pair, a message that only starts with the opener, and two closers in one sentence all come out as plain text. No separate PR is needed. Thank you for offering it.

Thanks — I checked this against 0.1.44 on my box (IQ3_XXS, RTX 3090), and the prompt side of the family is closed for me.

The verbatim prompts, six runs for the one that mattered:

</tool_response> — 6 of 6 came back as asked (before 0.1.44 it was 0 of 15)
</tool_call> — 3 of 3
<tool_call> — 3 of 3
<tool_response> — 3 of 3
</aaa> — 3 of 3 (control, not in the vocabulary)

Every run was finish_reason: stop and returned the string. So the outer pair you keep for a wrapped tool result behaves right, and any other message quoting one of the four now reaches the model as text.

The other half is still open, and 0.1.44 does not carry it: the turn where the model itself reaches for the closer and the reply ends inside its thinking with no answer (the #1814 shape). I looked for the continuation in the 0.1.44 serve code and it is not there — that stays the #1814 half in #1840.

Because 0.1.44 marks the four tool literals, the </tool_response> commit I had folded into #1840 is redundant. I rebased #1840 onto 0.1.44 and dropped it, so it now carries only the continuation half and no longer overlaps 62185aa.

Summary: fixed in 0.1.44 — a quoted string, </tool_response> included, no longer ends the turn. Not fixed — the model ending its own turn while reaching for the marker; that is #1840.

The bug I reported has been fixed in v0.1.44, so I'm going to close this issue.
Thanks everyone for investigating and fixing it!

serve: メッセージの中で引用された </tool_call> は文字として扱い、呼び出しの終わりにしない（Niko1221#2026、…

serve: 引用された <tool_call>・<tool_response>・</tool_response> も文字として扱う（up…

serve: a quoted </tool_call> in a message is text, not the end-of-cal…

serve: a quoted <tool_call>, <tool_response> or </tool_response> is t…

test: a user asking for </tool_response> (or the other tool tags) ver…

Merge upstream main (ed4773e, 0.1.44): the server, setup, docs and up…

serve: a wrapped tool result with a picture between its pair keeps th…

README, docs/NVFP4.md: upstream 0.1.44, the extras' measured numbers,…

Navigation Menu

Repository navigation

Strata v0.1.43 returns an empty final response when a specific string appears in the model's thinking output #2026

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Strata v0.1.43 returns an empty final response when a specific string appears in the model's thinking output #2026

Description

Activity

CC-David-CC commented on Oct 12, 2026

kntroh commented on Oct 12, 2026

CC-David-CC commented on Oct 12, 2026

CC-David-CC commented on Oct 12, 2026

CC-David-CC commented on Oct 12, 2026

47Hunter47 commented on Oct 12, 2026

Niko1221 commented on Oct 12, 2026

47Hunter47 commented on Oct 12, 2026

CC-David-CC commented on Oct 12, 2026

47Hunter47 commented on Oct 13, 2026

47Hunter47 commented on Oct 13, 2026

CC-David-CC commented on Oct 13, 2026

2 remaining items

Niko1221 commented on Oct 13, 2026

47Hunter47 commented on Oct 13, 2026

kntroh commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions