[Nipale-ai]

Problem

When a reply makes a tool call, the whole (stream: false) reply shows no text. The stream still sends the whitespace around the call blocks as text.

Nemotron 3.5 Lightning writes \n<tool_call>…</tool_call>\n. On 1.0.1 this gives (measured with the model). The code path is unchanged in 1.0.2, and the new tests fail the same way there:

Route	Whole reply	Stream
/v1/chat/completions	content: null	two content: "\n" deltas. One lands between two argument deltas of the same call (" … \n … }).
/v1/messages	[tool_use], no text block	[text "\n", tool_use, text "\n"]
/v1/responses	[function_call], no message item	[message "\n", function_call, message "\n"]

Clients that stream send this text back as history. Nemotron's template does not trim earlier assistant turns. The next prompt therefore has <think></think>\n\n\n<tool_call> where a non-streaming client sends <think></think>\n<tool_call>. The goldens show the same mismatch: tools/two_tools has content: null, and tools/two_tools-stream streams "\n".

Change

Server code only. Tokens, engine and kernels do not change.

• reply_text.zig: two small helpers.
  • heldBack: with tools, the streamed text without its trailing whitespace. It returns a prefix, so the stream's pointer checks still hold.
  • finishedProse: hideToolCalls of the finished reply. Once a call is hidden, its trailing whitespace is not sent either.
• chat.zig: two one-line changes. Generation.commit uses heldBack and Generation.finish uses finishedProse. A reply with tools but no call still sends its whitespace at the end, so plain prose does not change.
• stream_preflight_test.zig: a fake engine (CallEngine) answers two calls byte by byte, with a \n between them. Three new tests send this reply through the chat, Messages and Responses request translations. Each test checks that the streamed chat content equals the whole reply's content and that both have 2 calls.
• golden_check.zig: three frozen 0.6.6 goldens now differ on purpose and are listed as known diffs: tools/two_tools-stream, tools/two_tools-single-stream and anthropic/tool-use-stream. The stream no longer sends "\n", so the two two_tools goldens also lose their second role line. The new tests catch a revert. The known-diff entries alone would not.
• Other behaviour change: an invalid call with whitespace around it, for example "\n<tool_call>{nope}</tool_call>\n". The stock stream sends only "\n\n", but the whole reply has the full text. With this change, the stream sends the full text at the end, the same as the whole reply.

Tests (f8fe17d, Zig 0.17.0, aarch64 Linux)

• New tests:
  • Stock with only the new tests: 61 passed, 3 failed. The 3 failures are the new tests: the stream has "\n", the whole reply has none.
  • This change: 64 + 28 pass. Stock: 61 + 28 pass.
• How the server tests run on Linux:
  • build.zig adds the server tests and test-golden only on macOS (metalTargets → nativeServer).
  • zig build test test-golden therefore stops with no such step: test-golden, on stock and with this change.
  • zig build test on Linux runs only the CUDA host tests, not the server tests. Its one failure (direct_io, on stock and with this change) is fixed by the open core: the direct_io test's reference read goes through the page cache #486.
  • So I ran the server tests directly:
    • zig/tests/server/zig_test.sh (results above).
    • tool_parse_test.zig: 14 pass on stock and with this change.
    • The test-golden graph from build.zig, built by hand: fake_serve + golden_check, plus -lc because Linux needs it. Stock: 312/319 equal + 7 known. This change: 309/319 equal + 10 known. 0 unexpected on both.
• Not run: the macOS host suite (zig build test test-golden -Dcpu=apple_m1) and the Metal model check were not run for this change. The server code is the same for both engines.
• tools/zig/lean_check.py: 512 problems on f8fe17d, 512 with this change. chat.zig has 599 lines.
• zig fmt --check: clean on the 4 changed files.

Measured on 1.0.1 (041d14a)

Setup:

• NVIDIA DGX Spark (GB10), Ubuntu 24.04.5 (DGX OS), driver 580.178.04, CUDA 13.0.
• Checkpoint TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit, revision d9d758fb83953437f7263256b0d96157e2a348b8.
• Server flags: serve <checkpoint> --parallel 8 --context 262144 --temperature 0 --no-thinking.

Results:

• Build: zig build native -Dtarget=aarch64-linux-gnu.2.28 -Dversion=1.0.1, with the 16 fatbins taken from the 1.0.1 binary and strip on as in the release.
  • Unpatched: byte for byte the 1.0.1 release binary (sha256 1ff1f6ea…).
  • Patched: ff116abb…, 128 bytes larger, with the same 16 fatbins.
• Stream vs whole, 14 cases: 6 differ before, 1 after. The one left is reasoning only (see Not covered). Text deltas inside tool-call streams: 9 before, 0 after.
• Messages and Responses with a tool: the stream equals the whole reply after the change.
• Token output: 64 checks, all equal to stock.
  • 8 prompts, each run solo, without drafts, and 8 at once: 24 text-hash checks and 24 token_sha checks.
  • 8 more prompts, streamed, solo and 8 at once: 16 streamed-text sha checks.
• tool-eval-bench f7c34a1 (it streams every turn; seed 42, 69 scenarios). Run on an earlier draft of this fix, not re-run on the final patch:
  • Stock: 53.
  • With the change: 54, and 54 again on a second run.
  • Only TC-60 changed (fail → partial). Turn 1 is the same call. Turn 3 now sees the history without \n\n.

Not covered

• Leading whitespace of prose, at the start or after a call, is still streamed. Examples: "\nText<tool_call>…" streams the \n, and <tool_call>…</tool_call>\nDone. streams "\nDone.".
• Prose before or after a call is still streamed, while the whole reply has no text. This is older behaviour, not changed here.
• A thinking reply's streamed reasoning_content keeps a trailing \n that the whole reply strips (tools/xml_tool-stream). This is the same pattern and belongs in a separate change.

Reproduce

curl -sN localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"local-model","stream":true,"temperature":0,
 "messages":[{"role":"user","content":"What is the weather in Berlin?"}],
 "tools":[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object",
 "properties":{"city":{"type":"string"}},"required":["city"]}}}]}' | grep '"content"'

On 1.0.1 this prints two "content": "\n" deltas. With this change it prints none.

[plotarmordev]

Receipt from a DGX Spark (GB10, Nemotron 3.5 Lightning 4-bit @ d9d758fb, --context 16384 --parallel 4 --no-thinking, greedy, one get_weather tool), same request whole vs streamed:

Route	main (f8fe17d2) stream	this PR (7da967e1) stream	whole reply (both)
/v1/chat/completions	content deltas ["\n", "\n"]	none	content: null, 1 call
/v1/messages	[text, tool_use, text]	[tool_use]	[tool_use]
/v1/responses	[message, function_call, message]	[function_call]	[function_call]

So the stream now matches the whole reply on all three routes.

[ashhart]

Thank you, @Nipale-ai, for tracing the whitespace mismatch across chat, Messages and Responses. The three new stream regressions pass, and the change keeps trailing whitespace pending until it can distinguish ordinary prose from a hidden tool call. The known golden differences record that deliberate protocol correction; reasoning whitespace and prose around calls remain separate work.