Repository navigation
Description
devstral tool parser: parallel tool calls end up in the first call's arguments
Describe the bug
When Devstral emits more than one tool call in one reply, the devstral
tool parser returns a single tool call. The other calls end up inside the
first call's arguments, together with the raw [TOOL_CALLS] and [ARGS]
markers. The arguments are then no longer valid JSON, so clients reject
the call.
Streaming and unary responses both behave this way.
Devstral's own chat template renders several calls in exactly this form:
[TOOL_CALLS]name[ARGS]{...}[TOOL_CALLS]name[ARGS]{...}. A reply with
parallel calls is therefore normal model output.
To reproduce
- OVMS 2026.4.1 (Linux, GPU plugin). The parser code is unchanged on main
at 3b33556 (2026-10-26). - Model:
mistralai/Devstral-Small-2-24B-Instruct-2512, exported with
optimum-intel as a text-generation model in int4. - Server options:
--task text_generation --tool_parser devstral
Request:
curl -s http://localhost:8000/v3/chat/completions -H 'Content-Type: application/json' -d '{
"model": "devstral", "temperature": 0.15, "max_tokens": 300,
"tools": [{"type": "function", "function": {"name": "grep",
"description": "Search a regex.",
"parameters": {"type": "object",
"properties": {"path": {"type": "string"}, "pattern": {"type": "string"}},
"required": ["path", "pattern"]}}}],
"messages": [{"role": "user", "content":
"Search guix/gnu/packages for tbb, for pugixml and for onednn, each with its own grep call, all at once."}]}'Actual response (unary):
"tool_calls": [
{
"id": "6sWZt5NCM",
"type": "function",
"function": {
"name": "grep",
"arguments": "{\"path\": \"guix/gnu/packages\", \"pattern\": \"tbb\"}[TOOL_CALLS]grep[ARGS]{\"path\": \"guix/gnu/packages\", \"pattern\": \"pugixml\"}[TOOL_CALLS]grep[ARGS]{\"path\": \"guix/gnu/packages\", \"pattern\": \"onednn\"}"
}
}
]With "stream": true the same thing happens. All deltas carry index: 0.
After the first call's closing "}, the stream continues with an
arguments delta "[TOOL_CALLS]", then "grep", "[ARGS]", and the next
call's JSON, all still on index 0.
Expected behavior
Three tool calls with indexes 0, 1 and 2, each with its own id, name
and valid JSON arguments. In streaming, the second call should start a
new delta with index: 1, its name and a new id.
Cause
In src/llm/io_processing/devstral/tool_parser.cpp, parseChunk looks
for [TOOL_CALLS] only in the AWAITING_START_TAG state (around line 74).
That is also the only place where toolCallIndex is incremented. Once the
first [ARGS] has been seen, the parser is in PROCESSING_ARGS (around
line 129), and that state appends everything up to </s> to the current
arguments. A later [TOOL_CALLS] is never checked there, so the parser
never returns to AWAITING_ARGS_TAG for the next call.
src/test/llm/output_parsers/devstral_output_parser_test.cpp has no
case with more than one tool call. It covers single calls, empty
arguments, content before a call, invalid order, a missing [ARGS], and
streaming of a single call.
Suggested fix
In PROCESSING_ARGS, treat [TOOL_CALLS] like </s>. The arguments
before it close the current call, toolCallIndex is incremented, and the
state goes back to AWAITING_ARGS_TAG for the next name. The unary path
needs the same split. Tests with two and three calls, unary and
streaming, would cover it.
Workaround
On the client side, cut the arguments at the first [TOOL_CALLS]. That
keeps only the first call, and the model issues the next one in its
following turn.
Activity