Skip to content

/v1/tokenize returns a flat list when the text array holds a single string #4664

Closed

Description

@Daksha1611

Describe the bug

The shape of the /v1/tokenize response follows the number of strings in the request instead of the
form of the request. src/tokenize/tokenize_parser.cpp nests the tokens only when the tokenizer
output has more than one row (outputShape[0] > 1), and a one-element array produces one row, the
same as a plain string:

{"text": "hello world"}              ->  {"tokens":[14990,1879]}
{"text": ["hello world"]}           ->  {"tokens":[14990,1879]}
{"text": ["hello world", "hi"]}     ->  {"tokens":[[14990,1879],[6023]]}

A client that sends a list and reads tokens[i] for each input gets a list of token lists for
two or more inputs, but a flat list of integers for exactly one. Batches that happen to contain one
string need special handling on the client.

None of the tests send a one-element array: the list cases in tokenize_parser_test.cpp and
tokenize_endpoint_test.cpp all use two or more strings. The docs show only the single-string
response.

To Reproduce

curl http://localhost:8000/v1/tokenize -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "text": ["hello world"]}'

Reproduced with the parser and a real Qwen3 tokenizer, following the same sequence as
src/llm/servable.cpp: parse the request, encode, build the response.

Expected behavior

An array of strings always gives a list of token lists, including an array with one string.

Note that this changes the response for clients that currently send a one-element array and rely on
the flat list.

Logs

None.

Configuration

  1. OVMS version: main @ eec7daa
  2. any servable that serves /v1/tokenize (LLM, VLM, embeddings, rerank)
  3. N/A
  4. N/A
  5. N/A

Additional context

I have a patch with tests and docs and will open a PR.

Activity

  1. added a commit that references this issue on Oct 26, 2026
    89dde36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions