[Daksha1611]

🛠 Summary

Fixes #4664.

The /v1/tokenize response nested the tokens only when the tokenizer returned more than one row, so
"text": ["hello world"] got the same flat list as "text": "hello world".

TokenizeRequest now records whether text was given as an array of strings (inputIsBatch), and
parseTokenizeResponse() takes it as a new last parameter, defaulting to false. When it is set the
response is always a list of token lists. The three callers (LLM servable, embeddings and rerank
calculators) pass it through.

This changes the response for a client that sends a one-element array and reads the flat list. A
single string is unchanged, as are arrays of two or more strings. The docs now show the list form.

Tests: three request tests (string, one-element array, longer array) and three response tests in
tokenize_parser_test.cpp.

I have no OVMS build container available, so this is not compiled with Bazel. I compiled the real
tokenize_parser.cpp and tokenize_parser_test.cpp against the OpenVINO runtime with abseil stubbed;
all 24 tests in the file pass. I also ran the request through a real Qwen3 tokenizer: before,
["hello world"] gave [14990,1879]; now it gives [[14990,1879]]. The one-argument additions in the
three callers could not be compiled here.

🧪 Checklist

• Unit tests added.
• The documentation updated.
• Change follows security best practices.

[Copilot]

🔵 Needs a closer look

The shared parser interface and production callers still need validation with actual dependencies in a full Bazel build.

No open issues 🎉

What changed in this PR

Fixes #4664 by preserving the request’s input form in /v1/tokenize responses: a one-string array now returns nested token lists, while a plain string remains unchanged.

Changes:

• Track batch input and pass it through all three tokenizer integrations.
• Adapt embeddings parsing to the shared parser’s updated return type.
• Add regression tests and document list responses.

File	Description
src/​tokenize/​tokenize_parser.hpp	Declare batch tracking and updated parser interfaces.
src/​tokenize/​tokenize_parser.cpp	Preserve input form and serialize batches consistently.
src/​test/​tokenize_parser_test.cpp	Test parsing and response shapes.
src/​rerank/​rerank_calculator_ov.cc	Forward batch metadata for rerank tokenization.
src/​llm/​servable.cpp	Forward batch metadata for LLM tokenization.
src/​embeddings/​embeddings_calculator_ov.cc	Forward batch metadata for embeddings tokenization.
src/​embeddings/​embeddings_api.cpp	Handle the updated shared parser result.
docs/​model_server_rest_api_tokenizer.md	Document nested responses for string arrays.

_{Review effort: Balanced}

[Daksha1611]

@przepeck I've pushed the change you suggested (parseInput now reports the unbatched case and errors through a status) and all checks are green. Could you take another look when you have a moment? Thanks.

[przepeck]

PR merged, thank you for your contribution @Daksha1611