[kossum]

Problem and result

Clients that apply their own chat template need to send a raw prompt to /v1/completions. Strata currently has no endpoint for that.

This adds the legacy OpenAI endpoint with streaming and non-streaming responses, text_completion objects, and token usage. The server doesn't apply the chat template again.

Related discussion

#2277

Approach

The handler uses the existing generation path and accepts one string prompt. max_tokens: 0 or a negative value uses the remaining context.

Explicit thinking-disable flags append the template's no-thinking tail. Without those flags, the prompt passes through unchanged.

Streaming sends each piece of text once. With echo: true, it sends the prompt once at the start; the finish chunk has empty text.

This also keeps the structured-output error handler in the chat endpoint, so a failed streamed response sends an SSE error and [DONE], not another HTTP response inside the stream. Engine errors after streaming starts are handled as SSE errors too.

Validation

Rebased onto main e88ee39d (0.2.0-beta). These local checks ran on 7e08755f.

Passed:

Windows, Python 3.13.2, with jsonschema installed:

uv run --no-project --with regex --with jinja2 --with jsonschema python -m unittest serve.test_server serve.test_structured serve.test_monitor

368 tests, OK, 3 skipped.

Linux, Python 3.11 in Docker, using a dedicated venv with only the pinned requirements.txt packages:

python tools/minimal_server_smoke.py --report /tmp/minimal-server-smoke.json

Reported "passed": true.

git diff --check upstream/main also passed.

Failed:

The full suite in the same Linux venv:

python -m unittest discover -s serve -p 'test_*.py'

888 tests, 2 failures, 18 skipped:

• test_lifeline.Lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed
• test_server.VisionTempFiles.test_work_dir_has_no_space_when_temp_has_one

I reran both tests on clean main e88ee39d with the same container and dependencies. Both fail there with the same assertions. I haven't changed either behavior in this PR.

Not run:

• Live-model testing on this rebased revision. The earlier live checks were on an older revision.
• CUDA, HIP, SYCL or Metal builds. This PR changes Python serving code, not native backend code.
• A byte-for-byte comparison of all existing endpoints against the release.

Limits

Only a string prompt and n=1 / best_of=1 are supported. Suffix and logprobs are rejected.

stream_options accepts only include_usage, and only with streaming. Usage still comes on the finish chunk, not a separate choices: [] chunk, so this doesn't yet implement the full OpenAI streaming-usage format.

[qiuzan2001]

Two findings (Linux, Python 3.14; serve.test_server.MaxTokens passes):

1. serve/test_structured.py fails on this branch once jsonschema is installed (it is skipped without it): test_invalid_generation_is_an_error_and_never_streams_content gets a raw HTTP/1.0 502 Bad Gateway ... {"error": {"type": "structured_output_failed", ...}} block written into the SSE stream instead of data: {"error": ...} followed by data: [DONE], for all three invalid scripts. On main 296b59d the same file passes (10 OK). The except StructuredOutputError that the streamed chat path relied on now sits in _completions, so a streamed /v1/chat/completions with response_format reaches do_POST's generic handler after the 200 headers are out.
2. In _completions' chunks(), every token is already streamed as a delta, and the final chunk again carries "text": (prompt if echo else "") + text, the whole completion. A client that concatenates the streamed text fields gets the answer twice; OpenAI's final chunk has an empty text with the finish_reason. The new tests check only usage and [DONE] on the stream.

[kossum]

Thanks for catching both. Fixed in the latest push.

I accidentally moved the StructuredOutputError handler out of _openai when adding _completions. It's back now, so a failed streamed response sends an SSE error followed by [DONE] instead of writing another HTTP response into the stream.

The final completion chunk now has empty text. With echo: true, the prompt goes out once at the start. Added tests that concatenate the streamed text and compare it with the non-stream response, with echo on and off.

You were also right about jsonschema: I didn't have it installed, so that test was being skipped. Reran with it installed and checked both fixes against the live model too. I also added a streaming failure check to the dependency-absent test so this doesn't slip through again.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[kossum]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.

[kossum]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.