OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine by gt50 · Pull Request #1916 · Niko1221/Strata

gt50 · 2026-10-11T03:07:11Z

Title

OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine

Summary

"logprobs": true (with "top_logprobs": 0..20) on /v1/chat/completions now returns choices[0].logprobs.content, as OpenAI and llama-server do. Until now only the SYCL port had it (835d30b); the CUDA/HIP engine and serve/ accepted the field and returned nothing. A client that marks values the model was unsure of (the weakest token of a value under 50%) needs it.

What changed

Engine (src/program/generate.cpp): the GEN key logprobs=K writes LP <logprob> [id:logprob ...] after each T <id>, from the verify window's head logits (Verifier::copy_logits), before penalties, temperature and sampling. It is the SYCL port's code, placed at both token outputs (the plain and the pipelined path). Without the key, no LP lines and nothing else changes.
serve/server.py:
- sampling_keys sends logprobs=K only when the request asks; an engine without LP lines ignores the key and the reply has no logprobs.
- StrataEngine.generate reads a token's LP line before it yields the token (the engine writes both in the same flush), so the logprobs belong to the right text chunk. When nothing follows a token within 0.5 s, it is yielded without one. A line read this way is handed to the early-stop drain, so a DONE read after the last token is not waited for again (tested).
- Each content / reasoning chunk carries the entries of the tokens that spell its text. token is the text the token completes (an incremental UTF-8 decode, as the reply's text is made): a character split across tokens is in the token that finishes it, so the tokens, joined, spell the reply. bytes are the token's own bytes. Tokens that make no text (</think> and the newlines after it) and the stop token are left out.
- openai_collect merges them into choices[0].logprobs (content, refusal, and reasoning_content with thinking on).
- response_format json_schema: with logprobs, the answer is returned as the model wrote it (validated as before, the same value) when its tokens spell it, including the JSON inside a ```json fence; otherwise the canonical JSON without logprobs. Without logprobs, the canonical JSON as before.
docs/DETAILS.md: a "Logprobs" paragraph after the sampling one.
serve/test_logprobs.py: the request keys, the engine line pairing (with and without LP lines, not asked, early stop), the API streamed and not (multi-byte characters, top_logprobs, the stop token), thinking, json_schema with a fenced answer, and the span helper.

Extra Notes

Default path: a request without logprobs sends the same GEN line, reads the same lines and returns the same reply (tested). The engine prints LP lines only when asked.
Measured on a Radeon 8060S (Ryzen AI Max+ 395, Windows 11, UD-IQ4_XS, --spec 4 --mtp), this branch built with tools/hip/build_windows.bat (gfx1151, engine 0.1.42):
- A greedy 255-token answer: the same text with and without logprobs, 38.5 and 38.1 tok/s; 254 entries (the stop token left out).
- {"rent": "$1,250", "quote": "Rent is $1,250 a month."} from a two-field json_schema at temperature 0: 26 tokens that spell the content.
- Thinking on: content and reasoning_content each spelled by their tokens. A streamed json_object answer and a sentence with café, 東京 and €5: spelled too.
- The same patch on 0.1.40.2 gave the same results; that build is what I run day to day.
Not covered: the Responses and Anthropic APIs, and --parallel batch slots (their BT lines get no LP lines). The legacy completions form "logprobs": N is taken as top_logprobs N.
Tests: all serve test files pass on main 61b3fb5 (704 tests).

🤖 Generated with Claude Code

karls0r · 2026-10-11T10:53:17Z

🤖 Contributed by an AI agent (Ren, running Claude) on behalf of @karls0r. Measured and written by the agent; Karl reviewed the plan.

Thanks for this — it is exactly what we needed for typed decisions (one token, probability per option letter), the use case of #675.

Measured on 2× RTX 3090 (24 GB each), layer split 22, Fedora/CachyOS, engine built from this branch (c8ee496, CUDA, sm_86), UD Qwen3.8-Flash-Next GSQ-RCO IQ3_S, --spec 4 --mtp, --kv k8v4:

Without "parallel": works. "logprobs": true, "top_logprobs": 20, "max_tokens": 1 → A at −0.002, B at −9.28. On our 156-case test set (German mail/web text: prompt-injection yes/no, mail triage, a sorting task), the decision is as accurate as asking the model to answer with thinking on (injection 100 %, triage 94 %), p50 0.6 s per decision instead of 2.4–4.2 s and 200–430 output tokens. The same five requests repeated give identical log-probabilities.
With "parallel": 2 (and --batch-groups 2, our production config): "logprobs": null on every reply, even a single request.

Why: with "parallel", StrataEngine.generate() goes straight to generate_batched(). A one-token request there is always an admission (server.py if alone and left > 1: → BGEN; the engine reads BGEN as GEN 1 with the same keys, so logprobs=K arrives and the engine does write the LP line). But _control(), which reads the admission's control lines, has no LP branch, so the line is dropped.

A small fix on top of your branch (server only, engine unchanged), +57/−8 in serve/server.py plus a test file — branch karls0r:logprobs-batched (your commit + one on top, diff of the fix alone):

_control(): with logprobs asked, a T line is held until the next line; its LP line goes to lp_lines before the token is passed on. No second read, nothing skipped: any other line, or 0.5 s of silence, passes the held token on first (so an engine without LP lines only delays a token by one line).
lp_lines per thread when batch slots are on, the same way last is (several requests run at once; resetting a shared deque let one request take another's entries).
A batch slot's BT lines carry no LP lines, so past the first token a reply would get partial or misplaced entries. Instead: logprobs with "parallel" and max_tokens != 1 is a 400 (fail closed, like logit_bias with batch). One-token decisions — the main use — work.
serve/test_logprobs_parallel.py: the fake engine of test_parallel.py with LP lines behind the real StrataEngine/Service: one decision, four at once (each gets its own entries), not asked (unchanged), longer replies refused, an engine without LP lines.

Checked: all serve tests on the branch: 698 passed, 11 skipped. Live with "parallel": 2: 109 cases serially and two at a time — every reply has logprobs, |Δp| to the run without "parallel" mean 0.007 (max 0.08, batch numerics); a decision while one 900-token chat runs: 0.32 s with logprobs; while two run (both slots busy) it waits for a slot: 8.2 s, still with logprobs. Without logprobs the GEN/BGEN line is unchanged (sampling_keys adds the key only when asked).

Not covered / not tested: logprobs for the slot phase (BT → would need an LP line per slot from the batch verifier; left out on purpose), HIP and SYCL builds (the change is Python only), the Responses and Anthropic APIs.

Would you like this as a PR into gt50:logprobs, or should I open it as a follow-up once #1916 is merged?

gt50 · 2026-10-16T04:26:14Z

@karls0r thank you for the measurements and the fix, and sorry for the slow reply. Please open it as a PR into gt50:logprobs, so #1916 doesn't land with "logprobs": null under "parallel". I'll review it there.

Refusing logprobs with "parallel" and max_tokens != 1 with a 400 makes sense to me: one-token decisions work, and nothing returns partial or misplaced entries. LP lines for the BT slot phase can be a later step.

Once it's in, I'll rebase the branch onto current main (it is behind) and run the serve tests again.

Niko1221 · 2026-10-17T07:14:12Z

Thank you. A feature, outside this speed pass.

settinghead · 2026-10-18T17:06:13Z

Another data point for this PR, plus a rebase onto v0.1.45 in case it saves time.

Rebase. This branch (c8ee4967) plus @karls0r's parallel fix (73f0771), cherry-picked onto v0.1.45 (477ac13c): zigzag-tech/Strata zz/logprobs. There were two conflicts, both small:

generate.cpp: the logprobs key sits next to the pcie_frac line, which now also sets req_pcie_set.
server.py: _control() gained on_quiet (the BYIELD slot give-way). The fix's want_lp is now passed next to it at both call sites.

emit_lp still reads the right row in 0.1.45's merged first window: that loop starts at i = a = T-1. With the fork's test venv, all serve tests pass: 856 passed, 11 skipped.

Measured. RTX 3090 (24 GB, sm_86, CUDA 13.1), Linux, Qwen3.8-Flash-Next GSQ-RCO Q2_0, --spec 4 --mtp, --kv int8, "parallel": 3.

Typed decisions with max_tokens: 1 return top_logprobs under "parallel". max_tokens: 5 with logprobs gets the 400.
98-row held-out set: is a coding-agent pane blocked on its user? 6-way choice, each row asked in both option orders, 196 requests.
- No failures.
- Accuracy 0.898 (forward) / 0.867 (reversed).
- Right in both orders: 0.857.
- p50 1.9 s at a median 1,420-token prompt.

One more thing we needed for vLLM-shaped decision clients was continue_final_message. The prompt ends with an open assistant turn ({"answer": "), so the label is the next token. It's a separate commit on the same branch (7df1309): thinking off only, otherwise a 400. It doesn't depend on this PR. Happy to open it separately if that's useful.

- engine: the GEN key logprobs=K (0..20) writes "LP logprob id:logprob ..." after each "T id", from the verify window's head logits (as the SYCL port's 835d30b); absent key, no LP lines - serve: generate() reads a token's LP line before yielding the token; each text chunk carries the logprobs of the tokens that spell it (markup such as </think> left out, a character split across tokens in the token that finishes it); openai_collect merges them into choices[0].logprobs; a json_schema answer is returned as written when its tokens spell it - tests: serve/test_logprobs.py; docs/DETAILS.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…; 400 for longer replies With "parallel" set, generate() goes to generate_batched(), whose control-line reader (_control) had no LP handling: the engine wrote the LP line (BGEN is read as GEN 1, logprobs=K included) and the server dropped it, so every reply came back with "logprobs": null. - _control(): with logprobs asked, a T line is held until the next line; its LP line goes to lp_lines before the token is passed on (no second read, nothing skipped: any other line, or 0.5 s of silence, passes it on first) - lp_lines per thread with batch slots, as last is (several requests run at once) - a batch slot's BT lines carry no LP lines, so logprobs with parallel and max_tokens != 1 is a 400 (fail closed, as logit_bias does) instead of partial or misplaced entries - serve/test_logprobs_parallel.py: the fake engine of test_parallel with LP lines; one decision, four at once (each gets its own), not asked, longer replies refused, an engine without LP lines Contributed by an AI agent (Ren, Claude) on behalf of karls0r; measured on 2x RTX 3090, layer split.

@karls0r

…t for replicas without them ReplicaEngine.batch counts each replica without batch slots as one, so with replicas and "parallel" off, logprobs with max_tokens above 1 got the 400 that says "run without parallel". batch_slots_on() asks each replica's own engine. Tests: test_logprobs.Replicas (a 200-token reply with logprobs through two replicas without slots; the 400 once one has them). docs/DETAILS.md: the one-token limit with "parallel", with @karls0r's measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gt50 · 2026-10-19T09:15:52Z

@settinghead thank you for the rebase and the RTX 3090 numbers. The 196-request test, run in both option orders, is a good check for this PR's main use.

The branch is now on v0.1.45 (4098d58), with @karls0r's fix merged in. I resolved the same two conflicts the same way you did: the logprobs key next to pcie_frac / req_pcie_set, and want_lp passed next to on_quiet at both _control() call sites. One commit on top: with replicas and "parallel" off, the 400 for longer replies no longer fires. ReplicaEngine.batch counts each replica as at least 1, so the check now looks for batch slots on each replica's own engine (test: test_logprobs.Replicas).

Checked on a Radeon 8060S (HIP, Windows, UD-IQ4_XS, --spec 4 --mtp, --kv int8):

406 serve tests pass.
Without "parallel": one-token decisions return top_logprobs, and asking again gives identical numbers. A 171-token answer has the same text with and without logprobs (decode 40.8 and 40.9 tok/s).
With "parallel": 2 (--batch 2): four decisions at once get the same answers and log-probabilities as one at a time. max_tokens: 5 gets the 400. A decision beside a 435-token reply took 0.44 s, with logprobs.

continue_final_message: yes, please open it as its own PR, since it doesn't depend on this one.

For anyone testing on Strix Halo: v0.1.45 can stall on the first prompt there with gfx1151's default switches on. #3151 has the fix; until it lands, STRATA_PLE_BATCH=0 avoids it.

Niko1221 · 2026-10-19T16:59:36Z

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

…plica server without slots A batch slot prints "BT <slot> <id>" and its decode loop never calls emit_lp_line, so logprobs was refused outright there. An admission does write one: a max_tokens 1 request is asked of the engine as BGEN and read as "GEN 1" (generate.cpp:12090), the single-token path that prints "T <id>" and the LP line after it in one flush. _control() now reads that line (want_lp) and keeps it on the request's own LPLines, held on its cancel Event -- the engine's lp_lines is one attribute and several slots read control lines at once, so it cannot be shared between them. A longer reply is still a 400: past that first token its entries would cover a prefix only, which a client cannot tell from the model's own answer. This is the line gt50 drew on Niko1221#1916 ("LP lines for the BT slot phase can be a later step"), and the read itself follows the server-only fix karls0r posted there (+57/-8, engine unchanged). The other half is a refusal that should not have been there at all. ReplicaEngine.batch totals how many requests its engines run at once and counts a replica without slots as one (serve/replicas.py), and the check asked it "does this engine use batch slots". A server started with --replicas 2 and no --parallel therefore answered 400 to every logprobs request, while reading every reply by the GEN path, which does write the lines. batch_slots_on() asks each replica's own figure instead. Tests: serve.test_logprobs 32/32 -- the two new ones in Replicas are no slots behind two replicas, and slots on one of them; serve.test_logprobs_parallel 6/6 -- a decision, allowed_token_ids on the admission, four decisions at once each with its own, not asked, a longer reply a 400, and an engine that writes no LP lines at all. Hardware, Swift-1.5 IQ3_XXS, --spec 4 --mtp, --parallel 2: max_tokens 1 + top_logprobs 5 -> 200, one entry spelling the reply (A -0.062427, then B -2.848, " A" -7.415); allowed_token_ids over 11 ids -> 200, the named ids as the candidates, sum(exp) 1.0, a chosen token outside them keeping its own logprob; max_tokens 5, max_tokens absent and max_completion_tokens 0 -> 400 naming max_tokens 1; max_completion_tokens 1 -> 200 with its entry; streamed -> one chunk carrying the entry for "Odd"; four decisions at once on two slots -> 200 each, 1.6 s for all four, every entry matching its reply; a plain request and a 64-token reply unchanged. The whole serve suite: 1073 tests, 1 failure, 1 error, 21 skipped. Both are known on a clean upstream/main here and touch no logprobs code: test_responses.test_server_close_ends_the_sampler (1 != 2, order-sensitive -- it passes when its file runs alone, which it did in the 83-test run just before this one) and test_winjob's running() on a pid this environment cannot read.

karls0r mentioned this pull request Oct 11, 2026

Feature request: logprobs / top_logprobs on /v1/chat/completions (typed decisions from one forward pass) #675

Closed

qiuzan2001 mentioned this pull request Oct 15, 2026

[Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #2254

Closed

karls0r mentioned this pull request Oct 16, 2026

serve: logprobs with "parallel" (batch slots) for one-token decisions; 400 for longer replies gt50/Strata#1

Merged

gt50 and others added 3 commits October 18, 2026 23:15

gt50 force-pushed the logprobs branch from 6ca853b to 4098d58 Compare October 19, 2026 06:20

gt50 mentioned this pull request Oct 19, 2026

qsa: the default block-scores kernel without #2234's per-query keys (a gfx1151 verify window stalled on v0.1.45); the batch rows get their own kernel #3151

Closed

Niko1221 closed this Oct 19, 2026

xiezhipeng-git mentioned this pull request Oct 19, 2026

[Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine #3200

Closed

Navigation Menu

Repository navigation

OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine - #1916

OpenAI logprobs on /v1/chat/completions for the CUDA/HIP engine#1916
gt50 wants to merge 3 commits into
Niko1221:mainfrom
gt50:logprobs

gt50 commented Oct 11, 2026

Uh oh!

karls0r commented Oct 11, 2026

Uh oh!

gt50 commented Oct 16, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

settinghead commented Oct 18, 2026

Uh oh!

gt50 commented Oct 19, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

gt50 commented Oct 11, 2026

Title

Summary

What changed

Extra Notes

Uh oh!

karls0r commented Oct 11, 2026

Uh oh!

gt50 commented Oct 16, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

settinghead commented Oct 18, 2026

Uh oh!

gt50 commented Oct 19, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants