[karls0r]

🤖 Contributed by an AI agent (Ren, running Claude) on behalf of @karls0r.

As discussed in Niko1221#1916 (comment): logprobs with "parallel" (batch slots) for one-token decisions, and a 400 for longer replies. One commit on top of logprobs (c8ee496), server only, engine unchanged.

Problem: with "parallel", generate() goes to generate_batched(). A one-token request there is an admission (BGEN, read by the engine as GEN 1 with logprobs=K), so the engine writes the LP line, but _control() had no LP branch and dropped it → "logprobs": null on every reply.

Change (serve/server.py +57/−8):

• _control(): with logprobs asked, a T line is held until the next line; its LP line goes to lp_lines before the token is passed on. Any other line, or 0.5 s of silence, passes the held token on first, so an engine without LP lines only delays a token by one line.
• lp_lines per thread with batch slots, like last (concurrent requests must not share it).
• logprobs + "parallel" + max_tokens != 1 → 400 (fail closed, like logit_bias with batch), because BT lines carry no LP lines. LP for the BT slot phase is left for later.

Tests: serve/test_logprobs_parallel.py (fake engine of test_parallel.py with LP lines behind the real StrataEngine/Service): one decision, four at once (each gets its own entries), not asked (unchanged), longer replies refused, engine without LP lines.

Checked:

• serve tests on this branch: 698 passed, 11 skipped on the build host; on a second Linux box 1 env-only failure in test_keepawake, identical on c8ee496.
• Live on 2× RTX 3090 (CUDA, sm_86) with "parallel": 2, --batch-groups 2: 109 cases serially and two at a time, every reply has logprobs; |Δp| to the run without "parallel" mean 0.007 (max 0.08). The same patch has run in our production since 2026-10-11 (rebased onto v0.1.43.1, where the only conflict is the batch loop now in try/except from Batch slots return HTTP 400 "BGEN: no such free slot" instead of queueing; the affected conversation stays broken Niko1221/Strata#1994).

Not tested: HIP/SYCL builds (change is Python only), Responses and Anthropic APIs.

[gt50]

Thank you, this is a clean fix. Reviewed and merging.

Checked:

• _control(): both callers read it to the end, so a one-token admission (T, LP, DONE) passes the token with its LP line and leaves DONE in the queue. An engine without LP lines passes the token on at the next line or after 0.5 s, as described.
• lp_lines per thread with batch slots, like last. batch = 0 is a class attribute, so the property is safe before __init__ sets it.
• Tests: serve.test_logprobs_parallel, serve.test_logprobs, serve.test_parallel and serve.test_server pass on this branch (332 tests, Windows, Python only; I did not run it against a live engine).

One thing for a follow-up, not blocking: with replicas, ReplicaEngine.batch counts each engine as at least 1 (serve/replicas.py: max(1, ...)). So getattr(svc.engine, "batch", 0) is non-zero even when no replica has batch slots, and logprobs with max_tokens above 1 get the 400 telling the user to "run without parallel" although "parallel" is off. Checking for slots on the replicas' own engines (svc.engine.engines when it has them) would keep the 400 to real batch slots.