[Engine] T=0 answers are not reproducible: byte-identical requests differ within one session (0.1.52 and 0.1.54) #4673

Summary

temperature: 0 on the local server does not give the same answer twice, on two engine builds, at text level and in the final answer. And it is not the sampling: at T=0.8 the same seed twice gives different continuations, so the OpenAI seed does not pin a response here either (parsed and forwarded, never rejected; no effect visible). This is the Windows/CUDA server path, engine 0.1.52 (release 0.2.7.1) and 0.1.54 (release 0.2.8). It matters because the benchmark culture here compares outputs for equality ("outputs differed, quality not evaluated" in a #4522 test right now): on this engine an output-equality check proves nothing, and a single wrong answer can be the engine, not the model, not the PR.

Area

[Engine] (server and engine behaviour; the mechanism stays open - suspects below, no kernel claim)

Desktop app, Terminal or source?

Terminal install, engine started from a run config, Windows 11, 0.1.52-windows-x86_64-cuda-cuda13 and 0.1.54-windows-x86_64-cuda-cuda13.

What happened (measured)

Byte-identical POST /v1/chat/completions bodies, back to back in one session:

• temperature: 0: 4/4 prompts differ on 0.1.54 and 4/4 on 0.1.52; across two same-config sessions the final answer of a logic puzzle flips (verbatim pair below).
• temperature: 0.8 + same seed: 1234 twice: differs (content and reasoning).
• Decisive pair: a prompt asking for one 16-character hex string at temperature: 0.9, seed: 999 twice gave 3f9ab71d0e4c2a85 then 3f9a2bc7d4e18f60.
• logprobs: true works (per-token entries come back) - it is a working feature, not a stub; only its top_logprobs was empty in our replies.
• enable_thinking: false (the server's off-switch, distinct from a plain thinking field the server does not read): the template's empty-thinking path is taken, yet the model still writes reasoning (the reasoning_content reply field, ~700 tokens at max_tokens: 300), and within one batch of five byte-identical T=0 requests the answer field came back empty all five times while the batch before it answered - the emptiness itself varies.

Smallest steps or command that reproduce it (~30 s, paste-ready)

PowerShell (Windows), against a loaded model - set $model to the name shown by GET /v1/models:

$model  = "REPLACE_ME"
$prompt = "Five cards A, B, C, D, E occupy slots 1 to 5 in some order. A is immediately left of B. C is immediately right of B. D is immediately left of A. E is not in slot 5. Name the card in slot 3, then justify in exactly two sentences."
$url    = "http://127.0.0.1:8080/v1/chat/completions"
$body   = @{ model = $model; messages = @(@{ role = "user"; content = $prompt });
            max_tokens = 1500; temperature = 0 } | ConvertTo-Json -Depth 5
$out = 1..3 | ForEach-Object {
  (Invoke-RestMethod $url -Method Post -ContentType "application/json" -Body ([Text.Encoding]::UTF8.GetBytes($body))).choices[0].message.content
}
if ($out[0] -eq $out[1] -and $out[1] -eq $out[2]) { "PASS: three T=0 runs identical" }
else { "DIFFER at T=0:"; $out }

bash/macOS/Linux (needs jq):

MODEL=REPLACE_ME   # name shown by: curl -s localhost:127.0.0.1:8080/v1/models
PROMPT='Five cards A, B, C, D, E occupy slots 1 to 5 in some order. A is immediately left of B. C is immediately right of B. D is immediately left of A. E is not in slot 5. Name the card in slot 3, then justify in exactly two sentences.'
for i in 1 2 3; do
  curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}],\"max_tokens\":1500,\"temperature\":0}" \
  | jq -r '.choices[0].message.content' | sha256sum
done

Three identical hashes = deterministic. max_tokens: 1500 is deliberate: reasoning models spend the budget before the answer. To check the seed claim, add seed = 1234; temperature = 0.8 and compare the reasoning_content too (a content-only compare can match on the answer while the reasoning diverges).

What it looks like when it bites (verbatim, engine 0.1.52, same run config, two sessions)

Session A1 (two identical runs, both this):
The card in slot 3 is B. The constraints force the consecutive block D-A-B-C, and since E cannot occupy slot 5, this block must start in slot 2, placing B in slot 3.
Session A2:
The card in slot 3 is A. The constraints force the consecutive block D-A-B-C, and since E cannot be in slot 5, the block must occupy slots 2 through 5, leaving A in slot 3.
A1 is wrong and reads fine (block at 2-5 means slot 5 is E, which the puzzle forbids).

Reproduction status

Reproduced locally, repeatedly, two engine builds: 4/4 within-session (both builds), the final-answer flip above, the seed pair and the hex-string pair today on 0.1.54.

Graphics card, memory and system

2x RTX 4090 (24 GB), TR3970X, 128 GiB RAM, Windows 11. Our config is a split (--layer-split 22, --pipeline-windows 2, --spec 4 with the MTP drafter); NOT tested: single-card, --spec 0, --pipeline-windows 0, HIP, Metal. Suspects, as suspects only: the speculative accept path at T=0, the second pipeline window / continuous batching, KV-cache state carried from the previous request, non-deterministic kernels.

Strata version

Control/CLI 0.2.8; engine 0.1.52-windows-x86_64-cuda-cuda13 (release 0.2.7.1) and 0.1.54-windows-x86_64-cuda-cuda13 (release 0.2.8) - identical behaviour.

Model, size and settings

Qwen3.8-Flash-Next UD-Q4_K_XL native pack; requests set only the fields named per test above; server defaults otherwise; logprobs returned entries, so the requests were not silently dropped.

What we checked before blaming the model or a knob

Bodies byte-identical (the script re-sends the same string); the seed field is parsed and forwarded both in the server (serve/server.py around the sampling-key build) and in the engine (seed key in the request parse) - static reading of the 0.2.8 source, unconfirmed static analysis; the hex-string pair above is the executed proof that the seed changes nothing; a trim on/off spot-check differs between identical-config sessions just as much as between trim arms - the knob under test does not need to exist for outputs to differ.

What we would like

1. One docs line per build: what temperature: 0 guarantees on this server (greedy is not the same as reproducible) - and the same line for seed (accepted; no reproducibility promised).
2. If reproducibility is meant to be possible: where the variance is expected from, so reports can be triaged (the spec path? batching?).
3. If others run the script and reply with a row - engine / backend / single|split / spec on|off / result PASS|DIFFER - we will know quickly whether this is our split path or universal.

(AI-assisted: the model served by this repo's engine ran the probes and drafted this; the request/reply pairs above are ours, verbatim. An earlier revision of this issue called seed "ignored" before we read the source; corrected here, same day.)