[Engine] 0.2.2 short replies lose 15-30% in serve/server.py, not in the engine (6xV100, sm_70, crossed test) #3667

Area

[Engine] (speed, kernels, models, the server and API)

Desktop app, Terminal or source?

Source (a serve/server.py start on a source-built engine)

What happened

On a 6× Tesla V100-PCIE-32GB box (sm_70), 0.2.2 loses 15–30 % on short replies against 0.1.44.1 — but a crossed test shows the engine is not the cause: the 0.2.2 serve/server.py flow is. The same 0.1.44.1 engine binary is fast under the 0.1.44.1 server and slow under the 0.2.2 server, and the 0.1.47 engine is fast under the 0.1.44.1 server.

This extends @t-sm0's finding in #3252 (A100: the same 0.1.44.1 binary measured 111.3–119.3 tok/s through the 0.1.x flow vs 92.0 through the 0.2.x terminal flow) to sm_70 and to the spec/short-reply path, with the engine and server factors separated.

Crossed 2×2 (same box, pack and args; one variable changed at a time). Probe = the chat section of strata_bench.py: the three questions of tools/ab_engine.py, greedy, max_tokens 100, thinking off, 3 runs each (medians).

• Changing the server moves the number; changing the engine does not.
• The crossed arms used a slightly longer prompt tag (2 tokens), which shifts q2/q3 a little; q1 is the regression-sensitive one and is unaffected.
• Engine argv is byte-identical in the two arms that differ only by server (checked with ps), including --spec 3 --spec-min-p 0.5 --prompt-cache 24 --prompt-cache-every 4096 --pipeline-windows 2 --layer-split auto.

Engine's own per-request lines — same 46-token prompt, same acceptance, only the server differs (the engine cancels out):

# 0.1.44.1 server + 0.1.47 engine
strata serve: prompt 46 tokens = 0 reused + 46 read in 461 ms (99.8 tok/s), 100 generated in 744 ms (134.4 tok/s), drafts accepted 63 of 72, 1 checkpoints
strata serve: prompt 46 tokens = 0 reused + 46 read in 461 ms (99.8 tok/s), 100 generated in 740 ms (135.1 tok/s), drafts accepted 63 of 71, 1 checkpoints

# 0.2.2 server + 0.1.44.1 engine
strata serve: prompt 46 tokens = 0 reused + 46 read in 466 ms (98.8 tok/s), 100 generated in 1037 ms (96.4 tok/s), drafts accepted 62 of 70, 1 checkpoints
strata serve: prompt 46 tokens = 0 reused + 46 read in 466 ms (98.7 tok/s), 100 generated in 1036 ms (96.5 tok/s), drafts accepted 62 of 70, 1 checkpoints

Draft acceptance is 89 % in both (63/71 vs 62/70); only the per-window cost changed (740 ms vs 1036 ms for 100 tokens).

What is unchanged

• Prefill 4K/16K/32K: 1288.2 / 2440.8 / 3345.7 → 1304.2 / 2445.8 / 3375.0 tok/s (noise).
• First token 4K/16K/32K: 3.18 / 6.66 / 9.74 → 3.12 / 6.66 / 9.64 s.
• Single-stream 256-token generation: 79.5 → 79.2 tok/s (client-side, n=10) — so it is not a fixed per-request cost.
• Outputs are bit-identical: 4/4 greedy prompts, same sha256 and completion_tokens across both versions.
• STRATA_NUMA_POOL=0 changes nothing (the box has 2 NUMA nodes, so 0.2.2 turns the split on).

Question: is there a knob in 0.2.2 that restores the 0.1.x engine-driving path, or a known change in serve/server.py that adds this per-request cost on short replies? We stay on 0.1.44.1 for now (0.2.2's engine itself is fine) and can run any A/B or a per-commit server.py bisect on this box — both engines are already built and the machine is dedicated to this model.

Smallest steps or command that reproduce it

1. Build both engines from source for sm_70 (-DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON): tag v0.1.44.1 and tag v0.2.2.
2. Start serve/server.py from one tag with the production config, swapping only the config's exe (and vision.exe) to the other engine, and vice versa.
3. Run python3 tools/probe_chat_decode.py --runs 3 (or the chat section of strata_bench.py) against each arm; compare q1 and the engine's 100 generated in … ms line.

No special tooling beyond the repository is needed.

Reproduction status

Reproduced locally

Strata version

Strata 0.2.2 · engine 0.1.47 (011c9e60) — reference arms on 0.1.44.1 (296b59d2, engine 0.1.44.1)

Graphics card, memory and system

6× Tesla V100-PCIE-32GB (sm_70, 32 GB each) · 502 GiB RAM · Ubuntu 22.04.3, kernel 5.15, driver 580.178.04, CUDA 12.9

Model, size and settings

Qwen3.8-Flash-Next Unsloth UD-Q4_K_XL (native experts pack + MTP rt) · 262144 context · --kv int8 · pictures on · --spec 3 --spec-min-p 0.5 · 6-GPU layer split (--layer-split auto) · 4096 expert slots per card

Logs

strata generate: expert cache auto: 24.87 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 4096 slots
strata generate: layer split: CUDA0 runs layers 0-7
strata generate: layer split: CUDA1 runs layers 8-15, expert cache 4096 slots (11.72 GiB), 4096 of its 4096 profiled pairs; slot 0 verified
strata generate: expert cache 4096 slots, 12.40 GiB of VRAM
strata serve: prompt 46 tokens = 0 reused + 46 read in 466 ms (98.7 tok/s), 100 generated in 1036 ms (96.5 tok/s), drafts accepted 62 of 70, 1 checkpoints