Repository navigation
Server wedge: generation stalls mid-decode; /health OK but /slots hangs; needs SIGKILL #27388
Description
Server wedge: generation stalls mid-decode; /health OK but /slots hangs; process must be SIGKILLed
Summary
Under long-running load the server occasionally stops generating mid-decode:
print_timing lines cease, GPU utilization drops to 0%, the process stays alive,
/health still returns {"status":"ok"}, but /slots hangs (no response)
and the client stream goes idle until the client-side timeout. In one instance
the process also ignored SIGTERM and required kill -9.
Observed 3 times in 2 days across two builds and two different models, with
and without MTP, so it does not appear specific to a model, speculative
decoding, or a recent regression.
Environment
- llama.cpp CUDA backend; observed on build 452 (
9558fa4) and build 561 (ee4c505) - 2× RTX 3090 (PCIe/PHB, no NVLink), driver 570.133.07, CUDA 12.8, Linux
- No
Xiderrors indmesgin any incident (not a GPU fault)
Incidents
A) Qwen3.6-35B-A3B Q8_0 (MoE, no MTP), single slot, 32K ctx, long-context
eval task: generation stopped after ~286 tokens (timing lines ceased), client
timed out after 20 min, server log showed cancel task then the process kept
spinning (R state), ignored SIGTERM, needed SIGKILL. Immediately before the
stall: slot get_availabl: … selected slot by LCP similarity, f_sim_best = 0.999.
B) Qwen3.8-27B Q6_K (embedded MTP, --spec-type draft-mtp --spec-draft-n-max 3),
--parallel 2, two concurrent agent workloads: slot 1 decoding normally
(~40 tok/s, n_decoded = 8337), then all timing output stops; /health OK,
/slots hangs, GPU 0%. Restart required. Full journal excerpt (926 lines around
the stall) available on request.
C) Same symptom pattern on build 561 during a client 300 s stream-idle
timeout; restart healed it.
Server invocation (incident B)
llama-server -m Qwen3.8-27B-Q6_K.gguf -mm mmproj-F16.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 \
--jinja -ngl 999 --ctx-size 458752 --parallel 2 \
-ctk q8_0 -ctv q8_0 --cache-reuse 0 --no-context-shift \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
(--cache-reuse 0 --no-context-shift were added after incident A as a hunch;
incident B shows they do not prevent it.)
Notes
/healthresponding while/slotshangs suggests the slots endpoint blocks on
a mutex held by the stuck decode path — a liveness probe on/slots(or a
watchdog around the decode loop) would at least make the state detectable.- Happy to reproduce with added instrumentation and to share full journals.
Activity
loviswaternakamoto commentedon Sep 16, 2026
Reproducing this on CPU / aarch64, with a hard receipt for the
"process ignores SIGTERM" case:
Environment
b10991(llama-server --version: 0.4.1-dev @930e2fa),CPU-only aarch64 (4 hardware threads usable), Ubuntu 24.04, kernel
6.17.0-1020-oracle
UD-Q4_K_XLMTP GGUF, draft heads in-file)--spec-type draft-mtp --parallel 1 --cont-batching -fa on -ctk q8_0 -ctv q8_0 --ctx-size 16384 --threads 3Symptom — identical signature to the OP
Under use, a slot wedges: the first requests answer normally (~1.6 s), then
every completion queues forever.
/healthkeeps returning{"status":"ok"}while/slotsnever answers — matching your mutex reading.The trigger correlates with clients disconnecting mid-generation
(aborted streaming requests).
The SIGTERM receipt (systemd journal)
SIGTERM was ignored for the full 90 s default
TimeoutStopSec; SIGKILLrequired. So the wedge also blocks orderly shutdown, not just new requests.
MTP vs plain decoding, same binary + same model file
--spec-type): a 12-case stability battery passes,including recovery from a deliberately aborted 400-token generation.
--spec-type draft-mtp: the wedge above, within minutes ofinterrupted clients.
Small sample, and given this issue reproduces without MTP on GPU, I read MTP
as an amplifier rather than the root cause — plausibly the abort path racing
draft/decode bookkeeping widens the window enough for a single-slot CPU
setup to hit it quickly.
Your
/slots-not-/healthliveness observation is the practical mitigationon our side: a watchdog probing
/slotswith a hard timeout detects thestate immediately (healthy
/slotsanswers in <1 ms here), which alsosuggests a simple in-server detectability hook if a full fix takes a while.
27388 COMMENT POSTED — the approved CPU/aarch64 reproduction report i…
mab776 commentedon Sep 19, 2026
Got this too, on completely different hardware :
Blackwell + CUDA. With the CPU/aarch64 report above that's three backends now, so it really doesn't look like a GPU thing.
My setup:
Build 11012 (master
4bc272fd7+ #28770 + #28213), RTX PRO 5000 48GB + RTX 5070 Ti 16GB, driver 595.84, CUDA 13.2. Model is Qwen3.8-Flash-Next (qwen4exp, 512-expert MoE) with vision. Client is VS Code Copilot chat over SSE. That's a third spec method too (ngram-mod, where you had draft-mtp and none). To be clear I'm not using MTP, but I am still speculating, so this doesn't contradict the amplifier idea above, it adds to it.It died mid-sentence during a long agentic session, it happened a few times, normally stop + resend message or stop + send "continue" will unfreeze it. This time I was annoyed so I've checked the logs. For this run, the server had been up 14h30 and served 87 tasks, only one never released. The whole log for that task is two lines:
then nothing at all. Same beginning as your incident : it wedges right after a high-similarity cache hit. Mine was 51162 of 51927 tokens served from cache, only 644 left to prefill.
Two things were different from your reports though.
/slotsdidn't hang for me. It answered fine and just showed the frozen state:Polled it for ~5min and nothing moved.
And the process wasn't spinning. Your "incident A" was stuck in
Rand ignored SIGTERM. Mine was the opposite: all 42 threads inS, 0 CPU ticks over 6 seconds (/proc/<pid>/stat), GPU at 0%. Just blocked, not looping.One more thing on the client side. When I looked there was no TCP connection left to the server, VS Code had already timed out and dropped it, but the slot was still
is_processing.--timeoutis 3600 by default so I guess the slot stays held for an hour, and with--parallel 1that's the whole server blocked. I restarted instead of waiting so that part is a guess.The stop + resend thing I mentioned at the start is probably the interesting part : stop is a real abort so the task gets cancelled and the next request goes through fine. That would explain why this is only annoying when you're sitting in front of the chat, and fatal when an agent is running on its own.
Happy to test a patch on Blackwell if that helps.
Hazarth commentedon Oct 5, 2026
Same here. The decode just stops mid generation, no more logs or anything else.
I'm running a Qwen3.6 35B MoE model optimized for single user / local inference only:
Though I've seen this with Qwen 3.8 27B and Cyber-Tiel-Coder-35B-A3B-UD-Q4_K_XL as well.
I've noticed this issue for the first time last week I think. But I'm also not updating llama.cpp all the time so I can't confirm when the problem started exactly.
/health responds fine
/slots responds fine, so this might not be exactly the same problem as OP mentions, or it could be related and slightly unpredictable (memory bashing of some sort?) It's hard to believe there would be two unrelated problems that cause stalls during decoding
If I stop generation and tell the model to continue, it will continue, however it will re-processes the entire context first.
I'm gonna bleeding edge and see if anything in the latest master fixes it, but I don't expect much :D