flm serve crashes with glibc heap corruption (double free / unaligned tcache) under concurrent request bursts — Linux / XDNA2, v0.9.43

Environment


FLM version	v0.9.43 (`flm --version` → `FLM v0.9.43`; `/api/version` → `{"version":"0.9.43"}`) — latest release
OS	Ubuntu 24.04, glibc 2.39 (`GLIBC 2.39-0ubuntu8.7`)
CPU/NPU	AMD Ryzen AI MAX+ 395 "Strix Halo", NPU aie2p (6×8), single `/dev/accel/accel0`
NPU firmware	1.1.2.65
XDNA driver	amdxdna 2.23.0_20260413 (XRT 2.23.0)
Model	`qwen3-it:4b` (Qwen3-4B-Instruct-2507-NPU2, Q4_1, 32k ctx)
Launch flags	`flm serve qwen3-it:4b --pmode turbo --ctx-len 32768 --prefill-chunk-len 8192 --port 52625`
Linked libs	`libxrt_coreutil.so.2`, `libboost_program_options.so.1.83.0`, `libgomp.so.1` (OpenMP)

Symptom

flm serve (running as a long-lived server) aborts intermittently with glibc heap-corruption diagnostics on stderr, immediately after the model-list banner, then the supervisor restarts it (each restart reloads the NPU model). Observed signatures:

double free or corruption (out)
corrupted size vs. prev_size while consolidating
malloc(): unaligned tcache chunk detected
[FLM]  Using custom model list path: .../share/flm/model_list.json

The abort is a SIGABRT from glibc's allocator, not an OOM (no exit code 137, memory was not exhausted) and not a config/port error. It occurred repeatedly during a burst of ~10 concurrent clients hitting the server in a short window, then stopped once traffic serialized (the current process has since been stable for 8+ hours under light load).

Trigger (best hypothesis)

Concurrent-request bursts. The crashes clustered when ~10 independent clients issued /v1/chat/completions requests near-simultaneously. Under steady/serialized traffic we have not reproduced it.
--ctx-len 32768 appears within limits (the model advertises 32k; community reports run --ctx-len 32000 fine), so context length alone does not look like the cause.
--prefill-chunk-len 8192 (2× the 4096 default) and --pmode turbo are the most aggressive settings in play and are plausible contributors to the timing window, but we have not isolated a single flag.
Probable code area: the v0.9.41 changelog ("fixes redundant tool schema injection in cached conversations") and v0.9.43 ("makes cached conversations more reliable by preserving complete message histories") indicate the cached-conversation / message-history subsystem was under active change around this build. A double-free / corrupted size vs prev_size / unaligned-tcache signature is consistent with a string/vector reallocation race in that path when multiple requests are in flight through the server's request queue while OpenMP worker threads are active.

What would help

Whether the internal request queue (npu_request_queue_, max_connections_ in src/server/server.hpp) fully serializes access to the conversation/KV-cache structures, or whether connection-accept / cache-reuse can run concurrently with an in-flight generation.
Whether building/running under a hardened allocator (e.g. MALLOC_CHECK_=3, or a jemalloc/tcmalloc preload) reproduces or pinpoints the corruption site.
Any known concurrency fix landing after v0.9.43.

Reproduction (approximate)

flm serve qwen3-it:4b --pmode turbo --ctx-len 32768 --prefill-chunk-len 8192 --port 52625
Fire ~10 concurrent /v1/chat/completions requests (multi-turn / with prior message history) in a tight window, repeat under sustained load.
Observe intermittent double free or corruption / unaligned tcache aborts on the server.

(We could not do controlled flag-isolation on our host because it has a single NPU that a live production tier is consuming — isolating would require a maintenance window or a second NPU box.)

Empirical reproduction data (16-concurrent burst matrix)

We reproduced this under a controlled drain-window (single-NPU host, serve.sh-driven, 16-concurrent × 8 waves = 128 requests per config):

Config	flags	crashes	HTTP 200	server-death errors
A (aggressive)	`--pmode turbo --prefill-chunk-len 8192`	1	57	28
C (aggressive + arena)	above + `MALLOC_ARENA_MAX=1`	1	8	30
B (conservative)	`--pmode performance --prefill-chunk-len 4096`	0	80	5

Concurrency-threshold: config A at 12-concurrent = 0 crashes; corruption appears ~16+ concurrent → confirms the concurrent-request-burst trigger.
Conservative flags help (crashes 1→0, server-deaths 28→5) at negligible latency cost (NPU-bound, ~27-29s p50 either way — the NPU serialization is the bottleneck, not clock).
MALLOC_ARENA_MAX=1 ruled out (aggressive+arena still crashed) → not a multi-arena glibc race; corruption tracks prefill-buffer size / turbo.

Probable site: the v0.9.43 KV-cache/conversation-handling redesign (token-level diff-and-splice over a retained mutable message-history buffer, plus reasoning-content isolation) — this is exactly the realloc surface a double-free / corrupted-size / unaligned-tcache points at, raced under the concurrent request queue + OpenMP threads. v0.9.44 ("Portable Linux Support") does not touch this path.

n=1 crash-run per config (one 128-req burst each) — direction consistent + mechanistically sensible, not high-n statistical.

Adding a Windows / XDNA2 data point and a more minimal reproduction that isolates the trigger to a single concurrent request hitting an in-flight streaming generation (not a 10-client burst). Likely the same root cause as this issue, and it matches the Windows-under-load symptom in #555.

Environment

OS: Windows 11 Pro (build 26200)
FLM: v0.9.45 (latest release)
CPU/NPU: AMD Ryzen AI 9 HX PRO 370 (Strix Point, XDNA2) — a different NPU than this issue's Strix Halo / aie2p
NPU driver: 32.0.20102.3930 (current driver shipped in Adrenalin 26.6.x — see "driver ruled out" below)
Model: qwen3.6-moe:35b-a3b
Launch: flm serve qwen3.6-moe:35b-a3b (defaults; --preemption = 0)

Minimal reproduction (2 requests, not a burst)

flm serve qwen3.6-moe:35b-a3b
Send request A: POST /api/chat with "stream": true. While A is still generating tokens…
Send request B: any second POST /api/chat (streaming or not, with or without tools).
FLM starts a checkpoint to preempt A, then the flm serve process dies (exit code 5 / access violation).

What isolates the trigger

Only one concurrent request is needed — not a burst of ~10. The precondition is that request A is stream:true and mid-generation.
Two non-streaming (stream:false) concurrent requests are safe — they queue (NPU busy, request queued (1/10)) and both complete. The crash is specific to preempting an in-flight streaming generation.
tools is not required; it also crashes with plain user messages.

The crash always coincides with the checkpoint/preemption path. Log tail right before death:

[🕒] NPU busy, request queued (1/10): POST /api/chat
[FLM] Creating checkpoint at context length 9, general_token_idx 9
[FLM] Total images: 0
<flm serve exits, code 5>

--preemption defaults to 0, yet checkpoints are still created for this model and the crash still happens. Setting --preemption explicitly (0 or 1) does not change it.
Model-dependent: qwen3.6-moe (which checkpoints) crashes; gemma3:* (which never checkpoints) survives the identical concurrency pattern. That points at the checkpoint/restore path rather than the socket/queue layer.
Driver ruled out: NPU driver 32.0.20102.3930 is the current Adrenalin 26.6.x driver, and FLM v0.9.45 is the latest release — so this is not a stale-driver / stale-build artifact.

Why it bites in normal use

Open WebUI sends the chat as stream:true and concurrently fires its background Title / Tags / Follow-up task-model calls against the same model → a guaranteed second /api/chat during streaming → reliable crash. The only thing that fully stops it today is disabling those background tasks so exactly one request reaches FLM per message.

Question

Is there a serve flag to force strict serialization (finish the active generation before admitting the next request), or to fully disable checkpoint/preemption for MoE models? If not, would it be feasible for request admission to hold a new /api/chat until the current streaming generation releases the NPU, instead of checkpointing mid-stream? That would make any client (Open WebUI, Frigate in #555, etc.) safe without client-side workarounds.

perf: drop the 333ms NPU cooldown between queued requests

Independent confirmation + minimal repro recipe (client-cancel path, not raw concurrency). Crashes on v1.0.5 and v1.0.6 alike.

Setup: flm serve qwen3.6-moe:35b-a3b --host 0.0.0.0 --port 52625 --ctx-len 65000 — Ubuntu 26.04.1, kernel 7.0.0-31, AMD Ryzen AI 5 340 (Strix / XDNA2), 91 GB RAM. Tested --pmode turbo and --pmode performance (both crash, so power mode is not the variable).

What we see. Our client (a game agent) sends one chat/completions request per action with a different prompt each time, so nearly every request is a Prompt cache miss + Clearing context.... When the agent times out mid-generation, the disconnect lands in the cancel path:

[🔒 ]  TCP connection closed - Remote: 127.0.0.1:34088
[FLM]  Prompt cache miss.
[FLM]  Clearing context...
[🔴 ]  Client disconnected; cancelling active request
[FLM]  Start prefill...
[FLM]  Creating checkpoint at context length 0
[❌ ]  Prefill Cancelled!
[🟡 ]  Dequeuing NPU request (1 remaining)...
[🔒 ]  TCP connection closed - Remote endpoint unavailable
[🔴 ]  Client disconnected; cancelling active request
[FLM]  Prompt cache miss.
[FLM]  Clearing context...
double free or corruption (out)

Under organic agentic load: 2 crashes per ~10 h on v1.0.5 (double free or corruption (out) and malloc(): unaligned tcache chunk detected).

Minimal repro (NEW — sub-minute, no hours of soak)

While the server is warm, fire overlapping long-prompt POSTs where most clients disconnect mid-prefill:

URL=http://127.0.0.1:52625/v1/chat/completions
BIG=$(python3 -c "print('Describe the geology of deepslate layers in exhaustive detail. ' * 400)")
for i in 1 2 3 4; do
  curl -s -o /dev/null --max-time $((2 + i)) -X POST "$URL" -H 'Content-Type: application/json' \
    -d "{\"model\":\"qwen3.6-moe:35b-a3b\",\"messages\":[{\"role\":\"user\",\"content\":\"$BIG round $i\"}],\"max_tokens\":64}" &
done
curl -s -o /dev/null --max-time 30 -X POST "$URL" -H 'Content-Type: application/json' \
  -d "{\"model\":\"qwen3.6-moe:35b-a3b\",\"messages\":[{\"role\":\"user\",\"content\":\"$BIG slow\"}],\"max_tokens\":200}" &
wait

Repeat 3 rounds spaced ~5 s. On v1.0.6 (released Sep 18) this aborted on round 1, in under a minute, with the identical double free or corruption (out) after Prefill Cancelled! → dequeue → second disconnect.

Version matrix (single machine, single config)

Version	Traffic	Outcome
1.0.5 `--pmode turbo`	agent soak ~7 h	crash (double free / unaligned tcache)
1.0.5 `--pmode performance`	agent soak	crash (2× in 10 h)
1.0.6 `--pmode performance`	10 min soak + cancel burst above	crash in <1 min

v1.0.6's changelog contains no heap fix, and we confirm the bug is unchanged.

Suspected root cause

Race between the cancellation path and the queue consumer: on Client disconnected; cancelling active request the cancelled request's buffers appear to be freed, but the dequeued-then-immediately-abandoned work still touches them (note the second TCP connection closed arriving between Prompt cache miss and Clearing context...). The Creating checkpoint at context length 0 right before Prefill Cancelled! suggests the cancel runs against a degenerate/zeroed context — plausibly where the corrupt free originates.

Questions: Is concurrent cancel + dequeue safe at all? Does the queue assume clients never disconnect? Happy to validate an ASAN/sanitizer build against the recipe above — the repro is now deterministic, which should make this much easier to chase than our original hours-long soak.

Navigation Menu

Repository navigation

flm serve crashes with glibc heap corruption (double free / unaligned tcache) under concurrent request bursts — Linux / XDNA2, v0.9.43 #608

Environment

Symptom

Trigger (best hypothesis)

What would help

Reproduction (approximate)

Empirical reproduction data (16-concurrent burst matrix)

Environment

Minimal reproduction (2 requests, not a burst)

What isolates the trigger

Why it bites in normal use

Question

Minimal repro (NEW — sub-minute, no hours of soak)

Version matrix (single machine, single config)

Suspected root cause

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

flm serve crashes with glibc heap corruption (double free / unaligned tcache) under concurrent request bursts — Linux / XDNA2, v0.9.43 #608

Description

Environment

Symptom

Trigger (best hypothesis)

What would help

Reproduction (approximate)

Activity

jiribrachtel commented on Jul 10, 2026

Empirical reproduction data (16-concurrent burst matrix)

glazperle commented on Jul 15, 2026

Environment

Minimal reproduction (2 requests, not a burst)

What isolates the trigger

Why it bites in normal use

Question

elmapache1620 commented on Sep 20, 2026

Minimal repro (NEW — sub-minute, no hours of soak)

Version matrix (single machine, single config)

Suspected root cause

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions