Repository navigation
[Bug]: d3 decision requests keep ~13 CPU cores busy (thread pool spin-waits); no way to cap threads #882
Description
Environment: FastFlowLM 1.0.11, Windows 11 Pro 26200, Ryzen AI 9 HX PRO 475 (12 cores / 24 threads, XDNA 2), NPU driver 32.0.20102.3930. flm serve --decision d3:nano (no chat model loaded).
What we see: back-to-back /v1/systemone requests run at about 0.40 s each with the NPU busy roughly 56% of the time, but flm.exe uses 12.9 cores (54% of all logical CPUs) while doing so: about 5.3 CPU-seconds per 0.4 s decision, 37% of it kernel time, spread over about 25 threads at about 44% each. At idle the server uses 0 CPU. A chat model on the same server (qwen3.6-moe:35b-a3b decode) uses only about 4% CPU, so this is specific to the decision path.
Restricting CPU affinity shows about half of it is waiting, not work:
flm.exe CPU affinity |
Decision median | CPU used | Kernel share |
|---|---|---|---|
| 24 logical CPUs (default) | 399 ms | 12.9 cores | 37% |
| 8 logical CPUs | 426 ms (+7%) | 6.4 cores | 68% |
| 2 logical CPUs | 822 ms | 1.8 cores | 40% |
d3_npu.dll contains CPU matmul paths (cpu_mm, cpu_func, lm_head, AVX-512) and sizes its pool from hardware_concurrency. It looks like the worker threads spin and yield while the NPU runs.
Impact: a decision stream next to other CPU work slows both. With llama.cpp chat on the CPU at the same time, decisions went from 400 to 527 ms (+32%) and chat dropped 13%.
Requests:
- An option or env var to cap the decision path's CPU threads (for example
--threadsorFLM_CPU_THREADS). - Block on the NPU completion (fence or event) instead of spin-waiting, or spin briefly and then sleep.
Happy to re-test a build on this machine.
Activity
chetan-metrum-ai commentedon Oct 30, 2026
Same pattern, much stronger, with qwen3.8-mtp:27b on FLM 1.0.11 (same machine). One 32-token answer at
temperature: 0, freshly started server each time:flm.exeCPU affinityThe NPU stays 95–97% busy in every case, but restricting the process to fewer CPUs makes decode and first token faster, so the extra host threads are contending rather than doing useful work. The 27B module seems to create hundreds of threads (197–488 in a fresh process, against 26 for qwen3.5:9b). A cap on host threads, or blocking waits instead of spinning, would likely help the 27B more than d3.
tawei-amd commentedon Oct 30, 2026
Thanks for the detailed report and the affinity tables, they pointed us straight at the cause. We are working on a fix and are aiming for the next release.
What we found: the host-side loops of the d3 engine run on OpenMP, and an OpenMP team keeps its idle workers spinning through every NPU dispatch (a decision alternates short host loops with blocking NPU waits, once per layer). Each calling thread of the server also gets its own team, so the thread count grows well past one pool (about 120 threads on our test machine). We reproduced the CPU burn on Linux with a single calling thread (about 11.6 cores busy for a 390-token request, with the same latency as 4.7 cores). We have not been able to test Windows, and your 37% kernel time suggests the Windows OpenMP runtime spins harder than libgomp.
The fix replaces OpenMP in d3 with one shared worker pool whose idle workers spin for a few microseconds and then sleep, so it applies to every d3 model. We will also keep the thread count adjustable. Please note that about 4 of those cores are real parallel work, so we do not expect CPU use to fall to the single-thread level.
Since you offered to re-test: once a build is ready we would be glad if you could run it on your machine and compare it with your numbers above. We will post here when it is available.