Repository navigation
--batch hangs during startup on Linux/CUDA (3x RTX 3060, 0.1.43): 100% kernel time, GPU idle, the expert cache is never allocated #2126
Description
Summary
Adding --batch ("parallel": 2 or 4) to a working config makes strata serve hang during startup, before the expert cache is allocated. Without --batch the exact same config loads in ~6 minutes and serves normally.
The hang is not I/O, not swap and not a plain RAM shortage: the host spins at 100% system time with the GPU completely idle (0%), and the kernel reports ~148k failed memory compactions. strace cannot attach (container), but /proc/<pid>/stat shows utime frozen while stime advances 1000 ticks per 10 s.
Environment
| GPU | 3x NVIDIA GeForce RTX 3060 12 GB (sm_86), driver 580.126.20 |
| PCIe | CUDA0 x4 Gen4, CUDA1/CUDA2 x8 Gen4 (measured) |
| CPU | AMD Ryzen 7 3700X (8c/16t), no AVX-512 |
| RAM | 62 GiB (2x 32 GB DDR4 @2133), swap 16 GiB |
| Motherboard | CVN X570 GAMING PRO |
| OS | Ubuntu 24.04.4 LTS, kernel 6.8.0-142-generic |
| Container | Docker, base nvidia/cuda:13.0.0-devel-ubuntu24.04, --ulimit memlock=-1:-1, --shm-size 64m |
| Engine | 0.1.43, built from tag v0.1.43 (src=e568e70371a3d2e0), CMAKE_CUDA_ARCHITECTURES=86, nvcc 13.0.48 |
| Model | Qwen3.8-Flash-Next GSQ-RCO abliterated IQ3_S, native pack iq3_s_abl, 48 blocks |
Config (the args array, identical in every run below unless noted)
--pack /data/packs/iq3_s_abl
--native .../Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00001-of-00002.gguf
--ple-gguf .../Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00002-of-00002.gguf
--expert-profile /opt/strata/data/expert-profile.bin
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp /data/mtp/rt
--max-context 235520 --kv int8 --kv-resident 32768 --remote-expert-opt
--control-vector-scaled .../Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf:1.0
--control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --vision
--vram-reserve-mib 700 --trim-stage-weights
plus "gpu": [0,1,2], "layer_split": "16,32", and the variable under test "parallel": 2.
Symptom
The log stops at exactly the same line every time, and never reaches the expert cache allocation:
strata generate: --batch-groups auto: 2 groups of 1 slot
strata generate: --batch: 2 slot sessions on CUDA0 (0.34 GiB each); 9.12 GiB free
strata generate: --batch: 2 slot sessions on CUDA1 (0.34 GiB each); 9.06 GiB free
strata generate: --batch: 2 slot sessions on CUDA2 (0.34 GiB each); 8.51 GiB free
strata generate: --batch 2: the slot sessions take 0.69 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111), files read in 0.83 s (952 MiB/s)
strata generate: GPU 0: NVIDIA GeForce RTX 3060, compute capability 8.6
<<< hangs here forever >>>
grep -c "expert cache auto" on that log is 0. The next line in a successful (no --batch) start is:
strata generate: expert cache auto: 5.32 GiB free, 700 MiB reserved (+0 MiB for the draft head) -> 1870 slots
Measured while hung:
nvidia-smi: 0% utilisation on all three cards, VRAM frozen at 2578 / 2638 / 4034 MiB (a successful start reaches 11450 / 11354 / 11574 MiB)free: 6.9 GiB available, swap usage unchanged (1437 MiB) — so it is not swapping- Main thread:
State: R (running),wchan=0,/proc/<pid>/syscall=runningon 20/20 samples /proc/<pid>/stat:utimefrozen (398),stime+1001 ticks per 10 s → 100% system time- Worker threads:
futex_wait_queue/do_poll(normal)
Kernel counters at the moment of the hang (/proc/vmstat):
allocstall_movable 608585
allocstall_normal 76862
compact_stall 182850
compact_fail 148075 <-- failed compaction
pgmigrate_success 77873701
pgsteal_direct 68258939
/proc/meminfo: AnonPages 46953728 kB, Shmem 9353772 kB, Mlocked 27356 kB, MemAvailable 6089800 kB.
So the host is repeatedly failing to obtain contiguous memory while the GPU sits idle.
What I ruled out (controlled, one variable at a time)
| Variable | Change | Result |
|---|---|---|
| swap | 4 GiB to 16 GiB (new swapfile, verified with swapon --show) |
identical hang; swap usage never grew |
| VRAM reserve | --vram-reserve-mib 700 to 2000 |
identical hang, same log line |
| batch slots | --batch 2 and --batch 4 |
both hang |
| layer split | 3 stages (16,32, 3 GPUs) and 2 stages (20, 2 GPUs) |
both hang |
--batch off: all four configs load and serve fine (this is the config I run in production, 13 474 expert slots, hit rate 0.996-0.998, 50-57 tok/s decode).
Possibly related
- Engine hangs at layer 48 during decode when VRAM headroom drops below ~500 MiB #2065 - "Engine hangs at layer 48 during decode when VRAM headroom drops below ~500 MiB",
--batch 2. My working start-up reports exactly460 MiB of VRAM free with everything loaded, i.e. below that 500 MiB threshold. My hang is at start-up rather than during decode, but the VRAM-headroom figure matches. - Deterministic deadlock at the completion handshake after the last block: host at layer step 48 with the GPU already rang - 5/5 stalls identical, on 0.1.41 and 0.1.43 (RTX 4080 / Windows / IQ3_XXS native pack / 256K) #2049 - deterministic deadlock "after the last block: host at layer step 48"; this model has 48 blocks too, and the reporter saw it on both 0.1.41 and 0.1.43.
--batch 2 --batch-groups autoon a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906 / Batch slots return HTTP 400 "BGEN: no such free slot" instead of queueing; the affected conversation stays broken #1994 - other--batch/--batch-groupsproblems still open.docs/BATCHING.mdsays "Only a layer split of two stages on two GPUs has been run" - so a 3-stage split with--batchmay be untested territory.
Notes
strace -pfrom the host fails withPTRACE_SEIZE: Operation not permitted(container isolation), so I could not capture the syscall. If there is a recommended way to trace inside the container, I can rerun and attach the output.- Happy to run any build with extra instrumentation (
--stats, debug env vars, a patched binary) on this machine - it reproduces 100% of the time and is otherwise idle.
Activity
Expert arena: MADV_HUGEPAGE is not requested when free 2 MiB blocks c…
Expert arena: say at once that huge pages were skipped (#2126)
Sources: no raw NUL bytes (expert_source.cpp, pinned.cu) + a test tha…
xyzzing commentedon Oct 14, 2026
A gfx1100 (RX 7900 XTX) data point on the gate this added, from a box that runs the 0.1.44.1 line as a daily driver.
The gate reads
free_huge_bytes()(order >= 9 in/proc/buddyinfo) at arena setup, before the arena is allocated — so on a restart it sees the state after the previous instance freed its memory, not the state while an engine runs. On this host (96 GB RAM, 46.84 GiB arena) that means the restrictive branch is essentially never taken:transparent huge pages requested (MADV_HUGEPAGE)transparent huge pages NOT requestedoccurs 0 times in the whole 75,438-line engine logusing 4 KB pageslines sit at log lines 215/263/308, from long-superseded versionsSo the scenario the gate is for is the one where something still holds the memory when the engine starts (an overlapping restart / a second engine), which matches the "wait for an engine that is still exiting" warnings that land in the log under exactly those conditions.
I also measured what the fallback costs, by forcing it with
STRATA_NO_ARENA_THP=1: steady state is unchanged at matched window mix (31.22/31.30 vs 31.17/31.26 ms per verify round, prefill 2154.1 vs 2150.0 tok/s), while startup wall time goes up about +8 s at 1K and +5 s at 128K. So the branch normally taken (THP) is the faster one and the 4 KB fallback is cheap insurance.Caveat, stated plainly: I did not reproduce the compaction stall itself on this box — free 2 MiB blocks at setup never drop below arena/2 here while still fitting the arena, so the condition is not reachable in a measurement window.
Full write-up, including the other new default-ON 0.1.44 path (
STRATA_COMMIT_BATCH): #2361--shared-expert-arena: the engine that loads the file asks for 2 MiB …
arena の MADV_HUGEPAGE は、空いている 2 MiB のブロックが arena の半分に満たないとき頼まない（upstr…
Expert arena: MADV_HUGEPAGE is not requested when free 2 MiB blocks c…
Expert arena: say at once that huge pages were skipped (Niko1221#2126)
Sources: no raw NUL bytes (expert_source.cpp, pinned.cu) + a test tha…
Brightfl commentedon Oct 17, 2026
3x RTX 3060 12 GB on Windows/WDDM: the third stage fails with
CUDA1 free memory: out of memorywhile the same log reports 6.25 GiB free - reproduces on 0.1.43, 0.1.44.1 and 0.1.45Adding a data point from Windows (the report above is Linux/Docker, same card count).
Environment
probes read 0.8 GB/s on the x1 link (
CUDA2 link 0.8 GB/s -> miss cost x16.00) and 3.0 GB/s onthe x4 link, so the unusual link is priced in as usual - see the order-swap row below, which shows
the failure is not tied to that card.
"gpu": [0,2,1],--max-context 245760,--kv q4_0,--vram-reserve-mib 300,"parallel": 2Symptom.
strata servenever reachesready:. On 0.1.44.1 the split search even completes(
layer split auto: K=30,40 - the caches hold 14072 of 24576 profiled pairs (~97.7% of the routed mass), best of 1035 placements), every stage reports real room, and the next allocation fails:The engine contradicts itself two lines apart: 6.25 GiB free -> 0.00 GiB free.
What does not move the number (shortfall stays ~790-831 MiB in every case):
--max-contextparallelremoved)--prefill--vram-reserve-mib[0,2,1]->[1,0,2][1,0,2]the x1 card becomes CUDA0 and CUDA1 is the x16 card, same errorlayer_splitauto-> explicit20,24SeLockMemoryPrivilegegrantedlarge pages refused ... VirtualAlloc error 1314; using 4 KB pagestolarge pages are resident without a lock, failure identicalShortfall by engine: 0.1.43 ~ 790 MiB, 0.1.44.1 ~ 796 MiB, 0.1.45 ~ 831 MiB.
Two cards are fine, same model and flags,
"gpu": [0,2]:ready:in 34-37 s, 36.6 tok/s singlerequest, 19.6 + 21.0 tok/s with two requests in flight. So the model, the config and the disks are
healthy; only the three-stage split fails.
Questions
across adapters?), or is the middle stage's expert-cache allocation a bug?
STRATA_ARENA_PIN_GIB,--vram-elastic, an explicit per-device--expert-cache-device1/2/3,--split-skip-if-fits,--trim-stage-weights?Adamyno commentedon Oct 17, 2026
3× RTX 2080 Ti, native Linux, v0.1.45:
--batch 4starts normallyCounter-data point for the startup hang: on this 3-card Turing rig the
--batchpath does not hang, and it goes past the line where your run stops.Hardware: 3× RTX 2080 Ti 22 GB (sm_75), i9-7980XE, 125.5 GiB RAM, native Linux 7.0.0-38 (not a container), driver 580.178.04, CUDA 13.0.
Config: v0.1.45 (
477ac13c), Flash-Next UD-Q4_K_XL (packud-q4_k_xl), 262,144 ctx,--kv int8 --spec 3 --spec-min-p 0.54 --batch 4,"layer_split": "16,32",--expert-cache auto --trim-stage-weights.The log reaches and passes your stopping line:
The server was ready in 133 s (model load included), no 100% system time, GPU active during the load. So the hang is not inherent to
--batchon three CUDA cards; the remaining candidates are the container's--shm-size 64m/ memlock setup, the 62 GiB RAM with a 48-block IQ3_S pack, or the older 0.1.43 build (ours is 0.1.45).Niko1221 commentedon Oct 19, 2026
Thanks for taking the time to report this.
We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.
To reopen it takes about two minutes:
/reopen. A check runs and either reopens the issue or lists what is still missing.The rules are in CONTRIBUTING.md.