Repository navigation
NUMA pool split on a two-socket Xeon (2x 6226R, 2x R9700): on/off numbers, and the "arena path" limitation #3708
Description
From Claude after some testing, hopefully helpful:
NUMA pool split on a two-socket Xeon (2x 6226R, 2x R9700): on/off numbers, and the "arena path" limitation
The 0.2.2 notes ask two-socket owners to send the start-up lines and a decode run with and without STRATA_NUMA_POOL=0. Here is that pair. Short version: with the split on, the engine says it cannot place this model's experts per node, 3.7K-prompt prefill is 7% slower and decode is unstable (40-60 tok/s run to run); with the split off, 0.2.2 is at parity with 0.2.1. I kept STRATA_NUMA_POOL=0.
Machine
- 2x Xeon Gold 6226R (16c/32t each, 2 NUMA nodes), 128 GB ECC. IOMMU on, passthrough.
- 2x Radeon AI PRO R9700 (gfx1201), one per socket, Resizable BAR on (32 GB BAR).
- Linux 7.0.14 host (Proxmox), Strata runs inside an LXC with 32 cores whose CPU set spans both sockets (
Mems_allowed_list: 0-1), 75 GB RAM cap. - Source build (
gitcheckout atv0.2.2,./setup.sh --update) against the system ROCm 10.0 (hipBLASLt 1.4.1, calibrated gfx1201 table,STRATA_HIP_WMMA=1). Engine 0.1.47. - Model: Qwen3.8-Flash-Next IQ3_XXS (SC117 GSQ-RCO abliterated files,
--gguf-dir), layer split auto over both cards (layers 0-23 / 24-47), 100% of the experts resident in VRAM, MTP on,--kv int8, 64K context, CPU picture reader.
Start-up lines
0.2.1 (control):
CPU pool: 10 workers by default (21 physical cores, 32 logical processors; bounded by the host's CPU socket)
10 pool workers on logical processors 6,8,9,10,11,12,13,15,35,39, the host thread on 1
0.2.2, default (split on):
strata cpu pool: NUMA pool on: 2 NUMA nodes found, one slice of the pool per node (STRATA_NUMA_POOL=0 turns it off)
strata cpu pool: NUMA split on: slice 0 = node 0 (11 threads), slice 1 = node 1 (9 threads)
CPU pool tasks/phase: 60 (automatic), participating threads: 20
CPU pool: 19 workers by default (21 physical cores, 32 logical processors; bounded by every CPU socket (STRATA_NUMA_SPLIT))
NUMA split on, but this model's experts come from the arena path: memory is not placed per node (known limitation)
CPU workers on logical processors 6,19,8,21,9,24,10,25,11,26,12,30,13,48,15,49,35,60,39 (19 workers, one per physical core; host on 1)
0.2.2, STRATA_NUMA_POOL=0:
strata cpu pool: NUMA pool off (STRATA_NUMA_POOL=0): 2 NUMA nodes found, the pool runs as before
CPU pool tasks/phase: 33 (automatic), participating threads: 11
CPU pool: 10 workers by default (21 physical cores, 32 logical processors; bounded by the host's CPU socket)
CPU workers on logical processors 6,8,9,10,11,12,13,15,35,39 (10 workers, one per physical core; host on 1)
Layer split, expert placement and VRAM free (4586 / 4160 MiB with everything loaded) were identical in all three.
Numbers
Fresh engine start for every variant, 2 warm-ups then 5 runs, greedy, 512 output tokens, fresh prompts (a nonce defeats the prefix cache). Medians, tok/s:
| 0.2.1 | 0.2.2 split off | 0.2.2 split on | |
|---|---|---|---|
| 3.7K prompt, prefill | 1810 | 1830 | 1697 |
| 3.7K prompt, decode | 56.3 | 53.9 | 56.7 |
| 16K prompt, prefill | 2944 | 2907 | 2839 |
| 16K prompt, decode | 58.0 | 57.7 | 56.5 |
3.7K decode per run: 0.2.1 58.6 / 56.3 / 55.2 / 55.4 / 58.8; split off 52.4 / 53.9 / 52.5 / 57.6 / 59.2; split on 40.0 / 46.5 / 59.5 / 56.7 / 57.7 (warm-up 2 was 28.3). An earlier 3-run set with the split on sat at 45-47 throughout. Load time: 70 s off, 90 s on.
Draft acceptance was 49-57% in every variant, so the decode spread with the split on is not an acceptance effect.
Reading
With 100% of the experts in VRAM the CPU pool has little decode work here, so the split cannot gain much on this setup, and the "arena path: memory is not placed per node" line says it cannot do the placement that would make the second socket's workers useful. What is left is 19 workers instead of 10 reading memory that is partly remote, which shows up as the slower 3.7K prefill and the unstable decode. A model with experts partly on the CPU might read differently; I have not tested one.
Happy to run anything else on this box (e.g. a pack with experts partly in RAM, or STRATA_NUMA_SPLIT=1 with other pool sizes).
Activity
github-actions commentedon Oct 22, 2026
Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:
Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment
/reopen.Details: CONTRIBUTING.md
baptisterajaut commentedon Oct 22, 2026
Woopsie, that one is mine: the arena-path line comes from my NUMA-split branch (2496795), written when the split was opt-in. Back then only someone who asked for the split got it, so a warning was enough. With 0.2.2 turning it on by itself, the arena path now gets the split and the both-sockets worker count (19 instead of 10 on your box) with no per-node placement behind it: half the workers read the other socket's memory and every layer's pool barrier crosses the UPI, for very little CPU work with every expert in VRAM. Your 7 % slower 3.7K prefill and the 40-60 tok/s decode spread would fit that.
The fix keeps the automatic pool off on the arena path: the one-socket worker count, and a log line that says why;
STRATA_NUMA_SPLIT=1still forces the split. I'm checking it on my box against the same arena-path case before opening the PR, and will link it here. Until thenSTRATA_NUMA_POOL=0is the right setting for your setup.Once the PR is up, your start-up lines and a 3.7K run with it would confirm it on your box, if you have time.
baptisterajaut commentedon Oct 22, 2026
The fix is up: #3730 (branch
fix/numa-split-arenaon baptisterajaut/Strata, on top of v0.2.2).I could not check the arena path itself: my box has 64 GB, and the only models on it (Flash-Next IQ4_XS, about 50 GiB of experts) get OOM-killed while the arena loads, before the pool is even built. Your setup is the case it was written for, so a run there would be the real test.
If you have time, a build of that branch without
STRATA_NUMA_POOL=0should printNUMA pool off: this model's experts come from the arena path, which does not place memory per nodeat start, with the 10 workers of the split-off case. Your 3.7K run with it, against your split-off numbers (1830 prefill, 53.9 decode), would tell whether that is enough on your box.Cams-Junk commentedon Oct 23, 2026
Tested #3730 (8a04c14, built from source, HIP/ROCm 10.0) on 2x Xeon Gold 6226R (2 NUMA nodes), 2x R9700 32 GB, Qwen3.8-Flash-Next IQ3_XXS, layer split 0-23/24-47, every expert in VRAM, LXC with CPUs on both sockets.
Start-up without
STRATA_NUMA_POOL:All 10 workers and the host thread are on node 0, and the pool lines are identical to a
STRATA_NUMA_POOL=0start.Medians of 5 (2 warm-ups), fresh start each, ~3.7K and ~16K prompts, 512 greedy tokens, tok/s:
3.7K decode per run: no override 60.2 / 57.1 / 58.7 / 58.7 / 58.5; STRATA_NUMA_POOL=0 26.6 / 49.2 / 43.3 / 53.4 / 58.6. Both rows run the same one-socket pool, so the gap between them is this box's fresh-start variance, not the patch.
The default now picks the one-socket pool on the arena path by itself and is on par with 0.2.2 +
STRATA_NUMA_POOL=0. Looks good from here.Tested #3730 (8a04c14, built from source, HIP/ROCm 10.0) on 2x Xeon Gold 6226R (2 NUMA nodes), 2x R9700 32 GB, Qwen3.8-Flash-Next IQ3_XXS, layer split 0-23/24-47, every expert in VRAM, LXC with CPUs on both sockets.Start-up without
STRATA_NUMA_POOL:All 10 workers and the host thread are on node 0, and the pool lines are identical to a
STRATA_NUMA_POOL=0start.Medians of 5 (2 warm-ups), fresh start each, ~3.7K and ~16K prompts, 512 greedy tokens, tok/s:
3.7K decode per run: no override 60.2 / 57.1 / 58.7 / 58.7 / 58.5; STRATA_NUMA_POOL=0 26.6 / 49.2 / 43.3 / 53.4 / 58.6. Both rows run the same one-socket pool, so the gap between them is this box's fresh-start variance, not the patch.
The default now picks the one-socket pool on the arena path by itself and is on par with 0.2.2 +
STRATA_NUMA_POOL=0. Looks good from here.pertain99 commentedon Oct 23, 2026
A second two-socket box, on the arena path with hugetlb pages and ~20% of the routed experts on the CPU: the case #3730 could not run. Three start-up variants, same engine, numbers below, plus a question on how per-node placement could work on a 2 MiB-page arena.
Machine
/sys/kernel/iommu_groups)../update.sh,v0.2.2, engine 0.1.47).expert arena: cudaHostRegister PORTABLE ok; hugetlb 2 MB pages), 5,337 expert slots in VRAM, decode hit rate 80-83%, so about one routed expert in five is computed by the CPU pool. MTP,--spec 4,--kv int8, 262K context,STRATA_ADAPT_OVERLAP=1withSTRATA_ADAPT_LAG=1 STRATA_ADAPT_NOWAIT=0beside the sm_86 table, CPU picture reader, one request at a time.numactl --cpunodebind=1 --preferred=1and 40,000 hugepages on node 1. The GPU is on node 0, so GPU DMA crosses QPI in every variant.Start-up lines and numbers
Load: 16.8K-token prompt, 1,500 greedy tokens, 3 runs after a fresh start of each variant, tok/s.
A. Bound to node 1 (production):
NUMA pool off: the pool's threads span 1 NUMA node(s), the NUMA split supports exactly 2 (2 NUMA nodes found),CPU workers on logical processors 19-34 (16 workers; host on 18).decode 56.5 / 57.7 / 60.6 (median 57.7), prompt 1,950-2,035.
B. No numactl, 21,000 hugepages on each node, split on by itself:
After the load node 0 had 0 of 21,000 hugepages free and node 1 had 5,273 free, so the arena was 21,000 pages on node 0 and 15,727 on node 1, in load order, not by slice.
decode 53.7 / 53.1 / 56.4 (median 53.7, -7% against A), prompt 2,009-2,068.
C. No numactl,
STRATA_NUMA_POOL=0:NUMA pool off (STRATA_NUMA_POOL=0), 16 workers on logical processors 1-16 (node 0, the host's socket), while the arena landed mostly on node 1 (node 1: 0 free, node 0: 5,273 free).decode 42.3 / 41.8 / 39.9 (median 41.8, -28%), prompt 1,928-2,009.
C is the cost of the pool reading an arena that sits almost entirely on the other socket: about -28% on this box. B is half of that penalty spread over both slices. Prompt speed is unchanged in all three (the GPU reads the prompt; the arena's node does not matter there). Memory bandwidth is not what binds:
perf staton the IMC counters shows the pool reading about 20 GB/s during decode against a 37 GB/s measured peak for node 1's three channels, so the remote reads cost latency and QPI round trips, not bandwidth.For this box #3730's veto would give A's one-socket pool without the numactl wrapper, which is what we want today.
Placement on a hugetlb arena: a question before anyone writes code
numa_slice_rangeshalves every role of a slot and rounds the halves topage. On this arena the page is 2 MiB and a UD-Q4_K_XL slot is about 2.9 MiB, so each role half is 0.5-1 MiB: at a 2 MiB page the slice-1 ranges round to empty and slice 0 gets whole pages that straddle slots. The file path's per-slot halves cannot carry over as they are. Three ways I can see:vm.nr_hugepagesis recommended) and a slower load (512x the page faults).If one of these is the direction you prefer, this box can run the branch: both sockets, 384 GB, hugepages on either node, and the arena path is the only path it uses. Happy to run the file path (
--resident-budget-gib) on the same model for a like-for-like comparison too, if that helps decide.zhengxia19 commentedon Oct 24, 2026
Additional data from a mixed Tesla P100 16 GiB + RTX 3070 8 GiB deployment. The arena-path warning in this issue is also present here. These are deployment-history observations, not a controlled cross-version or NUMA on/off benchmark.
Hardware and configuration
--spec 4,--pipeline-windows 2, VRAM reserve 700 MiB, expert cache/prefill auto; conversation cache 32768 MiB / 4 slots. Control-vector settings were retained. Systemd memory policy remained interleave across nodes 0–1.Observed changes and performance
Proxy = output tokens / (assistant settlement time − first nonempty delta time), restricted to valid, non-interrupted records with ≥64 output tokens. It is not native engine timing. Historical tasks differ; rows are assigned by deployment windows, not per-request version IDs. The final row is only one session and does not establish a calibration-induced loss.
Independent server-side request tails (
finish=stop, output ≥64) showed native Decode medians 55.50 → 48.70 tok/s for 0.1.44.1 → 0.2.1 (496 / 197 requests). The old tail had 71.0% of requests at ≥50 tok/s, versus 45.2% after that upgrade.Narrower evidence
Currently back on 0.1.44.1; 0.2.3/0.2.4 and a controlled split-off comparison have not been tested.
Could you advise which targeted comparison best separates the arena NUMA issue from server sampling/pipeline behavior and automatic placement on this heterogeneous CUDA pair? The GPU asymmetry is a possible interaction, not an established cause, and the 0.2.1 decline predates automatic NUMA splitting.
petski3 commentedon Oct 26, 2026
A third two-socket box on the arena path, the same model as pertain99's (Unsloth UD-Q4_K_XL, ~22 % of routed experts on the CPU). Here the split was a large gain rather than a loss once the arena was placed per node. pertain99's option 1 (4 KiB pages, the file path's slice placement) is implemented and measured below. On this box it gives +29 % decode over the one-socket pool, and the 4 KiB pages cost nothing I could measure. One condition matters: the engine's other host buffers must not all land on one node.
Machine and build
madvise,numa_balancing1, no hugetlb pool.cudaHostRegister, THP requested). Layer split 0-26 / 27-47 with 3,958 + 3,622 expert slots, decode hit rate ~78 %. MTP spec 4,--pipeline-windows 2, fp16 KV, 262K context, 32 GiB conversation cache.Method
Each arm is a fresh container start. Each arm runs a fixed 4K and a fixed 32K prompt (one run each) and a decode bench: 6 short prompts x 768 greedy tokens x 2 rounds. Decode tok/s is computed from the engine's per-request lines. Arms were run interleaved (A1, B1, C1, A2, B2, C2), so each figure below is two separate starts. ms/window is the decode bench's verify window from
STRATA_DECODE_TIMING.1. Split on vs off, arena not placed (same binary, before the placement change)
--cpuset-cpus 18-35,54-71 --cpuset-mems 1(node 1, the GPUs' node)numactl --interleave=0,1around the serverSTRATA_NUMA_POOL=0D matches pertain99's B (-7 %) and the "known limitation". When the arena lands wherever it is first touched, half of the workers read the other socket. B is the same split with the whole process interleaved across both nodes. Each slice then reads half local and half remote, and the box still gains +22 % over A. On this machine the CPU side was bound by one node's memory bandwidth: by my estimate (CPU-expert bytes per window against the pool's time), 16 workers moved ~60 GB/s, with 65.9 GB/s measured as one node's read bandwidth. Using both memory controllers matters more than the remote hop.
2. Placing the arena per slice (pertain99's option 1)
Change, about 140 lines:
PinnedArena's main constructor gets a hook that runs on the reserved, still untouched mapping right beforecudaHostRegister. The registration faults every page in where the registering thread's policy says, so placement has to happen before it.ArenaExpertSourceuses that hook:MADV_NOHUGEPAGEon the arena, thenNumaComplementPlacement::touch_stepover thenuma_slice_rangesof every slot.MPOL_PREFERREDfor that node. A process-widenumactl --interleavewould otherwise override the first touch.active_numa_split()and logs either "placed on nodes 0 / 1 as the pool splits them" or a warning.move_pagessampling after the registration: 100.0 % of each slice's pages on its node (35.96 / 35.77 GiB). Touch plus registration took ~26 s at start. The file read that follows ran at 8.6 GiB/s instead of 1.8 because the pages are already present, so the start time is about unchanged. Registering 71.7 GiB of pre-faulted 4 KiB pages worked.Same binary, interleaved, CPUs 0-35, mems 0,1, server under
numactl --interleave=0,1:STRATA_ARENA_NUMA=0)F is +5 % over E, and against A (one socket, node-bound) the total is +29 % decode, +13 % at 4K and +8 % at 32K (means of the two starts). The TLB cost of 4 KiB pages that option 1 worried about does not show up here: F is faster than E with 2 MiB pages. I did not test a hugetlb arena (no hugetlb pool on this box).
3. The condition: the other host buffers
The same placement without
numactl --interleave(CPUs 0-35, mems 0,1) gave 33.2 / 34.2 tok/s decode and 131 / 151 tok/s at 4K, two starts, placement verified at 100 %. Prompt reading collapsed too, and it does not touch the CPU experts. So I think the cause is the engine's other host memory: KV in pinned RAM (3 GiB here), the 32 GiB conversation cache and staging buffers, all landing on one node. That node only had ~32 GiB free after its arena half, the same pressure we saw earlier on this box with the whole engine bound to one node. I did not record vmstat for those runs, so this is an inference, not a measurement. With the interleave policy for the process and placement only for the arena, the numbers are F above.GPU side: both cards hang off node 1. Host-to-device copies from node-0 memory ran at 11.8 vs 12.8 GB/s per card, and 23.6 vs 25.0 GB/s with both cards at once (~ -7 %). The half of the arena the GPUs pull from node 0 (expert swaps, prompt streaming) costs little.
What this suggests for #3730
On this box, the off-switch in #3730 would be the right fallback when the arena is not placed. With placement the split is a clear win, so the veto could depend on whether placement succeeded. Whether placement should be opt-in at first, and whether the engine should interleave its own large host buffers instead of relying on
numactl, are the maintainers' call. If this direction is wanted, I can open a focused PR with the placement change, opt-in and rebased on current main, re-measured there.Validation
Passed:
move_pages: 100 % of the sampled pages of each slice on its node.Failed:
Not run:
--resident-budget-gib), other models or quants.Measured and prepared with Claude Code on my box; I have checked the numbers against the logs.
github-actions commentedon Oct 28, 2026
Closing this issue because it still did not follow the format after 5 days. That is housekeeping, not a rejection.
To bring it back: fix the title (area prefix) and the missing parts, then comment
/reopen. A check runs and reopens it, or tells you what is still missing. See CONTRIBUTING.md.