Linux expert arena (0.1.39): MADV_HUGEPAGE + defrag=madvise makes the arena's faults 20x slower (25 s start -> 432 s), and 'loaded ... at X GiB/s' does not show it

Update (measured after this report). The 3.06-3.12 GiB/s load line above is not what this bug costs.
The same profile that crawled here reached ready at 432 s while still printing loaded 39.97 GiB at 2.86 GiB/s,
and the same profile with the MADV_HUGEPAGE request skipped is ready in ~25 s. The isolated 20x madvise
measurement, the /proc numbers, the switch and the note change are in the comment below.

Symptom

Engine 0.1.39 (source build), arena profile (strata-iq3_xxs.json: IQ3_XXS native pack, experts in RAM, no
--mmap-experts, --max-context 131072 --kv-resident 32768, MTP and vision on).
Machine: Manjaro Linux, kernel 6.12.108-1-MANJARO, i7-11700B, 62 GiB RAM, RTX 5070 Ti (16.3 GiB).

With the RAM free the fill is fast, and the log names the request that is in play:

strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_HUGETLB unavailable (needed 20464 2 MiB pages, vm.nr_hugepages=0); transparent huge pages requested (MADV_HUGEPAGE)
strata generate: loaded 39.97 GiB at 3.12 GiB/s

(a later start of the same profile, same engine: strata generate: loaded 39.97 GiB at 3.06 GiB/s)

With the RAM already in use, the same 39.97 GiB fill crawls at 60-77 MB/s, one core at 100 %, no disk I/O,
and the PC becomes unusable while it fills. Two starts of the arena profile did that in one afternoon; both were
stopped, and the engine's output is buffered, so those logs end at the line before the arena note:

strata generate: GPU 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0     <- last line in those logs

The numbers below are live /proc readings taken while the fill was in progress, not log lines:

RSS deltas: +484 MB in 8 s, +616 MB in 8 s (60-77 MB/s); RSS 31.7 GB of ~43 GB when sampled
/proc/vmstat, 8 s window: compact_stall +858, compact_fail +858, thp_fault_alloc +0,
thp_fault_fallback +290, allocstall_normal +286, nr_free_pages -498 MB
the process: 18,304 minor faults/s; at 4 KB per page that is 75 MB/s, the same number the RSS deltas gave.
CPU 95-100 % of one core. In the same window the device read 0-1 MB/s and wrote nothing.
what the user sees: the server repeats [strata] still starting (N s) - please wait ... (serve/server.py:307)

Why the `MADV_HUGEPAGE` request is my prime suspect

This machine: /sys/kernel/mm/transparent_hugepage/enabled = [always], defrag = [madvise],
vm.nr_hugepages = 0.

The kernel's own description of the defrag policies (Documentation/admin-guide/mm/transhuge.rst):

madvise: "will enter direct reclaim like always but only for regions that are have used madvise(MADV_HUGEPAGE). This is the default behaviour."
always: "... an application requesting THP will stall on allocation failure and directly reclaim pages and compact memory in an effort to allocate a THP immediately."

With defrag=madvise, the arena's own madvise(MADV_HUGEPAGE) is what can put this 40 GiB mapping into
synchronous direct reclaim and compaction in the faulting thread. The counters point the same way: in that 8 s
window a compaction was tried ~107 times per second and never succeeded (compact_fail == compact_stall), no
new THP was allocated (thp_fault_alloc +0), the faults fell back to 4 KB pages (thp_fault_fallback +290), and
at 18,304 faults/s the faulting thread paid ~50 µs of kernel work per page - 75 MB/s with one core burnt.
(The arena does get huge pages earlier in a run: 13.8 GB of a 31 GB fill were backed by them when sampled.)

With the RAM still free there is nothing to compact, and the same request costs nothing (3.06-3.12 GiB/s above).
For reference, the 0.1.38 build (which has no THP request: its note ends in ; using 4 KB pages) filled the same
profile on this machine at 3.32 GiB/s, so the request buys nothing in this phase here; the TLB argument in the
comment above it is about the CPU expert pool during decode, not this fill.

Not the cause (checked)

disk: the device did not move during the window (0-1 MB/s reads, no writes); the expert bytes come from the
page cache and the pack
loader/pack: the same code path, same pack and same machine filled the same arena at 3.06-3.12 GiB/s minutes
earlier
VRAM: not used in this phase (the fill is host memory; the cache slots are chosen afterwards)
an OOM: no swap activity and no OOM kill, the fill kept progressing - the pages it got were simply 4 KB ones
STRATA_NO_LARGEPAGES=1 is not a workaround: it also skips the hugetlb attempt, i.e. it changes two things

Same mechanism reported elsewhere

runtime: MADV_HUGEPAGE causes stalls when allocating memory golang/go#61718 "runtime: MADV_HUGEPAGE causes stalls when allocating memory"; the Go runtime dropped its own
huge-page policy in 1.21.4 ("the Go runtime would no longer try to impose a huge page policy itself")
server: provide guidance and/or software control over transparent huge pages cockroachdb/cockroach#130241 "server: provide guidance and/or software control over transparent huge pages"
Transparent Huge Pages set to [always] is sub-optimal for many applications coreos/bugs#2635 transparent huge pages set to [always] are sub-optimal for many applications

What I would propose (implemented locally, with a selftest - happy to open a PR)

put the policy in the note, so a crawling start explains itself in the log:
...; transparent huge pages requested (MADV_HUGEPAGE; kernel defrag=madvise)
(one read of /sys/kernel/mm/transparent_hugepage/defrag; the note is what users paste into reports)
a same-run A/B switch for the request only: STRATA_NO_ARENA_THP=1 ->
; transparent huge pages skipped (STRATA_NO_ARENA_THP); using 4 KB pages. Default unchanged.
Alternative if you prefer no new switch: skip the request, or warn, when
/sys/kernel/mm/transparent_hugepage/defrag is always, madvise or defer+madvise - i.e. whenever a
2 MiB block may be impossible to get synchronously.
src/core/pinned_thp_test.cpp: a selftest for the note in both cases (64 MiB arena, no model, no 40 GiB).

If you want a different default than 2, say so and I will follow that instead. I can also report a before/after
arena run with the switch on this machine, but that is the run that makes the PC unusable here, so it is a
one-shot: I would rather do it once you tell me what direction you want.

Correction + verification (same machine, same day)

1. The start does complete, and the printed rate does not show the stall. I let the second start above run to
the end instead of stopping it: the server reached ready at 432 s and the engine still printed
strata generate: loaded 39.97 GiB at 2.86 GiB/s. The same profile with the request skipped prints 3.29 GiB/s
and is ready in ~25 s. So this is not a slow fill counter: it is a 17x wall-clock difference that
load_gib_per_second() (printed at src/program/generate.cpp:3170) does not reflect - the fault path the /proc
numbers above describe is where the time goes. The headline number here is 432 s, not 75 MB/s.

2. The mechanism, isolated. A 2 GB MAP_ANONYMOUS mapping, one byte written per 4 KB page, same machine, same
minute, RAM already in use (server up, ~47 GB used):

same 2 GB fill	rate	/proc/vmstat delta over the fill
plain `mmap`	4440 MB/s (0.5 s)	compact_stall +0
+ `madvise(MADV_HUGEPAGE)`	216 MB/s (9.5 s)	compact_stall +2684, compact_fail +2677, thp_fault_alloc +102, thp_fault_fallback +926
plain `mmap` again	3436 MB/s (0.6 s)	compact_stall +0

That is the 20x, and the madvise is the only difference between the runs. /proc/buddyinfo at that moment:
Node 0, zone Normal 1141 2110 12921 9739 6912 4864 2970 1811 1123 1 0 - one free order-9 block, so a THP
allocation means compaction every time, and it fails (compact_fail == compact_stall).

3. The switch, on the real profile. Same binary, same config, same free-RAM baseline (used 5.7 GB / avail 58.0 GB before the 432 s start; used 5.2 GB / avail 58.6 GB before this one):

strata generate: expert arena: cudaHostRegister PORTABLE ok; ...; transparent huge pages skipped (STRATA_NO_ARENA_THP); using 4 KB pages
strata generate: loaded 39.97 GiB at 3.29 GiB/s
[strata] experts loaded: 39.97 GiB at 3.29 GiB/s (22 s so far)
ready: http://127.0.0.1:8090/v1

432 s -> ~25 s, same arena (39.97 GiB), same engine (0.1.39), same machine, same session.

A 62-line local patch in src/core/pinned.cu adds that switch, keeps the request by default, and puts the
kernel's own policy in the note (; transparent huge pages requested (MADV_HUGEPAGE; kernel defrag=madvise)) so a
crawling start is explainable from the log alone. src/core/pinned_thp_test.cpp checks the four note forms on a
64 MiB arena (no model, no 40 GiB) and passes here. Happy to send it as a PR if the direction is right.

The design question: with enabled=always + defrag=madvise + a fragmented buddy, this request is answered
with synchronous compaction in the faulting thread and buys nothing during this load. Should the arena skip
MADV_HUGEPAGE when the kernel's defrag policy is madvise/always (ask only where the kernel answers in the
background), or is the decode-time TLB argument worth keeping it as the default and the skip as an opt-in?

Noticed the same issue, except model loading was so slow I couldn't even wait for it to finish. Reverted to 0.1.38, and it loaded in under a minute. I had the same symptoms, little to no disk activity, slow strata RES growth, high system CPU load, also [kcompactd0] added 100% (1 core) CPU use, this kernel thread does page defrag I believe. Turning it off with sysctl -w vm.compaction_proactiveness=0 didn't help, it's a Google advice, might as well be unrelated.

I run Debian testing with a custom kernel, nothing fancy, just disabled unused hardware drivers and set timer to 1000 Hz. I've got 5090 and 128 GB of RAM.

Same thing here, on a third machine and kernel, with a larger arena.

Setup: Linux Mint 22, kernel 6.8.0-51-generic, 125 GiB RAM, 2x V100-PCIE-32GB, main at 6f32ec0 (source build, -DSTRATA_EXPERIMENTAL_SM60=ON), UD-IQ4_XS native pack, arena 55.43 GiB (needed 28383 2 MiB pages, vm.nr_hugepages=0). THP enabled=madvise, defrag=madvise.

Two hung starts on the same day. Live readings from the second one, 3 min into the start (the server was still not ready after 12 min):

process: 196 s system time against 1.8 s user time, one core at 100 %, RSS 69 GB
/proc/vmstat over 5 s: compact_stall +601, compact_fail +601, compact_migrate_scanned +145 M, thp_fault_fallback +198, allocstall_normal +206
/proc/buddyinfo: Node 0, zone Normal 1080 4103 93070 68226 49560 25266 2096 0 0 0 0 - no free block of order 7 or higher
/proc/meminfo: MemFree 11 GB, Cached 75 GB (largely the model's own GGUF shards), AnonHugePages 27.5 GB

The first hang looked the same (603 stalls in 5 s, all failed) and was stopped after ~6 minutes. The engine log ended at the GPU 0: ... line, as you describe.

Starts that did not hang, same binary and config, ready time:

arena pages	page cache	ready
THP requested	cold	35 s (2,397 compaction stalls)
THP requested	warm, ~60 GB free	20 s, 20 s (0 stalls)
`STRATA_NO_LARGEPAGES=1`	warm, ~60 GB free	26 s

So here the request does pay off when the 2 MiB blocks exist: 20 s against 26 s. The trouble is only when they don't. That points to keeping the huge pages but not getting them in the faulting thread. Two ways that don't stall:

fill the arena first, then madvise(MADV_HUGEPAGE): the faults take 4 KB pages at full speed, and khugepaged collapses the range in the background, at its own pace and with its own defrag setting
or after the fill, one madvise(MADV_COLLAPSE) (Linux 6.1+): a single synchronous attempt that fails cleanly instead of being retried on every fault

I have not measured either one yet, and I have not measured what the huge pages bring during decode.

On this machine STRATA_NO_LARGEPAGES=1 is a working workaround, since there is no hugetlb pool, so it only drops the THP request. It is now the default in my start scripts.

pinned arena: no VirtualLock on large pages, no MADV_HUGEPAGE under T…

docs: TROUBLESHOOTING for Smart App Control, the port error, the engi…

Fixed in 0.1.40: with transparent huge pages on [always] the arena no longer asks for MADV_HUGEPAGE on top, which is what sent every fault into direct reclaim and compaction. STRATA_NO_ARENA_THP=1 skips the request on any THP setting. Thanks for the isolated measurements.

pinned arena: no VirtualLock on large pages, no MADV_HUGEPAGE under T…

docs: TROUBLESHOOTING for Smart App Control, the port error, the engi…

Linux 6.18.54 with transparent huge pages set to madvise makes model loading extremely slow.

# cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

With THP disabled in the kernel, or after setting it to never, the model loads quickly.

Strata was built from the latest commit 82f46a8 (v0.1.40.1).

pinned arena: no VirtualLock on large pages, no MADV_HUGEPAGE under T…

docs: TROUBLESHOOTING for Smart App Control, the port error, the engi…

@fukc-gihtub

Thanks for the fix. One case it does not cover, and it is the one from my comment above and from @fukc-gihtub's: THP enabled=madvise.

The 0.1.40 code (src/core/pinned.cu, from line 242) leaves the request out only when enabled is always. With enabled=madvise it still calls madvise(MADV_HUGEPAGE), and the faults stall in direct compaction exactly as before, because the stall comes from defrag, not from enabled. My machine is set to enabled=madvise, defrag=madvise, the Ubuntu / Mint default. On 0.1.40.2 the note there would still read transparent huge pages requested (MADV_HUGEPAGE). I read that from the code and have not started the THP arena on 0.1.40.2 again, because of the hang. My starts run with STRATA_NO_LARGEPAGES=1 for now.

madvise is also the one setting where the request matters: there it is the only way the arena gets huge pages at all, so skipping it there loses them entirely. A gate on enabled cannot fix both sides. Options I see:

gate on /sys/kernel/mm/transparent_hugepage/defrag instead: for a madvised range, always, madvise and defer+madvise all mean synchronous reclaim and compaction on the fault, while defer and never don't stall the faulting thread
or keep the huge pages but take them off the fault path: madvise(MADV_HUGEPAGE) after the fill (khugepaged collapses in the background), or one MADV_COLLAPSE after the fill (Linux 6.1+)

The issue is closed, so I am leaving it here in case it is worth reopening. Happy to measure a candidate on the V100 machine, where the hang reproduces with a full page cache.

pinned arena: no VirtualLock on large pages, no MADV_HUGEPAGE under T…

docs: TROUBLESHOOTING for Smart App Control, the port error, the engi…

エキスパートのアリーナ: THP が always のときは MADV_HUGEPAGE を頼まない（upstream bb19bec の N…

Navigation Menu

Repository navigation

Linux expert arena (0.1.39): MADV_HUGEPAGE + defrag=madvise makes the arena's faults 20x slower (25 s start -> 432 s), and 'loaded ... at X GiB/s' does not show it #771

Symptom

Why the `MADV_HUGEPAGE` request is my prime suspect

Not the cause (checked)

Same mechanism reported elsewhere

What I would propose (implemented locally, with a selftest - happy to open a PR)

Correction + verification (same machine, same day)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Linux expert arena (0.1.39): MADV_HUGEPAGE + defrag=madvise makes the arena's faults 20x slower (25 s start -> 432 s), and 'loaded ... at X GiB/s' does not show it #771

Description

Symptom

Why the MADV_HUGEPAGE request is my prime suspect

Not the cause (checked)

Same mechanism reported elsewhere

What I would propose (implemented locally, with a selftest - happy to open a PR)

Activity

Alucard24 commented on Oct 4, 2026

Correction + verification (same machine, same day)

rkfg commented on Oct 5, 2026

fks commented on Oct 5, 2026

Niko1221 commented on Oct 6, 2026

fukc-gihtub commented on Oct 6, 2026

fks commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Why the `MADV_HUGEPAGE` request is my prime suspect