[gilby]

What this changes

The HIP Flash Next engine advises MADV_RANDOM over the n-gram table tensors' spans after mapping (File.adviseRandom, Checkpoint.adviseRandom), so a gather's random single-row reads fault only their pages instead of whole readahead windows. Fixes #1602. #1611 has landed, so this is one commit on main.

The issue guessed at first-use graph capture; measuring showed otherwise: with graphs off the first request was equally slow, the engine replays 4,563 of 4,571 rounds, and the first request read 1,419,132 KiB from disk with 11,262 major faults while the identical repeat read 0. The slowdown follows new prompt content (new n-gram rows), not server start. The CUDA path already advises its tables this way.

Receipt

• Environment: TensorFold 7c068895 plus hip flashnext: parallel n-gram gather, release weight pages after upload #1611 and this change, tensorfold-native 1.1.8 built with zig build native-hip -Dhip-arch=gfx1151, Ubuntu archive HIP 7.1 (clang 21), AMD Ryzen AI MAX+ 395 with Radeon 8060S gfx1151 (128 GB unified memory), page cache dropped before each server start. Checkpoint TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP revision 2b170fa6, MTP 3 drafts, --parallel 1 --context 131072 --prompt-cache-gib 0, fresh process per run, base and candidate on the same machine.
• Exactness: token_sha256 of tools/bench_openai.py, tools/bench_concurrent.py and both tools/bench_cache.py prompts identical to the base build; bench_concurrent --alone --serial every cell alone 3/3 and 12/12 equal, serial equal, 0 failed.
• Decode speed, tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0: first request after start (14-token prompt) 24.4 tok/s on the base, 35.8 / 35.9 / 34.9 tok/s with this change alone over three fresh processes, 35.5 tok/s stacked on hip flashnext: parallel n-gram gather, release weight pages after upload #1611; later requests unchanged (base 50.03 / 43.93, stacked 50.13 / 44.09 median code / chat).
• Prompt speed, tools/bench_cache.py --require-cold on public 7,377- and 29,364-token arrays, first cold request ttft_s: base 48.44 / 108.97 s; this change alone 28.9 s at 7,377; stacked on hip flashnext: parallel n-gram gather, release weight pages after upload #1611 25.30 / 102.22 s.
• Tests run: zig build test -Dhip=true --summary all 466/466 steps, 0 failed, on the stacked tree; a new host test checks that adviseRandom marks the tensor's pages random in /proc/self/smaps (fails with the advice disabled: 25 passed, 1 failed). python3 tools/zig/lean_check.py 425 on base and change. git diff --check clean.
• Not run: hip-kernel-test and the kernel contract (no kernel change). The residual cold cost is random-read latency (about 4 s for 5,888 rows on this disk); parallel reads could cut it further and are left out. Metal and CUDA (Linux HIP path only; the advice is a no-op elsewhere and on heap-copied files).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel change.)
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks — the random-access advice makes sense for scattered n-gram rows. This PR contains all of #1611 plus the madvise change; could you say whether it replaces 1611 or should be rebased on top of it once 1611 lands? Either works, but we'll only land one copy. Please also add a short description with before/after numbers (page faults, or gather / first-prefill time) so we can see what the advice buys.

[gilby]

It stacks on #1611 rather than replacing it: after #1611 lands I'll rebase this so it holds only the advice commit.

What the advice buys, on the same machine and checkpoint (TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP, gfx1151), from the description's receipt:

• Page faults: on the base, the first request after a fresh start read 1,419,132 KiB from disk with 11,262 major faults while the identical repeat read 0. The gather reads one 100-byte row per id at random rows, and without advice each faulted page pulled a readahead window; with MADV_RANDOM only the row pages are read.
• First request decode (14-token prompt): 24.4 tok/s on the base, 35.8 / 35.9 / 34.9 tok/s with the advice over three fresh processes (35.5 stacked on hip flashnext: parallel n-gram gather, release weight pages after upload #1611); later requests unchanged.
• First cold prefill (tools/bench_cache.py --require-cold, 7,377 tokens): 48.44 s on the base, 28.9 s with the advice alone, 25.30 s stacked on hip flashnext: parallel n-gram gather, release weight pages after upload #1611.

I also pushed a zig fmt fix for the failing check.

[gilby]

#1611 has landed, so this is now one commit on current main (cfcbecb) instead of a stack: the old copy of #1611's commit is gone, and the remaining commit is the random-access advice on the n-gram table's mappings. The only conflict was in core/checkpoint.zig, where main added prefetch beside it; both methods are kept. Nothing else in the change moved: safetensors.zig +11 and tables.zig +3, as before.

On the rebased head (b3af83a), one Strix Halo iGPU (gfx1151), ROCm 7.1 hipcc, zig 0.17.0:

• zig fmt clean, lean_check 424 problems, the same as main.
• zig build native-hip builds; zig build test -Dhip=true --summary all: 516/516 steps, 6820 of 6869 tests passed, 49 skipped, 0 failed (two more than main: this change's tests).

The advice only changes how the kernel reads pages of the table, so no token can move, but the speed receipts in the description were taken on the earlier head and I have not re-run them on this one. Not run: any other GPU or family.

[ashhart]

Thanks @gilby. This is a clean, exact change with the receipts we asked for: token_sha256 identical across every client, decode on the first request after start 24.4 → 35.8 tok/s, and a cold 7,377-token prompt 48.4 s → 28.9 s, because the gather's random reads now fault only their own pages. It matches what the CUDA path already does. We've taken it out of draft and queued it to land.

[ashhart]

Thank you @gilby. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @gilby! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3