Prompt path: the first chunk's n-gram (PLE) rows read beside layer 0, 256 at a time (same output) by sergqwer · Pull Request #374 · Niko1221/Strata

sergqwer · 2026-10-01T14:21:38Z

Layer 1 adds the per-layer embedding (the n-gram table) to every token of a chunk, so a chunk's rows must be read before layer 1.

Later chunks were fine: their rows were already read on a thread during the chunk before.
The first chunk's were read synchronously before layer 0: the GPU idled while the SSD read them. An nsys trace of a single-chunk 32K prompt shows ~245 ms of it.

What changes:

The first chunk's gather runs on a thread too, beside the embedding and layer 0. It lands just before layer 1 (ple_land: wait, upload, start the next chunk's gather), and a stage that ends before layer 1 lands it after its last layer.
--ple-inflight defaults to 256 (was 64).
- A 32K prompt is 512,720 rows in 213,358 4 KiB reads. At 64 outstanding reads the drive (Samsung 9100 PRO) ran ~620K IOPS with the GPU waiting. At 256 it saturates, and 1024 is no faster.
- Decode reads 16 rows a token and is unchanged.

The same rows in the same order give the same results.

Same output

First-token logits are bit-identical to 0.1.31 at 8K and 32K. They were dumped with #276's STRATA_DUMP_FIRST_LOGITS in test builds of both, with a fixed --expert-cache 14900.

Speed

Setup: IQ2_XS (ISTA), RTX 5090, 9950X3D, 128 GB, Windows 11, --max-context 262144 --kv int8 --prefill auto (8192-token chunks), --vram-reserve-mib 1500. The runs alternated between 0.1.31 and this PR.

	0.1.31	this PR
32K prompt (cache auto)	5,986 / 6,013 ms	5,843 / 5,808 ms (-2.9%)
8K prompt (fixed cache)	1,714 ms	1,624 ms (-5.3%)
host time blocked on the rows, 32K	493-503 ms	411-432 ms

The gain is the first chunk's read, so it is larger with fewer chunks. On our NVFP4 fork, a 32K prompt in one 32K chunk (#282's auto) was 7-9% faster from the overlap and 2% more from the read depth.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergqwer · 2026-10-01T15:10:42Z

Pushed b490d4c: ple_land now checks the upload of a chunk's rows, the event recorded after it, and the wait on the other buffer's event before the SSD thread refills it. Before, a failed record or wait let the thread write a buffer an upload might still read. A failure now ends the chunk with the CUDA error. The same tokens on the 32K prompt.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

… 256 at a time Layer 1 adds the per-layer embedding to every token of a chunk, so its rows must be in. A later chunk's rows were already read on a thread during the chunk before; the first chunk's were read synchronously before layer 0, so the GPU idled while the SSD read them (an nsys trace: ~245 ms of a single-chunk 32K prompt). Now the first chunk's gather starts on a thread too and lands just before layer 1 (ple_land), which also starts the next chunk's. The reads themselves went 64 at a time: a 32K prompt is 512,720 rows in 213,358 4 KiB reads, ~620K IOPS at queue depth 64. --ple-inflight now defaults to 256, where the drive saturates (1024 is no faster). Decode reads 16 rows a token and is unchanged. The same rows: the same results. IQ2_XS, RTX 5090: a 32K prompt 5986 / 6013 -> 5843 / 5808 ms, 8K 1714 -> 1624 ms; first-token logits bit-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…e checked From an audit: the upload of a chunk's rows, the event recorded after it and the wait on the other buffer's event before the SSD thread refills it were unchecked, so a failed record or wait let the thread write a buffer an upload might still read. A failure now ends the chunk with the CUDA error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergqwer · 2026-10-01T18:16:36Z

Rebased on 0.1.32 (c499bd1), no conflicts.

Checked on 0.1.32: IQ2_XS (ISTA), RTX 5090, a fixed --expert-cache, --vram-reserve-mib 1500. The first generated token's logits (STRATA_DUMP_FIRST_LOGITS, now in main) are byte-identical to main's after a 2K and a 32K prompt. Every target builds with STRATA_BUILD_TESTS.

The gain on 0.1.32, the same runs (prompt reading, one run each): 2K 858.8 -> 808.2 ms (-6%), 32K 5,757 -> 5,528 ms (-4.0%).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

BlueKingMuch · 2026-10-02T00:27:49Z

Data point from Windows: RTX 4080 SUPER 32 GB, Ryzen 7 5800X3D, PCIe 3.0 NVMe (--ple-io direct, default row cache), Windows 11; IQ3_S at 262K, --prefill 32768, so a 32K prompt is one chunk. Compared: 0.1.32 (+#363, #413, #346) before and after merging this PR together with #385, #372, #378 and #407; the phases this PR touches, from STRATA_PREFILL_TIMING=1, per 32K prompt:

	before	with this PR
first prompt after the start (its rows not in the row cache): chunk setup on the host	581 ms	1 ms
same prompt: GPU phases `embed+steps` / `combine`	606 / 702 ms	7 / 1205 ms
the same text again (rows in the row cache): chunk setup	57 ms	1 ms
same prompt: `embed+steps` / `combine`	82 / 702 ms	7 / 733 ms

The answers are the same. On this machine the first chunk's read takes longer than layer 0, so most of the wait moves to the start of layer 1, where it shows up in layer 0's combine phase: net -96 ms on the prompt whose rows come from the SSD, -44 ms when they are cached, -35 to -40 ms on 64K prompts. So the gain is real but small here (about 1 % of a 32K prompt); the overlap would need more work in front of layer 1, or an earlier start, to hide a read of this length.

sergqwer · 2026-10-02T12:16:18Z

@BlueKingMuch thank you for testing this and for the per-phase numbers on your machine.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…iko1221#374 Niko1221#363 Niko1221#407 Niko1221#413 onto 0.1.35 One dpct run of upstream main + the seven PRs, 3-way merged into the files they touch (sycl/tools/merge_upstream.py); their header changes are forked into sycl/include until upstream merges them. - Niko1221#385 stager waits for a buffer's previous DMA on every job (port: polled seq) - Niko1221#463 decode waits for an adaptive swap before reading residency - Niko1221#453 drafter's batched K/V on the KV-streaming ring - Niko1221#374 first chunk's PLE rows read beside layer 0; port fixes: the next read starts at c0 + T (the port's first chunk is shorter), and the upload is marked by a polled sequence number instead of a host event wait, which deadlocked prompts past ~4K tokens under the Level Zero v2 adapter - Niko1221#363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied to the port's lane kernels (fp contract off, as upstream's HIP form) - Niko1221#407 --adapt-tuned (opt-in), Niko1221#413 key-head DeltaNet recurrence (opt-in via STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding parity bench not ported) Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid, quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and 40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s; 128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (Niko1221#363). Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rb4SRxFuLpffQAAXBf4HxA

…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

Niko1221 · 2026-10-03T01:17:24Z

Released in 0.1.38: the first chunk's PLE rows read beside layer 0, 256 reads in flight. Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

Upstream v0.1.38: faster prompts (one gather per expert group Niko1221#372, the first chunk's PLE rows beside layer 0 Niko1221#374, DeltaNet three heads per thread Niko1221#413), --kv q4_0 prompts on tensor cores Niko1221#452, Q5_0 experts on the GPU Niko1221#473, IQ4_XS on AVX-2 Niko1221#415, unbuffered expert loading on Windows Niko1221#357/Niko1221#362, the --peer-device second-GPU expert tier Niko1221#531, a 6 GB card starts Niko1221#496, the Host header / DNS-rebinding checks, PR batch. Our side: the AVX1 floor (STRATA_ISA_FLOOR, cpu_avx1_ok, the relaxed gate, the scalar router fallback), the AVX1 router kernel (kq_avx1.cpp), the retune patch + setup hooks.

…the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered loads on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; setup requires it

…0.1.35 One dpct run of upstream main + the seven PRs, 3-way merged into the files they touch (sycl/tools/merge_upstream.py); their header changes are forked into sycl/include until upstream merges them. - #385 stager waits for a buffer's previous DMA on every job (port: polled seq) - #463 decode waits for an adaptive swap before reading residency - #453 drafter's batched K/V on the KV-streaming ring - #374 first chunk's PLE rows read beside layer 0; port fixes: the next read starts at c0 + T (the port's first chunk is shorter), and the upload is marked by a polled sequence number instead of a host event wait, which deadlocked prompts past ~4K tokens under the Level Zero v2 adapter - #363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied to the port's lane kernels (fp contract off, as upstream's HIP form) - #407 --adapt-tuned (opt-in), #413 key-head DeltaNet recurrence (opt-in via STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding parity bench not ported) Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid, quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and 40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s; 128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (#363). Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)

…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it

…iko1221#374 Niko1221#363 Niko1221#407 Niko1221#413 onto 0.1.35 One dpct run of upstream main + the seven PRs, 3-way merged into the files they touch (sycl/tools/merge_upstream.py); their header changes are forked into sycl/include until upstream merges them. - Niko1221#385 stager waits for a buffer's previous DMA on every job (port: polled seq) - Niko1221#463 decode waits for an adaptive swap before reading residency - Niko1221#453 drafter's batched K/V on the KV-streaming ring - Niko1221#374 first chunk's PLE rows read beside layer 0; port fixes: the next read starts at c0 + T (the port's first chunk is shorter), and the upload is marked by a polled sequence number instead of a host event wait, which deadlocked prompts past ~4K tokens under the Level Zero v2 adapter - Niko1221#363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied to the port's lane kernels (fp contract off, as upstream's HIP form) - Niko1221#407 --adapt-tuned (opt-in), Niko1221#413 key-head DeltaNet recurrence (opt-in via STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding parity bench not ported) Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid, quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and 40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s; 128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (Niko1221#363). Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)

…iko1221#452), Q5_0 experts on the GPU (Niko1221#473), IQ4_XS on AVX-2 (Niko1221#415), unbuffered loads on Windows (Niko1221#357 Niko1221#362), --peer-device (Niko1221#531), a 6 GB card starts (Niko1221#496), PR batch; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

…iko1221#374 Niko1221#363 Niko1221#407 Niko1221#413 onto 0.1.35 One dpct run of upstream main + the seven PRs, 3-way merged into the files they touch (sycl/tools/merge_upstream.py); their header changes are forked into sycl/include until upstream merges them. - Niko1221#385 stager waits for a buffer's previous DMA on every job (port: polled seq) - Niko1221#463 decode waits for an adaptive swap before reading residency - Niko1221#453 drafter's batched K/V on the KV-streaming ring - Niko1221#374 first chunk's PLE rows read beside layer 0; port fixes: the next read starts at c0 + T (the port's first chunk is shorter), and the upload is marked by a polled sequence number instead of a host event wait, which deadlocked prompts past ~4K tokens under the Level Zero v2 adapter - Niko1221#363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied to the port's lane kernels (fp contract off, as upstream's HIP form) - Niko1221#407 --adapt-tuned (opt-in), Niko1221#413 key-head DeltaNet recurrence (opt-in via STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding parity bench not ported) Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid, quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and 40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s; 128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (Niko1221#363). Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rb4SRxFuLpffQAAXBf4HxA

sergiywith and others added 2 commits October 1, 2026 20:48

sergqwer force-pushed the pr/ple-reads branch from b490d4c to 89e2994 Compare October 1, 2026 18:16

q8atnight mentioned this pull request Oct 2, 2026

prefill: a group's expert gathers in one launch (bit-identical, long prompts +4%) #439

Closed

Niko1221 closed this Oct 3, 2026

Navigation Menu

Repository navigation

Prompt path: the first chunk's n-gram (PLE) rows read beside layer 0, 256 at a time (same output) - #374

Prompt path: the first chunk's n-gram (PLE) rows read beside layer 0, 256 at a time (same output)#374
sergqwer wants to merge 2 commits into
Niko1221:mainfrom
sergqwer:pr/ple-reads

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

BlueKingMuch commented Oct 2, 2026

Uh oh!

sergqwer commented Oct 2, 2026

Uh oh!

Niko1221 commented Oct 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sergqwer commented Oct 1, 2026

Same output

Speed

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

sergqwer commented Oct 1, 2026

Uh oh!

BlueKingMuch commented Oct 2, 2026

Uh oh!

sergqwer commented Oct 2, 2026

Uh oh!

Niko1221 commented Oct 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants