Repository navigation
[Engine] Deterministic all-logits NaN (#879) (IQ3_S, CUDA, Windows) #3602
Description
[Engine] 0.1.47: #879 with direct PLE I/O; mmap passes the failing prompts
Two captured prompts still fail on engine 0.1.47 with default --ple-io direct: non-finite logits in the first decode window, after the entire prompt was read batched. The automatic retry also fails, even with --short-read 0. With --ple-io mmap, both prompts and a repeat of the 33K prompt pass the same first-output checks.
A separate diagnostic build finds non-finite values already in the gathered/uploaded PLE embedding, while the residual before PLE is finite. This narrows the investigation to PLE input acquisition/decoding/upload; it does not yet identify the exact faulty operation.
Environment and configuration
- Official v0.2.2 Windows CUDA 13 engine 0.1.47,
f-022-rel, commit76f90e58436d; unmodified release executable. - Windows 10 Pro 19045 in a Proxmox/QEMU VM, GPU passthrough.
- RTX 4070 SUPER 12 GB, sm_89, driver 610.88.
- Xeon E5-2697 v4, 16 vCPUs, 96 GiB RAM. Guest CPU flags include AVX2/FMA/F16C/BMI2, without AVX-512/VNNI.
- Existing Python server:
f57c0adplus a request-capture patch, not the v0.2.2 Python server. The engine itself reports All-logits-NaN degeneration (one repeated token forever) still fires on 0.1.39b / current main — first poison traced to the MoE output rows: garbage bits, varying layer #879 before the API returns 503. - Model:
ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, IQ3_S, two GGUF shards and the corresponding native pack. No external refusal-direction/control vector. No STRATA diagnostic or CPU/MMQ override environment during the release tests.
The same first-decode NaN was separately reproduced on SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF, IQ3_S. Its direct-I/O failure logs and new mmap/cold-restart tests are included below. A shared exact root cause is not proven.
Relevant arguments; paths below are placeholders:
--pack <iq3_s-pack> --native <shard1.gguf> --ple-gguf <shard2.gguf>
--expert-profile <expert-profile.bin> --expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5 --mtp <rt>
--max-context 262144 --kv int8 --kv-resident 32768
--vram-reserve-mib 1600 --pool-workers 8 --short-read 0 --vision
Vision encoding is on CPU. The candidate run adds only the explicit engine argument --ple-io mmap. Build and model identifiers, hashes and CPU flags are in environment.txt; download-receipt hashes are distinguished from hashes checked in this session.
Release 0.1.47: what was replayed
Requests were sent sequentially. The direct sequence used one cold-start engine process; mmap used a separate cold-start process. Each request below had zero prompt-prefix reuse, although expert-cache/process history persists between requests in a process.
| PLE I/O | Request | HTTP / output | Total time |
|---|---|---|---|
| direct | Short, 20 tokens | 200 / 5 tokens, Strata works. |
2.19 s |
| direct | Public min-wiki content from #3465, 1,717 tokens | 200 / 8 tokens | 9.09 s |
| direct | Captured user message, 33,133 tokens | 503 / 0 tokens; both attempts fail | 102.04 s |
| direct | Captured agent request, 61,244 tokens | 503 / 0 tokens; both attempts fail | 195.98 s |
| direct | Public-only Wikimedia repetition, 33,336 tokens | 200 / 8 tokens | 53.95 s |
| direct | Short after the long requests | 200 / 5 tokens, Strata works. |
1.31 s |
| mmap | Captured 33,133-token prompt | 200 / 8 tokens | 54.47 s |
| mmap | Captured 61,244-token prompt | 200 / 8 tokens | 98.76 s |
| mmap | Repeat of the 33,133-token prompt | 200 / 8 tokens | 52.34 s |
The 33K case is one user message, no tools, thinking disabled, prefixed with Diagnostic 1234567890.\n. The 61K case includes tools; streaming was disabled and its output capped. Long-request output limits were eight tokens: these are checks for finite first output, not validation of a complete answer. The 61K success contains reasoning tokens rather than a completed final answer.
Direct failures occur at positions 33132 / 61243, after all 33133 / 61244 prompt tokens were read. The retry drops cached state and rereads from token zero but fails again. Error:
the model's output became non-finite (NaN) twice in a row (#879);
the engine has dropped its cached state - send the request again
Private request bodies are excluded. The public controls pass, so they must not be presented as failing reproducers. Length alone does not explain the failure, and a failed long request does not invariably poison later short requests.
Additional SC117 mmap validation across a restart
SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF IQ3_S was then tested on the unmodified release 0.1.47, with --ple-io mmap, --short-read 0 and the other engine settings retained. No diagnostic/CPU/MMQ override environment was enabled. After the first replay set, the server was stopped and started by its tray controller; the replacement engine had a different process ID. The post-restart requests waited for READY.
| SC117 / release 0.1.47 / mmap | Prompt tokens | HTTP / output tokens | Total time |
|---|---|---|---|
| Short after startup | 20 | 200 / 5 | 1.55 s |
| Failing 33K prompt replay | 33133 | 200 / 8 | 51.68 s |
| Failing 61K prompt replay | 61244 | 200 / 8 | 92.14 s |
| Short after cold restart | 20 | 200 / 5 | 1.68 s |
| 33K replay after cold restart | 33133 | 200 / 8 | 51.31 s |
All five requests had zero prompt-prefix reuse. Short requests returned Strata works.; long cases were capped at eight output tokens, so they check finite first generation rather than full-answer quality. The same captured 33K/61K prompts were used. This extends the measured mmap workaround to SC117 and checks a further cold-start process; it still does not identify a particular I/O code defect.
Where non-finite values first appear in the diagnostic trace
Separate local CUDA build: source f57c0ad plus the attached diagnostic patch, sm_89, Release, STRATA_Q6K_EXPERTS=ON, STRATA_MMQ_KQUANTS=ON. This is not the release 0.1.47 binary. STRATA_DBG_NAN_WIN=1 and STRATA_DBG_NAN_END=1 enabled the probes. The 33K request was its first inference after startup; it failed on both attempts, HTTP 503 after 219.51 s.
First observed bad PLE input in this run:
pre-PLE residual: 0 non-finite; max finite |x| 0.522
PLE embedding: 96 non-finite; first token 4162, column 1632; max finite |x| 7.92e+06
post-PLE residual: 81920 non-finite; first token 4162
The embedding probe runs after its upload, before PLE projection GEMMs. Projections then become non-finite, followed by the layer-1 GDN input, recurrence state and MoE. Thus the downstream MoE/GDN reports do not establish those operations as the source. The trace has not yet separated disk/direct reads, row cache, decoding, host-buffer lifetime and upload.
The patch's label PLE normalized query scans a reused buffer after postops: at that point it holds normalized gated values, not the pristine query. Instrumentation adds copies/synchronizations and can change timing and failure boundaries.
Other checks already performed
| Check | Observation and limit |
|---|---|
| Token-to-table-row mapping | Reconstructed 33,133 IDs and compared the entire stream with Service.encode_prompt, not just its length. |
| PLE data on disk | Reread 3,120 FP16 scales at 39 positions around the first three bad embedding positions: 0 non-finite, max absolute scale about 0.00033236. This is not yet a byte-for-byte comparison of gathered CPU/GPU data with disk. |
| Dense weights | Scanned 38 stored F32/BF16/F16 tensors in layers 0/1: 0 non-finite. Native expert tensors and the entire PLE table were not exhaustively scanned. |
| Earlier #3465 narrowing | --short-read 0 initially passed prompts of 1,717 / 1,759 / 11,096 / 29,425 tokens. The later 33K/61K failures show that this was a workaround for the tested tail-window cases, not a general elimination of NaN. |
| CPU parity, earlier local build | iq_avx2_parity and iq_avx2_parity_iq3s_mt1: 2/2 passed, zero reported failures. This does not validate every inference/I/O path. |
| Earlier 0.1.46 CPU switches | STRATA_IQ256_GATHER=0 and STRATA_FORCE_AVX2=1 each passed a 33K replay, but a fresh-process control without either switch also passed. These runs do not establish a fix. |
| Earlier 0.1.46 debug scan | STRATA_DBG_NAN=1 reproduced the failure and detected bad MoE intermediates during batched reading. Its finite GDN slot-0 scan did not rule out other slots or an earlier bad input. |
| Earlier context/vision checks | With vision and KV residency removed and context falling back to 4096, short slices passed; the 33K prompt was rejected as too large. Restoring context 262144 without KV residency could not start with that measured VRAM budget. This does not isolate vision or KV streaming as the cause. |
| Earlier SC117 checks | STRATA_PREFILL_MMQ=0 did not eliminate #879. The FP16-KV attempt was interrupted/timed out and gives no conclusive result. These are not controlled 0.1.47 tests of the ISTA-DASLab model. |
Earlier checks used 0.1.46 or earlier local builds; they were not all rerun as a controlled 0.1.47 matrix. Small reduced triggers also changed across processes/instrumentation, so no reliable public minimal failing prompt is claimed.
Interpretation and remaining limits
--ple-io mmapis a tested workaround for this replay set, currently left enabled. It is not a demonstrated code fix or a guarantee for all future requests.- Auto prefill selected 3840 tokens with direct, 4352 with mmap, and 3584 in the diagnostic build. Separate processes, memory layout, history and chunk sizes are confounders; the I/O path has not been isolated with fixed chunks and repeated cold A/B runs.
- Static inspection of [Engine] Deterministic all-logits NaN (#879) on a fresh ~1.7K-token prompt of percent-encoded URLs (IQ3_S, CUDA, Windows) #3465's patch: its fallback handles NaN in a prompt-tail window by disabling short-read on retry. This failure is first decode with short-read already zero, so that fallback does not address this observed path.
Suggested next investigation
- Find the first data mismatch: compare the selected GGUF row bytes with Direct/PleReader output, decoded host floats, and uploaded GPU floats. Record the first row/block/column, expected/actual raw bits and whether values are finite. Start near token 4162 in this trace; the boundary may change between runs.
- Create a CPU-only row fixture: replay the implicated row IDs through
read_rowandgather_batchwith direct and mmap, comparing against buffered file reads. Test row-cache hits/misses, page-crossing rows, prefetch matching, completion tags and buffer reuse. These are investigation targets, not defects already proven. - Control the A/B: use the same fixed prefill chunk (for example 3840), model and arguments, multiple cold starts, and report every pass/failure. First verify that the fixed-chunk direct control still reproduces. Compare Windows storage/VM paths if a reader-level mismatch is found.
- Improve early diagnostics: an opt-in finiteness check on gathered PLE input could identify the first bad row before it contaminates recurrent state and surfaces later as All-logits-NaN degeneration (one repeated token forever) still fires on 0.1.39b / current main — first poison traced to the MoE output rows: garbage bits, varying layer #879. Keep the NaN guard enabled; do not hide the error or globally change defaults based on this one machine.
Not yet done: CPU/GPU byte-for-byte row comparison, CPU-only failing fixture, fixed-chunk multi-restart A/B, an isolated MTP-off test. The attached patch is instrumentation, not a production fix; its CUDA build was tested, while HIP/SYCL builds were not tested. No exact faulty operation or patch fixing it is claimed.
Attachments: logs.txt (relevant excerpts), measurements.json (sanitized results), environment.txt (configuration/identity), and diagnostic-probes.patch (instrumentation). No private prompts, local paths, account names, IP addresses or credentials are included.
The sanitized supporting files are inlined below.
logs.txt
A. OFFICIAL 0.1.47 / ISTA-DASLab GSQ-RCO IQ3_S / DIRECT
Initial attempt and automatic retry are separate prompt summaries. No request text is included.
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata serve: prompt chunk auto: 3840 tokens, a 199-slot ring
strata serve: 1178 MiB of VRAM free with everything loaded
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 51351 ms (645.2 tok/s), 0 generated in 52 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 50220 ms (659.8 tok/s), 0 generated in 53 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: non-finite logits (#879) in a first decode window at position 61243 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 61244 tokens = 0 reused + 61244 read in 106396 ms (575.6 tok/s), 0 generated in 47 ms (0.0 tok/s), drafts accepted 0 of 0, 5 checkpoints
strata serve: non-finite logits (#879) in a first decode window at position 61243 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 61244 tokens = 0 reused + 61244 read in 89188 ms (686.7 tok/s), 0 generated in 52 ms (0.0 tok/s), drafts accepted 0 of 0, 5 checkpoints
Passing direct controls:
strata serve: prompt 20 tokens = 0 reused + 20 read in 1767 ms (11.3 tok/s), 5 generated in 399 ms (12.5 tok/s), drafts accepted 3 of 3, 1 checkpoints
strata serve: prompt 1717 tokens = 0 reused + 1717 read in 8755 ms (196.1 tok/s), 8 generated in 312 ms (25.6 tok/s), drafts accepted 6 of 6, 1 checkpoints
strata serve: prompt 33336 tokens = 0 reused + 33336 read in 53487 ms (623.2 tok/s), 8 generated in 372 ms (21.5 tok/s), drafts accepted 3 of 7, 2 checkpoints
strata serve: prompt 20 tokens = 0 reused + 20 read in 1122 ms (17.8 tok/s), 5 generated in 180 ms (27.8 tok/s), drafts accepted 3 of 3, 1 checkpoints
B. SAME RELEASE/MODEL / --ple-io mmap / NO DIAGNOSTIC ENVIRONMENT
strata serve: prompt chunk auto: 4352 tokens, a 199-slot ring
strata serve: 1075 MiB of VRAM free with everything loaded
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 53338 ms (621.2 tok/s), 8 generated in 782 ms (10.2 tok/s), drafts accepted 3 of 6, 2 checkpoints
strata serve: prompt 61244 tokens = 0 reused + 61244 read in 98059 ms (624.6 tok/s), 8 generated in 406 ms (19.7 tok/s), drafts accepted 3 of 4, 5 checkpoints
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 51852 ms (639.0 tok/s), 8 generated in 388 ms (20.6 tok/s), drafts accepted 2 of 2, 2 checkpoints
C. SEPARATE LOCAL CUDA BUILD: f57c0ad + attached probes / DIRECT
Cold start, first inference: 33133-token replay. Not the official 0.1.47 executable.
strata dbg stage: pre-PLE residual start=3584 T=3584: 0 non-finite (first token -1 col -1), max finite |x| 0.522
strata dbg stage: PLE embedding start=3584 T=3584: 96 non-finite (first token 4162 col 1632), max finite |x| 7.92e+06
strata dbg stage: PLE projected key start=3584 T=3584: 20480 non-finite (first token 4162 col 0), max finite |x| 7.98e+06
strata dbg stage: PLE projected value start=3584 T=3584: 5120 non-finite (first token 4162 col 0), max finite |x| 1.69e+06
strata dbg stage: PLE normalized query start=3584 T=3584: 20480 non-finite (first token 4162 col 0), max finite |x| 18.9
strata dbg stage: PLE gated value start=3584 T=3584: 20480 non-finite (first token 4162 col 0), max finite |x| 1.4e+06
strata dbg stage: PLE gate start=3584 T=3584: 0 non-finite (first token -1 col -1), max finite |x| 0.974
strata dbg stage: post-PLE residual start=3584 T=3584: 81920 non-finite (first token 4162 col 0), max finite |x| 1.4e+06
strata dbg: layer1 GDN input p0=3584 T=3584: 20480 non-finite (first token +578 col 0), max finite |x| 6.55
strata dbg: GDN1 qkv p0=3584 T=3584: 81920 non-finite (first at token +578 col 0), max finite |x| 19.2
strata dbg: GDN1 hbuf p0=3584 T=3584: 266240 non-finite (first at token +578 col 0), max finite |x| 1.55
strata dbg: GDN1 z p0=3584 T=3584: 49152 non-finite (first at token +578 col 0), max finite |x| 7.09
strata dbg: GDN1 y p0=3584 T=3584: 18468864 non-finite (first at token +578 col 0), max finite |x| 0.0585
strata dbg: GDN1 state p0=3584 T=3584: 786432 non-finite (first at token +0 col 0), max finite |x| 0
strata dbg: layer1 MoE input p0=3584 T=3584: 7695360 non-finite (first token +578 col 0), max finite |x| 3.48
strata dbg: prompt end: last residual row 10240 non-finite (max |x| 0); PLE history 0 non-finite (max |x| 13.4); GDN state 0 0 non-finite (max |x| 2.08)
strata dbg: GDN slot 1 after 33126 tokens: 786432 non-finite (first at 0), max finite |x| 16.5
strata dbg: verify window at position 33132, row 0: 248320 of 248320 logits non-finite (token out 0)
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 119828 ms (276.5 tok/s), 0 generated in 110 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 99049 ms (334.5 tok/s), 0 generated in 61 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
D. ADDITIONAL SC117 ABLITERATED IQ3_S / OFFICIAL 0.1.47 / DIRECT
Earlier direct-I/O reproduction. New mmap results are in section F.
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 58747 ms (564.0 tok/s), 0 generated in 54 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: non-finite logits (#879) in a first decode window at position 33132 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 52544 ms (630.6 tok/s), 0 generated in 78 ms (0.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
E. EARLIER LOCAL CPU PARITY TESTS (not a fresh 0.1.47 inference test)
iq_avx2_parity: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (avx2 1, avx512 0), CPU -1, core type 00h
iq_avx2_parity: 0 failures
Test Passed.
iq_avx2_parity: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (avx2 1, avx512 0), CPU -1, core type 00h
iq_avx2_parity: 0 failures
Test Passed.
F. SC117 ABLITERATED IQ3_S / OFFICIAL 0.1.47 / MMAP / TWO COLD-START PROCESSES
Run 1: shortcut startup
strata serve: prompt chunk auto: 4096 tokens, a 199-slot ring
strata serve: 1214 MiB of VRAM free with everything loaded
strata serve: prompt 20 tokens = 0 reused + 20 read in 1318 ms (15.2 tok/s), 5 generated in 208 ms (24.1 tok/s), drafts accepted 3 of 3, 1 checkpoints
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 50995 ms (649.7 tok/s), 8 generated in 368 ms (21.7 tok/s), drafts accepted 5 of 5, 3 checkpoints
strata serve: prompt 61244 tokens = 0 reused + 61244 read in 91349 ms (670.4 tok/s), 8 generated in 543 ms (14.7 tok/s), drafts accepted 3 of 11, 5 checkpoints
Run 2: tray restart; new engine process
strata serve: prompt chunk auto: 4096 tokens, a 199-slot ring
strata serve: 1144 MiB of VRAM free with everything loaded
strata serve: prompt 20 tokens = 0 reused + 20 read in 1439 ms (13.9 tok/s), 5 generated in 212 ms (23.6 tok/s), drafts accepted 3 of 3, 1 checkpoints
strata serve: prompt 33133 tokens = 0 reused + 33133 read in 50602 ms (654.8 tok/s), 8 generated in 367 ms (21.8 tok/s), drafts accepted 5 of 5, 3 checkpoints
environment.txt
Engine: official v0.2.2 / 0.1.47 / f-022-rel / 76f90e58436d
CUDA: 13.0; Windows 10 Pro build 19045; Proxmox/QEMU VM, GPU passthrough
GPU: RTX 4070 SUPER 12 GB, sm_89; driver 610.88
CPU: Xeon E5-2697 v4, 16 vCPUs; RAM: 96 GiB
Guest CPU flags: 3dnow, 3dnowprefetch, abm, adx, aes, apic, avx, avx2, bmi1, bmi2, clflush, cmov, cx16, cx8, de, dts, erms, f16c, fma, fpu, fxsr, hle, ht, hypervisor, ia64, invpcid, lahf_lm, mca, mce, mmx, movbe, msr, mtrr, osxsave, pae, pat, pcid, pclmulqdq, pdcm, pge, pni, popcnt, pse, pse36, rdrnd, rdseed, rtm, sep, serial, smap, smep, ss, sse, sse2, sse4_1, sse4_2, ssse3, tm, tsc, tscdeadline, umip, vme, vmx, x2apic, xsave
API server: f57c0ad + request capture; not the v0.2.2 Python server
Model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF / IQ3_S
Pack conversion metadata: compat_bf16=false, no converted tensors
No external control vector; no diagnostic/CPU/MMQ overrides in release tests.
MTP enabled; vision encoder on CPU; expert profile enabled.
Arguments (placeholders replace local paths):
--pack <iq3_s-pack> --native <shard1.gguf> --ple-gguf <shard2.gguf>
--expert-profile <expert-profile.bin> --expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5 --mtp <rt>
--max-context 262144 --kv int8 --kv-resident 32768
--vram-reserve-mib 1600 --pool-workers 8 --short-read 0 --vision
Direct baseline: default --ple-io direct. Workaround: add --ple-io mmap.
Auto prefill chunk: direct 3840 / mmap 4352 / diagnostic build 3584.
VRAM free with everything loaded: direct 1178 MiB / mmap 1075 MiB.
Identity checks:
Release executable SHA256: cdc61c46c936a20d9e2c7225d2ee94a9d3feabb4ac7a330b7cf9e80b95f41d64
Release archive SHA256 (checked against published digest): 59833d875b2f3807791533f5dd077a0cb0ed08d4c1cab3136ad2ceb165871cd1
Pack index SHA256: b2a014878c3b61d6a6f915e194055a5e68ede34f9465dd1cb13249d902f008ac
Shard 1: 54817524224 bytes; saved download receipt: sha256 4c1eb2ceb4915e1192f4f386021897bde56a97f40a0bb78bb86465e0f7d2aca3
Shard 2: 28800138432 bytes; saved download receipt: sha256 316b46f3a2dbd68c900f43136ab9449f9dcc3725dfd8c794847c204bc161e113
GGUF hashes above are saved download receipts, not fresh full-file rehashes in this session.
GGUF header model name: Qwen3.8-Flash-Next GSQ-RCO; quantized_by: ISTA DASLab.
Separate diagnostic build:
Source: f57c0adcb31c00f3f216550c004dac4e713a9f70 + diagnostic-probes.patch
Executable SHA256: f5ef0caae8ece84b0367d6d2f0378e552636b59cc4c9fb07579d5c11d9658f1a
Release / sm_89 / STRATA_Q6K_EXPERTS=ON / STRATA_MMQ_KQUANTS=ON
Environment: STRATA_DBG_NAN_WIN=1, STRATA_DBG_NAN_END=1
This build predates 0.1.47 and adds synchronizations; it is not the release binary.
Probe label PLE normalized query refers to the reused normalized-gated buffer after postops.
Additional reproduction model: SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF IQ3_S.
Direct-I/O first-decode failure reproduced; mmap replays now pass before and after a cold restart.
SC117 mmap launch: same release 0.1.47 executable, SC117 pack/tokenizer/native/PLE files; no diagnostic environment.
SC117 mmap run 1: strata serve: prompt chunk auto: 4096 tokens, a 199-slot ring
SC117 mmap run 1: strata serve: 1214 MiB of VRAM free with everything loaded
SC117 mmap run 2: strata serve: prompt chunk auto: 4096 tokens, a 199-slot ring
SC117 mmap run 2: strata serve: 1144 MiB of VRAM free with everything loaded
measurements.json
{
"release_and_diagnostic_replays": [
{
"phase": "release-direct",
"test": "01-short",
"http_status": 200,
"prompt_tokens": 20,
"completion_tokens": 5,
"prompt_reused_tokens": 0,
"seconds": 2.19,
"nonfinite_error": false
},
{
"phase": "release-direct",
"test": "02-public-wiki",
"http_status": 200,
"prompt_tokens": 1717,
"completion_tokens": 8,
"prompt_reused_tokens": 0,
"seconds": 9.09,
"nonfinite_error": false
},
{
"phase": "release-direct",
"test": "03-repro-33133",
"http_status": 503,
"prompt_tokens": 33133,
"completion_tokens": 0,
"prompt_reused_tokens": 0,
"seconds": 102.04,
"nonfinite_error": true
},
{
"phase": "release-direct",
"test": "04-captured-61244",
"http_status": 503,
"prompt_tokens": 61244,
"completion_tokens": 0,
"prompt_reused_tokens": 0,
"seconds": 195.98,
"nonfinite_error": true
},
{
"phase": "release-direct",
"test": "05-public-long-wiki",
"http_status": 200,
"prompt_tokens": 33336,
"completion_tokens": 8,
"prompt_reused_tokens": 0,
"seconds": 53.95,
"nonfinite_error": false
},
{
"phase": "release-direct",
"test": "06-short-after-long",
"http_status": 200,
"prompt_tokens": 20,
"completion_tokens": 5,
"prompt_reused_tokens": 0,
"seconds": 1.31,
"nonfinite_error": false
},
{
"phase": "local-diagnostic-direct",
"test": "local-probe-33133",
"http_status": 503,
"prompt_tokens": 33133,
"completion_tokens": 0,
"prompt_reused_tokens": 0,
"seconds": 219.51,
"nonfinite_error": true
},
{
"phase": "release-mmap",
"test": "release-mmap-short",
"http_status": 200,
"prompt_tokens": 20,
"completion_tokens": 5,
"prompt_reused_tokens": 0,
"seconds": 3.43,
"nonfinite_error": false
},
{
"phase": "release-mmap",
"test": "09-mmap-33133",
"http_status": 200,
"prompt_tokens": 33133,
"completion_tokens": 8,
"prompt_reused_tokens": 0,
"seconds": 54.47,
"nonfinite_error": false
},
{
"phase": "release-mmap",
"test": "10-mmap-61244",
"http_status": 200,
"prompt_tokens": 61244,
"completion_tokens": 8,
"prompt_reused_tokens": 0,
"seconds": 98.76,
"nonfinite_error": false
},
{
"phase": "release-mmap",
"test": "11-mmap-33133-repeat",
"http_status": 200,
"prompt_tokens": 33133,
"completion_tokens": 8,
"prompt_reused_tokens": 0,
"seconds": 52.34,
"nonfinite_error": false
}
],
"offline_checks": {
"server_token_stream_exact_match": true,
"prompt_tokens": 33133,
"table_scale_positions": 39,
"table_scales_checked": 3120,
"nonfinite_scales": 0,
"max_abs_scale": 0.0003323554992675781,
"dense_tensors_checked": 38,
"dense_nonfinite": 0
},
"earlier_checks_not_a_new_0147_matrix": {
"cpu_parity": {
"tests": [
"iq_avx2_parity",
"iq_avx2_parity_iq3s_mt1"
],
"passed": 2,
"failed": 0
},
"engine_0146_33133_replays": [
{
"switch": "STRATA_IQ256_GATHER=0",
"passed": true,
"seconds": 105.5
},
{
"switch": "STRATA_FORCE_AVX2=1",
"passed": true,
"seconds": 103.9
},
{
"switch": "neither (fresh-process control)",
"passed": true,
"seconds": 139.0
}
],
"interpretation": "Control also passes: neither CPU switch is demonstrated to fix the fault.",
"context_4096": "Short slices pass; 33K prompt rejected. No isolation of vision/KV cause.",
"context_262144_without_residency": "Did not start with the measured VRAM budget.",
"sc117_mmq0": "Earlier #879 persisted; not a controlled 0.1.47 ISTA-DASLab test.",
"sc117_fp16_kv": "Interrupted/timed out: no conclusive result."
},
"not_done": [
"CPU/GPU/disk byte-for-byte row comparison",
"CPU-only failing row fixture",
"fixed-chunk multi-restart direct/mmap A/B",
"isolated MTP-off test"
],
"sc117_release_mmap_replays": [
{
"test": "01-sc117-short",
"http_status": 200,
"seconds": 1.55,
"prompt_tokens": 20,
"completion_tokens": 5,
"prompt_reused_tokens": 0
},
{
"test": "02-sc117-33133",
"http_status": 200,
"seconds": 51.68,
"prompt_tokens": 33133,
"completion_tokens": 8,
"prompt_reused_tokens": 0
},
{
"test": "03-sc117-61244",
"http_status": 200,
"seconds": 92.14,
"prompt_tokens": 61244,
"completion_tokens": 8,
"prompt_reused_tokens": 0
},
{
"test": "04-sc117-short-after-restart",
"http_status": 200,
"seconds": 1.68,
"prompt_tokens": 20,
"completion_tokens": 5,
"prompt_reused_tokens": 0
},
{
"test": "05-sc117-33133-after-restart",
"http_status": 200,
"seconds": 51.31,
"prompt_tokens": 33133,
"completion_tokens": 8,
"prompt_reused_tokens": 0
}
],
"sc117_restart_verified": {
"different_engine_process": true,
"ready_then_short_and_33133_passed": true,
"launch_and_restart_owned_by_tray": true,
"stdout_stderr_log_continues": true
}
}diagnostic-probes.patch
diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp
index e6ff842..a13f7e4 100644
--- a/include/strata/core/verify.hpp
+++ b/include/strata/core/verify.hpp
@@ -547,6 +547,11 @@ private:
float* h_ple_ = nullptr; float* m_ple_ = nullptr; // T * n_embd
int32_t* h_out_ = nullptr; int32_t* m_out_ = nullptr; // T argmax ids
uint32_t* h_nf_ = nullptr; uint32_t* m_nf_ = nullptr; // #879: kVerifyMaxT * kNonfiniteBlocks flags
+ // STRATA_DBG_NAN_WIN: per-layer non-finite flags scanned inside the window graph (device + host staging);
+ // entry l is layer l's input residual at pre(l), n_layers is the head's input, n_layers + 1 the mixed head input
+ uint32_t* dbg_lnf_ = nullptr; uint32_t* dbg_lnf_h_ = nullptr;
+ int64_t dbg_lnf_stride_ = 0; int64_t dbg_lnf_entries_ = 0;
+ float* dbg_lmax_ = nullptr; float* dbg_lmax_h_ = nullptr; // the same points' max |x| (one float an entry)
int nf_row_ = -1; ///< nonfinite_row()
int64_t nf_pos_ = -1; ///< nonfinite_pos()
uint32_t nf_mask_ = 0; ///< nonfinite_in(): bit t = row t
diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp
index 45bc70e..1d1e658 100644
--- a/include/strata/kernels/verify_kernels.hpp
+++ b/include/strata/kernels/verify_kernels.hpp
@@ -175,6 +175,10 @@ void row_top_prob_split(const float* logits, int n_rows, int n_vocab, const int3
/// be mapped host memory read after the window's own sync. Graph-capturable; CUDA and HIP.
inline constexpr int kNonfiniteBlocks = 16;
void logits_nonfinite_rows(const float* logits, int n_rows, int n, uint32_t* flags, void* stream);
+/// STRATA_DBG_NAN_WIN: *out = max(*out, max |x[0..n)|) by atomic max on the ordered float bits. `out` is one
+/// float per probe point, zeroed by the host before the window; a NaN shows as 0x7FFFFFFF-range bits (7FFFFFFF+
+/// ordered comparison still lands above every finite). Graph-capturable; CUDA and HIP.
+void dbg_maxabs(const float* x, int64_t n, float* out, void* stream);
/// True where row_top_prob_split runs as its own kernel (CUDA; AMD keeps the one-block row_top_prob, so it calls
/// that). STRATA_MULTI_BLOCK_ARGMAX=0: off.
bool multi_block_head_ops();
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index e8f747f..a3958be 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -1091,6 +1091,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
// ---- the window's inputs, from mapped staging. With branches the steps and positions (first read by the first
// layer's mixer) cross PCIe on side 1 beside the embedding and the first hyper-connection read.
+ if (dbg_lmax_ != nullptr) // STRATA_DBG_NAN_WIN: the max probes start from 0 on every replay
+ cudaMemsetAsync(dbg_lmax_, 0, (size_t) dbg_lnf_entries_ * sizeof(float), cs);
copy_i32_from_mapped(tok_, m_tok_, T, cs);
bool inputs_pending = false;
cudaStream_t in_s = cs;
@@ -1147,6 +1149,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
// ---------------------------------------------------------------- pre(l, group): up to the ring
auto pre = [&](int64_t l, int grp) -> bool {
const int tb = tb_[grp], te = te_[grp], n = te - tb;
+ if (dbg_lnf_ != nullptr && grp == 0 && G == 1) { // STRATA_DBG_NAN_WIN: layer l's input residual
+ strata::kernels::logits_nonfinite_rows(R_, (int) (T * HC), (int) N,
+ dbg_lnf_ + (size_t) (l - lb_) * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr)
+ strata::kernels::dbg_maxabs(R_, (int64_t) T * HC * N, dbg_lmax_ + (l - lb_), cs);
+ }
stamp(l, 0, grp);
const LayerView v(wt, l);
const char* pfx[2] = {"hc_attn_", "hc_ffn_"};
@@ -1165,6 +1173,17 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
if (grp == 0) {
if (ar_on()) wait_flag_ge(m_flag_, 1, cs);
copy_from_mapped(ple_, m_ple_, (int64_t) T * N, cs);
+ if (dbg_lnf_ != nullptr) { // STRATA_DBG_NAN_WIN sub-scans: layer 1's GDN state slot, then the staged PLE row
+ strata::kernels::logits_nonfinite_rows(ss.gdn_state + gdn_floats, (int) ((gdn_floats + N - 1) / N),
+ (int) N, dbg_lnf_ + (size_t) (g.n_layers + 5) * (size_t) dbg_lnf_stride_, cs);
+ strata::kernels::logits_nonfinite_rows(ple_, (int) T, (int) N,
+ dbg_lnf_ + (size_t) (g.n_layers + 2) * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr) {
+ strata::kernels::dbg_maxabs(ss.gdn_state + gdn_floats, (int64_t) gdn_floats,
+ dbg_lmax_ + (g.n_layers + 5), cs);
+ strata::kernels::dbg_maxabs(ple_, (int64_t) T * N, dbg_lmax_ + (g.n_layers + 2), cs);
+ }
+ }
}
float* normalized = (float*) ((uint8_t*) ss.ple.scratch + ple_block_scratch_bytes());
static const bool ple_batch_env = [] {
@@ -1258,6 +1277,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
if (!keep_rows) copy_from_mapped(hist_snap_ + (size_t) t * HS, hist, HS, cs);
}
pending = false;
+ if (dbg_lnf_ != nullptr && grp == 0) { // STRATA_DBG_NAN_WIN sub-scan: R_ after the PLE block wrote it
+ strata::kernels::logits_nonfinite_rows(R_, (int) (T * HC), (int) N,
+ dbg_lnf_ + (size_t) (g.n_layers + 3) * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr)
+ strata::kernels::dbg_maxabs(R_, (int64_t) T * HC * N, dbg_lmax_ + (g.n_layers + 3), cs);
+ }
}
auto gr_read_group = [&](int half, bool apply, float* inj_prev, float* inj_out) {
FusedGrArgs fa[kFusedGrMaxT];
@@ -1986,6 +2011,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
!cvec().covers(g.n_layers - 1);
auto post = [&](int64_t l, int grp) -> bool {
const int tb = tb_[grp], te = te_[grp], n = te - tb;
+ if (dbg_lnf_ != nullptr && grp == 0 && G == 1 && l == 1) { // STRATA_DBG_NAN_WIN sub-scan: R_ after layer 1's pre
+ strata::kernels::logits_nonfinite_rows(R_, (int) (T * HC), (int) N,
+ dbg_lnf_ + (size_t) (g.n_layers + 4) * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr)
+ strata::kernels::dbg_maxabs(R_, (int64_t) T * HC * N, dbg_lmax_ + (g.n_layers + 4), cs);
+ }
const uint32_t ring = (uint32_t) ((l - lb_) * G + grp + 1);
const int64_t cap = (int64_t) n * K, capx = (int64_t) max_t_ * K;
int32_t* pl = plan_ + (size_t) grp * (size_t) (plan_i32_ + 16);
@@ -2146,6 +2177,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
// ---- the head, T columns, and the argmax of each
stamp(g.n_layers, 0, 0);
+ if (dbg_lnf_ != nullptr && G == 1) { // STRATA_DBG_NAN_WIN: the head's input residual
+ strata::kernels::logits_nonfinite_rows(R_, (int) (T * HC), (int) N,
+ dbg_lnf_ + (size_t) g.n_layers * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr)
+ strata::kernels::dbg_maxabs(R_, (int64_t) T * HC * N, dbg_lmax_ + g.n_layers, cs);
+ }
{
const WeightRef *hn = wt.find("output_hc_norm.weight"), *hd = wt.find("output_hc_down.weight"),
*hu = wt.find("output_hc_up.weight");
@@ -2211,6 +2248,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
}
}
if (head_ != nullptr && head_->loaded()) {
+ if (dbg_lnf_ != nullptr && G == 1) { // STRATA_DBG_NAN_WIN: the mixed head input, before the Q8_1 quantize
+ strata::kernels::logits_nonfinite_rows(head_mixed_, T, (int) N,
+ dbg_lnf_ + (size_t) (g.n_layers + 1) * (size_t) dbg_lnf_stride_, cs);
+ if (dbg_lmax_ != nullptr)
+ strata::kernels::dbg_maxabs(head_mixed_, (int64_t) T * N, dbg_lmax_ + (g.n_layers + 1), cs);
+ }
try {
native_quantize_q8_1(head_mixed_, xq_, (int) N, T, cs);
if (g_mmvq_il() && strata::kernels::native_mmvq_il_supported(head_->type(), (int) T, (int) n_vocab_)) {
@@ -2290,6 +2333,28 @@ void Verifier::refresh_ar() {
bool Verifier::capture(int T, std::string& err) {
cudaGraphExec_t& exec_t = ar_off_ ? exec_nr_[T] : exec_[T];
if (exec_t != nullptr) return true;
+ if (dbg_lnf_ == nullptr && g_ != nullptr && std::getenv("STRATA_DBG_NAN_WIN") != nullptr) {
+ // STRATA_DBG_NAN_WIN: one flags word block per layer entry, written by scans captured into the window graph.
+ // Stride 8192 words leaves room for the sub-scans (the GDN state slot's ~115 rows), entries 50+ below.
+ dbg_lnf_stride_ = 8192;
+ dbg_lnf_entries_ = g_->n_layers + 10;
+ const size_t bytes = (size_t) dbg_lnf_stride_ * (size_t) dbg_lnf_entries_ * sizeof(uint32_t);
+ if (cudaMalloc(&dbg_lnf_, bytes) == cudaSuccess && cudaMemset(dbg_lnf_, 0, bytes) == cudaSuccess) {
+ dbg_lnf_h_ = new uint32_t[(size_t) dbg_lnf_stride_ * (size_t) dbg_lnf_entries_]();
+ if (cudaMalloc(&dbg_lmax_, (size_t) dbg_lnf_entries_ * sizeof(float)) == cudaSuccess &&
+ cudaMemset(dbg_lmax_, 0, (size_t) dbg_lnf_entries_ * sizeof(float)) == cudaSuccess)
+ dbg_lmax_h_ = new float[(size_t) dbg_lnf_entries_]();
+ else {
+ dbg_lmax_ = nullptr;
+ (void) cudaGetLastError();
+ }
+ std::fprintf(stderr, "strata verify: STRATA_DBG_NAN_WIN on: %lld scan entries (per-window layer inputs)\n",
+ (long long) dbg_lnf_entries_);
+ } else {
+ dbg_lnf_ = nullptr;
+ (void) cudaGetLastError();
+ }
+ }
{ // said before the capture: a process that exits inside it (#1275: Windows, 313 MiB free) leaves this line as the trace
size_t free_b = 0, total_b = 0;
if (cudaMemGetInfo(&free_b, &total_b) == cudaSuccess)
@@ -3005,7 +3070,10 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
}
for (int t = 0; t < T; ++t) out[t] = ((volatile int32_t*) h_out_)[t];
note_nonfinite(T, nullptr);
- if (static const bool dbg = std::getenv("STRATA_DBG_NAN") != nullptr; dbg) { // debug: the first non-finite head
+ if (static const bool dbg = std::getenv("STRATA_DBG_NAN") != nullptr; dbg ||
+ (std::getenv("STRATA_DBG_NAN_END") != nullptr && nf_row_ >= 0) ||
+ (dbg_lnf_ != nullptr && nf_row_ >= 0)) {
+ // With the lightweight hand-off probe, only copy logits after the normal guard has fired.
static bool reported = false;
if (!reported) {
std::vector<float> h((size_t) T * (size_t) n_vocab_);
@@ -3019,6 +3087,54 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
"(token out %d)\n", (long long) pos0, t, (long long) bad, (long long) n_vocab_, out[t]);
}
}
+ if (nf_row_ >= 0 && dbg_lnf_ != nullptr && !reported) reported = true; // the layer scan is the report
+ if (nf_row_ >= 0 && dbg_lnf_ != nullptr) {
+ const int64_t valid = (int64_t) T * g_->hc * strata::kernels::kNonfiniteBlocks; // this T's rows
+ if (cudaMemcpy(dbg_lnf_h_, dbg_lnf_, (size_t) dbg_lnf_stride_ * (size_t) dbg_lnf_entries_ * 4,
+ cudaMemcpyDeviceToHost) == cudaSuccess) {
+ int64_t first = -1;
+ std::string line;
+ char b[96];
+ for (int64_t e = 0; e < dbg_lnf_entries_; ++e) {
+ const int64_t words = e < g_->n_layers + 2 ? std::min<int64_t>(valid, dbg_lnf_stride_)
+ : dbg_lnf_stride_; // sub-scans: their own row counts
+ int64_t bad = 0;
+ for (int64_t w = 0; w < words; ++w)
+ bad += dbg_lnf_h_[(size_t) e * (size_t) dbg_lnf_stride_ + (size_t) w] != 0;
+ if (bad > 0) {
+ if (first < 0) first = e;
+ const char* nm = e == g_->n_layers + 2 ? "PLErow" : e == g_->n_layers + 3 ? "RafterPLE"
+ : e == g_->n_layers + 4 ? "RafterPre1" : e == g_->n_layers + 5 ? "GDNstate1" : "?";
+ std::snprintf(b, sizeof b, " %s%lld:%lldw", e < g_->n_layers ? "L" : e < g_->n_layers + 2 ? "H" : "",
+ e < g_->n_layers ? (long long) e
+ : e < g_->n_layers + 2 ? e - g_->n_layers
+ : (long long) 0, (long long) bad);
+ if (e >= g_->n_layers + 2) { std::snprintf(b, sizeof b, " %s:%lldw", nm, (long long) bad); }
+ line += b;
+ }
+ }
+ std::fprintf(stderr, "strata dbg: STRATA_DBG_NAN_WIN layer scan at position %lld (T=%d): entry = the "
+ "layer's input residual; L%lld = layer %lld's input, H0 = the head's input, "
+ "H1 = the mixed head input. First non-finite entry: %lld. Bad words:%s\n",
+ (long long) pos0, T, (long long) first,
+ first >= 0 && first < g_->n_layers ? (long long) first : -1LL, (long long) first,
+ line.empty() ? " none (every layer input finite: the NaN starts inside a layer's own "
+ "compute or the head)" : line.c_str());
+ if (dbg_lmax_ != nullptr && dbg_lmax_h_ != nullptr &&
+ cudaMemcpy(dbg_lmax_h_, dbg_lmax_, (size_t) dbg_lnf_entries_ * sizeof(float),
+ cudaMemcpyDeviceToHost) == cudaSuccess) {
+ std::string mx;
+ char c[64];
+ for (int64_t e = 0; e < dbg_lnf_entries_; ++e) {
+ if (e == g_->n_layers) mx += " | head:";
+ std::snprintf(c, sizeof c, " %lld=%g", (long long) e, dbg_lmax_h_[(size_t) e]);
+ mx += c;
+ }
+ std::fprintf(stderr, "strata dbg: STRATA_DBG_NAN_WIN max|x| per entry (50=PLErow 51=RafterPLE "
+ "52=RafterPre1 53=GDNstate1):%s\n", mx.c_str());
+ }
+ }
+ }
}
}
VDBG("window done\n");
diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu
index a451d9b..4e09bd6 100644
--- a/src/kernels/cuda/verify_kernels.cu
+++ b/src/kernels/cuda/verify_kernels.cu
@@ -943,6 +943,37 @@ void logits_nonfinite_rows(const float* logits, int n_rows, int n, uint32_t* fla
check("logits_nonfinite_rows");
}
+namespace {
+// STRATA_DBG_NAN_WIN: max |x| over n floats by atomic max on the ordered bits; fabsf makes a NaN positive, whose
+// bits sit above +inf's, so the winner reads back as nan/inf when it is one.
+__global__ void dbg_maxabs_kernel(const float* __restrict__ x, long long n, int* __restrict__ out) {
+ __shared__ int smax[256];
+ const long long i0 = (long long) blockIdx.x * blockDim.x + threadIdx.x;
+ const long long stride = (long long) gridDim.x * blockDim.x;
+ int m = 0;
+ for (long long i = i0; i < n; i += stride) {
+ const int v = (int) __float_as_uint(fabsf(x[i]));
+ if (v > m) m = v;
+ }
+ smax[threadIdx.x] = m;
+ __syncthreads();
+ for (int s = (int) blockDim.x / 2; s > 0; s >>= 1) {
+ if ((int) threadIdx.x < s && smax[threadIdx.x + s] > smax[threadIdx.x]) smax[threadIdx.x] = smax[threadIdx.x + s];
+ __syncthreads();
+ }
+ if (threadIdx.x == 0) atomicMax(out, smax[0]);
+}
+} // namespace
+
+void dbg_maxabs(const float* x, int64_t n, float* out, void* stream) {
+ if (x == nullptr || out == nullptr || n <= 0) return;
+ const int threads = 256;
+ const long long want = ((long long) n + threads - 1) / threads;
+ const long long cap = want < 4096 ? want : 4096;
+ dbg_maxabs_kernel<<<(unsigned) cap, threads, 0, (cudaStream_t) stream>>>(x, (long long) n, (int*) out);
+ check("dbg_maxabs");
+}
+
bool multi_block_head_ops() {
#if defined(__HIPCC__)
return false;
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index f00f1cc..0bb2f62 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -3039,6 +3039,21 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
std::fprintf(stderr, "strata pf-step: chunk from token %lld, layer %lld: %s took %.0f ms (%s / %s)\n",
(long long) p0, (long long) layer, what, ms, cudaGetErrorString(a), cudaGetErrorString(b));
};
+ auto nan_stage_probe = [&](const char* label, const float* ptr, int64_t rows, int64_t cols, int64_t start) {
+ if (std::getenv("STRATA_DBG_NAN_WIN") == nullptr) return;
+ cudaStreamSynchronize(m.cs);
+ std::vector<float> host((size_t) rows * cols);
+ if (cudaMemcpy(host.data(), ptr, host.size() * sizeof(float), cudaMemcpyDeviceToHost) != cudaSuccess) return;
+ int64_t bad = 0, first = -1;
+ double mx = 0;
+ for (size_t j = 0; j < host.size(); ++j) {
+ if (!std::isfinite(host[j])) { if (first < 0) first = (int64_t) j; ++bad; }
+ else mx = std::max(mx, (double) std::fabs(host[j]));
+ }
+ std::fprintf(stderr, "strata dbg stage: %s start=%lld T=%lld: %lld non-finite (first token %lld col %lld), max finite |x| %.3g\n",
+ label, (long long) start, (long long) rows, (long long) bad,
+ (long long) (first < 0 ? -1 : start + first / cols), (long long) (first < 0 ? -1 : first % cols), mx);
+ };
for (int64_t l = LB; l < LE; ++l) {
core::progress_beat(); // the serve watchdog: a prompt chunk of 8192 tokens is still moving
if (l > LB) pf_step("reading the prompt (batched, step sync): the experts and the rest of layer", l - 1);
@@ -3049,6 +3064,10 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
if (!ple_land()) return false;
}
// ---- the PLE block at layer 1, token by token (its conv reads the previous tokens' rows)
+ if (l == 1 && ple_on) {
+ nan_stage_probe("pre-PLE residual", m.R, T, D, p0);
+ nan_stage_probe("PLE embedding", m.ple_emb, T, N, p0);
+ }
if (l == 1 && ple_on && ple_batch) {
// the whole chunk at once, in sub-batches carved from the idle scratch region: the key and value
// projections as GEMMs (a token at a time they re-read ~52 MB of BF16 key per token on the IQ
@@ -3082,10 +3101,15 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
}
m.gemm.bf16(e16, pw.value_bf16, val, nb, N, N);
if (e16_lo) m.gemm.bf16(e16_lo, pw.value_bf16, val, nb, N, N, 0, 1.0f);
+ nan_stage_probe("PLE projected key", key, nb, HD, p0 + s0);
+ nan_stage_probe("PLE projected value", val, nb, N, p0 + s0);
try {
strata::kernels::native_ple_postops_batch(key, m.R + s0 * D, val, ss.ple.hist, pw, qn, gated,
gate, (int) nb, m.cs);
} catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; }
+ nan_stage_probe("PLE normalized query", qn, nb, HD, p0 + s0);
+ nan_stage_probe("PLE gated value", gated, nb, HD, p0 + s0);
+ nan_stage_probe("PLE gate", gate, nb, HC, p0 + s0);
}
stats_.ms_ple += ms_since(tp);
} else if (l == 1 && ple_on) {
@@ -3103,6 +3127,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
}
stats_.ms_ple += ms_since(tp);
}
+ if (l == 1 && ple_on) nan_stage_probe("post-PLE residual", m.R, T, D, p0);
for (int half = 0; half < 2; ++half) {
// ---- the hyper-connection read of this half
const char* pre = half == 0 ? "hc_attn_" : "hc_ffn_";
@@ -3182,6 +3207,23 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
m.mixed_bf_lo);
}
+ if (l == 1 && std::getenv("STRATA_DBG_NAN_WIN") != nullptr) {
+ cudaStreamSynchronize(m.cs);
+ std::vector<float> input((size_t) T * N);
+ if (cudaMemcpy(input.data(), m.mixed, input.size() * sizeof(float), cudaMemcpyDeviceToHost) == cudaSuccess) {
+ int64_t bad = 0, first = -1;
+ double mx = 0;
+ for (size_t j = 0; j < input.size(); ++j) {
+ if (!std::isfinite(input[j])) { if (first < 0) first = (int64_t) j; ++bad; }
+ else mx = std::max(mx, (double) std::fabs(input[j]));
+ }
+ std::fprintf(stderr, "strata dbg: layer1 %s input p0=%lld T=%lld: %lld non-finite "
+ "(first token +%lld col %lld), max finite |x| %.3g\n",
+ half == 0 ? "GDN" : "MoE", (long long) p0, (long long) T,
+ (long long) bad, (long long) (first < 0 ? -1 : first / N),
+ (long long) (first < 0 ? -1 : first % N), mx);
+ }
+ }
if (wd_q8.data) { // the HC GEMMs that read the scratch are enqueued: the next user waits for them
core::NativeDense::hc_bf16_release(m.cs, 0, 0);
core::NativeDense::hc_bf16_release(m.cs, 1, 0);
@@ -3208,6 +3250,40 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
const int64_t ld_y = pf_pad() && T >= std::max<int64_t>(pf_switch_min_t(), 64) ? ZV + ZV_PAD : 0;
gdn_recurrence(state, m.hbuf, m.gate, m.beta, m.z, (const float*) wnm->data, EPS, m.y, m.y_h, T, m.cs,
ld_y);
+ if (std::getenv("STRATA_DBG_NAN_WIN") != nullptr && gdn_index == 1) {
+ // The first slot the #879 poison shows in: layer 1's GDN inputs and outputs, token by token,
+ // so the non-finite value is pinned to one tensor and one prompt position.
+ cudaStreamSynchronize(m.cs);
+ static std::vector<float> dbgb;
+ auto scan = [&](const char* what, const float* d, int64_t rows, int64_t cols) {
+ if (rows <= 0 || cols <= 0) return;
+ const size_t n = (size_t) rows * (size_t) cols;
+ if (dbgb.size() < n) dbgb.resize(n);
+ if (cudaMemcpy(dbgb.data(), d, n * 4, cudaMemcpyDeviceToHost) != cudaSuccess) return;
+ int64_t bad = 0, first_row = -1, first_col = -1;
+ double mx = 0;
+ for (int64_t r = 0; r < rows; ++r)
+ for (int64_t c = 0; c < cols; ++c) {
+ const float x = dbgb[(size_t) r * (size_t) cols + (size_t) c];
+ if (!std::isfinite(x)) {
+ ++bad;
+ if (first_row < 0) { first_row = r; first_col = c; }
+ } else mx = std::max(mx, (double) std::fabs((double) x));
+ }
+ if (bad > 0 || mx > 65504.0 || p0 == 0)
+ std::fprintf(stderr, "strata dbg: GDN1 %s p0=%lld T=%lld: %lld non-finite (first at "
+ "token +%lld col %lld), max finite |x| %.3g\n", what,
+ (long long) p0, (long long) T, (long long) bad, (long long) first_row,
+ (long long) first_col, mx);
+ };
+ scan("qkv", m.qkv, T, (int64_t) g.ssm_conv_channels);
+ scan("hbuf", m.hbuf, T, (int64_t) g.ssm_conv_channels);
+ scan("gate", m.gate, T, (int64_t) g.ssm_v_heads);
+ scan("beta", m.beta, T, (int64_t) g.ssm_v_heads);
+ scan("z", m.z, T, (int64_t) g.ssm_value_dim);
+ scan("y", m.y, T, (int64_t) g.ssm_v_heads * (int64_t) g.ssm_state_size);
+ scan("state", state, g.ssm_state_size, (int64_t) g.ssm_state_size * (int64_t) g.ssm_v_heads);
+ }
pt.mark(kPfGdnOut, cs);
if (!native_proj(m.gemm, wo, m.y_h, m.bo, T, v.name("ssm_out.weight"), err, 0, ld_y)) return false;
++gdn_index;
@@ -4919,7 +4995,9 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
// stage can return while later GPUs are still processing the previous chunk.
ss.ple_prev[0] = prev[0];
ss.ple_prev[1] = prev[1];
- if (std::getenv("STRATA_DBG_NAN") != nullptr) { // debug: the state the prompt leaves for the token path
+ if (std::getenv("STRATA_DBG_NAN") != nullptr || std::getenv("STRATA_DBG_NAN_END") != nullptr ||
+ std::getenv("STRATA_DBG_NAN_WIN") != nullptr) {
+ // Inspect the hand-off without the expensive per-layer MoE scans enabled by STRATA_DBG_NAN.
cudaStreamSynchronize(m.cs);
auto bad = [&](const float* d, int64_t n) {
std::vector<float> h((size_t) n);
@@ -4936,6 +5014,25 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
std::fprintf(stderr, "; GDN state 0");
bad(ss.gdn_state, 64 * 1024);
std::fprintf(stderr, "\n");
+ // STRATA_DBG_NAN_WIN: every GDN slot (state + conv), as each chunk leaves it. A state can go non-finite
+ // while the outputs stay finite; the poison then surfaces in the next window's recurrence (#879).
+ std::vector<float> slot((size_t) gdn_floats);
+ for (int64_t i = 0; i < ss.gdn_alloc; ++i) {
+ if (cudaMemcpy(slot.data(), ss.gdn_state + (size_t) i * gdn_floats, slot.size() * 4,
+ cudaMemcpyDeviceToHost) != cudaSuccess)
+ break;
+ int64_t c = 0, c_at = -1;
+ double mx = 0;
+ for (int64_t v = 0; v < gdn_floats; ++v) {
+ const float x = slot[(size_t) v];
+ if (!std::isfinite(x)) { if (c == 0) c_at = v; ++c; }
+ else mx = std::max(mx, (double) std::fabs((double) x));
+ }
+ if (c > 0 || mx > 65504.0)
+ std::fprintf(stderr, "strata dbg: GDN slot %lld after %lld tokens: %lld non-finite (first at %lld), "
+ "max finite |x| %.3g\n", (long long) i, (long long) n, (long long) c,
+ (long long) c_at, mx);
+ }
}
if (cudaStreamSynchronize(m.cs) != cudaSuccess) {
err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError());
Activity
nanachi1212 commentedon Oct 21, 2026
Possible relation to #3602. Swift 1.5 IQ3_S on 0.1.47 (commit 76f90e5), Windows, CUDA 13, RTX 5080 16 GB, 64 GB RAM, no AVX-512.
Symptom: every answer is a run of
!!!!(finish=length). Tested with greedy, temp 1.0 + top-p/top-k, temp 0.7, thinking on and off (enable_thinking=false), with--spec 4kept, with and without--mtp, and with--vram-reserve-mib 1280. The engine loads normally and reports 97–100% expert-cache hits, with decode at 90–190 tok/s.Config used: no
--ple-ioflag, so the defaultdirectapplies. The startup log says--ple-io direct measures slow here ... kept direct.Not yet tested:
--ple-io mmap. I can run it and report back.nanachi1212 commentedon Oct 21, 2026
Follow-up:
--ple-io mmapdoes not fix Swift 1.5 IQ3_S here.Same setup as my earlier comment (0.1.47, commit 76f90e5, Windows CUDA 13, RTX 5080 16 GB, 64 GB RAM,
--spec 4and--mtpkept), now with--ple-io mmapadded. Test prompts (17 * 23 + 5, and a one-sentence Traditional Chinese question about photosynthesis) still return only!!!!and finish=length. So for Swift the PLE I/O mode does not look like the cause, and the mmap workaround from this issue does not carry over to the Swift pack.The engine loads normally in both modes and reports expert-cache hits of 97–100%.
PLE reader: a failed read no longer leaves the next ticket waiting fo…
PLE table: STRATA_PLE_CHECK=1 names the first row that decodes to an …
Merge f-027-gh-eng-d (#3602 #3948 #3638 #3702 #3797 #2409 #3844: audi…
Merge f-027-gh-eng: GitHub engine items for 0.2.7
Niko1221 commentedon Oct 27, 2026
Thank you. Fixed in 0.2.7: A failed read no longer leaves the next ticket waiting for ever; stress test for the direct table reader. We could check it with unit tests and builds, but not on your hardware or with your workload. Please try 0.2.7 and tell us how it goes.
Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7
fix(engine,serve): port four upstream v0.2.7 fixes
PLE reader: a failed read no longer leaves the next ticket waiting fo…
PLE table: STRATA_PLE_CHECK=1 names the first row that decodes to an …
fix: sanitize non-finite/implausible PLE table rows at decode (Niko12…
fix: sanitize non-finite/implausible PLE table rows at decode (Niko12…
fix: sanitize non-finite/implausible PLE table rows at decode (Niko12…
test: cover ple_sanitize_row with a host-only unit test (Niko1221#3602/…
test: cover ple_sanitize_row with a host-only unit test (Niko1221#3602/…
sycl: re-migrate the port to upstream 0.1.54 (from its last re-migrat…
PLE の読み: 読みが失敗した後の次の読みを待たせ続けない(upstream ebe3383)
engine: upstream Strata 0.2.4〜0.2.7.1 の修正を取り込む(Issue #49、その 2)
F-368: v0.2.7 correctness port set (upstream commits f8ff342/01e404c1…