The GGUF branch of load_experts_gguf still has the 4095-byte std::ifstream defect (0.02 -> 1.87 GiB/s, 42 min -> 54 s)

Follow-up to #89. That PR fixed the pack path (experts.bin -> load_experts_ranges, now fread 8 MiB). Its own description says:

src/core/expert_source.cpp has the same std::ifstream read in its GGUF branch (load_experts_gguf); it is not touched here because the pack path is the one this report measures, but it is the same pattern and probably wants the same fix.

This is that report: a normal Windows install that never builds experts.bin hits the defect, and the fix that works today is to make the engine take the fast path.

Summary

The engine's own line, same pack, same machine, only the pack contents changed:

strata generate: loaded 39.97 GiB at 0.02 GiB/s   <- no experts.bin (load_experts_gguf)
strata generate: loaded 39.97 GiB at 1.87 GiB/s   <- with experts.bin (load_experts_ranges)

Why this config is the one that hits it

ArenaExpertSource::open picks the reader by whether the pack has experts.bin:

// src/core/expert_source.cpp:1209 (upstream main)
const bool from_gguf = !std::ifstream(path, std::ios::binary) && lay.native && !lay.gguf_off.empty() && !gguf_.empty();
...
const LoadStats st = from_gguf ? load_experts_gguf(gguf_, a->data(), lay, threads)
                               : load_experts_ranges(path, a->data(), loff, lbytes, threads, /*chunk=*/8u << 20);

setup.py only writes experts.bin in the low-RAM branch:

# setup.py:1941  (normal install: no --experts-bin)
elif not (pack / "native_experts.txt").exists() or not (pack / "tokenizer" / "vocab.json").exists():
    # every tensor as the GGUF stores it; the experts are read from the GGUF at start (seconds to build)
    run([... "tools/iq_pack.py", "--gguf", str(shards[0]), "--out", str(pack)], env=env)
if low_ram and not (pack / "experts.bin").exists():
    run([... "tools/iq_pack.py", ..., "--experts-bin"], env=env)

So the comment "the experts are read from the GGUF at start (seconds to build)" holds only where std::ifstream issues bulk reads. On MSVC it does not, and the normal install takes the slow branch with no experts.bin to fall back on.

load_experts_gguf (upstream main, lines 1107/1133):

std::ifstream f;
...
f.read((char*) buf.data(), (std::streamsize) n);

The arithmetic on this machine

The GGUF branch reads per tensor role in chunks of per[r] * 16 experts. For IQ3_XXS layer 0's gate tensor that is 7,686,880 B (7.33 MB) per read. On MSVC, basic_filebuf::xsgetn loops fread(_Ptr, 1, 4095, _Myfile) (_Read_size = 4095), so:

• 7,686,880 / 4095 = 1,878 4095-byte calls per chunk
• 42,912,972,800 / 4095 = ~10.5 million calls for the whole arena

At the ~3.5 us/call this box shows, that is ~37 min of pure per-call overhead. The measured phase was 2,519 s = 42.0 min. The disk is not the limit: an unbuffered 1 MB sequential read of the same file measures 668 MB/s here.

Reproduction

Windows 11, MSVC 14.44.35207, CUDA 13.0, i9-14900KF, 64 GB DDR5-6000, RTX 5090 D v2 24 GB, ZHITAI Ti600 2TB NVMe. Pack: ISTA-DASLab GSQ-RCO IQ3_XXS (2 shards + PLE), 48 layers, 512 experts, arena 42,912,972,800 B. Engine 0.1.27, --expert-cache auto --spec 4 --mtp ... --max-context 131072 --kv int8 --kv-resident 32768 --vram-reserve-mib 700.

# the fix, one command, one time (~17 min, writes 39.97 GiB)
python tools/iq_pack.py --gguf <shard 1> --out <pack> --experts-bin

# then start as usual; the engine detects experts.bin by itself, no config change

Measured across 3 full stop/start cycles of the real launcher (.bat -> server.py -> strata.exe), timing from launch to /health = 200:

Average 64.1 s, best 59.0 s, worst 74.3 s. Before: 2,519 s. ~39x.

Decode was checked so the fix is not confused with a speed change: same prompt 6 times in a row, steady state from run 3 = 32.34 tok/s (35.71 / 31.05 / 32.20 / 30.39), expert cache hit 93-96%. Consistent with what this box did before the fix.

Why nobody reported it

• Linux/glibc is unaffected - the 4095 split is MSVC-specific.
• Anyone whose pack has experts.bin (low-RAM mode, or a Q2_0 AVX-512 build) is on the fast path and sees 35-55 s.
• The hint the engine prints points elsewhere:

strata generate: hint: ~24x below what this hardware streams from a normal launch. If Strata is
started by Task Scheduler or a service, register the task with Priority 4 (Normal) and ...

That text is about #126 (Task Scheduler, 0.05 vs 1.42 GiB/s). Here the process was started from a terminal at Normal priority - log line 15 (BelowNormal) and line 232 (High) are byte-identical at 0.02 GiB/s, and a disk-priority sweep (VeryLow/Low/Normal/High) showed no effect. The hint sent me after priority and large pages for a while; the real cause is the reader. Worth a separate hint when the load rate is under ~0.2 GiB/s and the pack has no experts.bin.

Suggested fix

Point load_experts_gguf at the same fopen/fread + _fseeki64 body load_experts_ranges now uses (#89), or reuse load_experts_ranges by feeding it the GGUF offsets lay.gguf_off already carries. The blob scatter step can stay as it is; only the read call needs to change. On this machine the pack path reached 1.83-2.04 GiB/s, so the GGUF path should land in the same range and the normal install would stop needing a 40 GB experts.bin just to start in a minute.

I am not attaching a patch - I have no MSVC tree here to build and verify against - but the measurements above should be enough to size it. Happy to re-run any measurement if useful.