[paltian]

Problem and result

STRATA_HC_Q8=1 reads the hyper-connection down/up projections from Q8_0 copies. On a single GPU it also leaves their bf16 copies out of the arena (add_hc_skip). On a layer split that skip was gated off (!multi_gpu in generate.cpp), so each card kept the bf16 copies and got the 0.63 GiB of Q8_0 copies on top. HC_Q8 cost VRAM on a split instead of saving it.

With this change a split skips the bf16 copies the same way, and each card gives the freed VRAM to its expert cache. Measured on 0.2.1 (c093ac26), UD-Q4_K_XL, 2x RTX 5060 Ti, --layer-split auto:

	before	after
canonical tensors skipped	303	495 (+192 = the 4x48 hc_*_down/up rows)
dense weights loaded from the pack	1416 MiB	216 MiB
expert cache, both cards	2967 slots (9.15 GiB)	3370 slots (10.29 GiB)
auto split	K=6 (CUDA1 2341 slots)	K=7 (CUDA1 2735 slots)
prompt 32k / 120k	1128 / 1334 tok/s	1160 / 1380 tok/s (+2.8% / +3.4%)
decode 4k / 32k (10 pairs)	28.2 / 33.1 tok/s	30.5 / 35.5 tok/s (+7.8% / +7.4%)

Related discussion

This is #2427, closed by the 2026-10-19 tracker reset, rebased on 0.2.1 and re-validated. It answers the review there: the default reach via setup (below), a same-config A/B on the split (Validation), and the native_dense.cpp comment (fixed). It extends the single-GPU HC_Q8 path (b60aab1) to a layer split. #704 was a requantization approach and was closed as "not with a layer split".

Default reach, needs the owner's call: setup.py:3190 writes STRATA_HC_Q8=1 for CUDA UD-* installs. So this change is active by default on every CUDA UD-* layer split, not only for users who set the variable. There the prompt path and the per-token read switch from the bf16 copy to the Q8_0 dequantized into scratch (the read one GPU has used since b60aab1). The freed VRAM also moves the auto placement (split point, prompt chunk, cache slots). Greedy output on such a split is therefore no longer bit-identical to 0.2.1 (KL below). STRATA_HC_Q8=0 keeps the full bf16 read as before.

Approach

One gate in src/program/generate.cpp: drop !multi_gpu from the add_hc_skip condition. Plus a comment fix in native_dense.cpp.

The prompt path and the token read already take these rows through hc_bf16_view, which dequantizes the Q8_0 into per-device scratch. So a split can drop the bf16 copies exactly as one GPU does. add_hc_skip names all hc_*_down/up rows. Each stage's loader still limits itself to its own layers through in_range, and a skipped row is quantized on the host from the pack file. Kernels are untouched.

What changes on a split: the verify windows already read the Q8_0 copies with and without the patch (STRATA_HC_Q8=1: 0.63 GiB of Q8_0 hyper-connection projections in both logs). The prompt path and the per-token read move from the bf16 copy to the Q8_0 through hc_bf16_view, as on one GPU.

Validation

Environment: AMD Ryzen 7 5700X3D (8C/16T), 94 GB RAM, 2x RTX 5060 Ti 16 GB (sm_120, driver 610.57.04; CUDA0 on a x4 link, probed 3.1 GB/s, CUDA1 x8, 14.3 GB/s), Debian kernel 6.12.107, CUDA build -DCMAKE_CUDA_ARCHITECTURES="86;120". Patched 3908d716 (= this branch) against stock c093ac26, both built with identical options. The server config is my production config (UD-Q4_K_XL, --spec 4 --mtp, --kv int8, --prefill auto, --layer-split auto, STRATA_HC_Q8=1).

Passed:

• ctest -j 4 on 3908d716: 66/68 passed. ple_parity and platform_memory_test (mlock, ulimit -l) fail identically on stock.
• A/B speed, docs/COMMUNITY_BENCHMARKS.md protocol: a fresh server per arm and round, alternating order, one discarded warm-up, unique prompt prefix (0 tokens reused), thinking off, temperature 0. Engine timings come from the engine log, TTFT from the client stream. All requests finish: stop, no errors.
  • Run 1: 5 rounds, ~4k/32k/120k prompts, 256 tokens. Prompt 4k +1.0% (4/5 pairs faster), 32k +2.8% (5/5, +2.1..+3.2%), 120k +3.4% (5/5, +3.0..+6.2%), TTFT 120k 93.0 → 90.1 s.
  • Run 2: 10 rounds, ~4k/32k prompts, 512 tokens. Prompt 32k +3.3% (10/10, +2.2..+4.1%). Decode 4k +7.8% (10/10, +1.0..+10.4%), 32k +7.4% (10/10, +2.7..+13.1%).
  • The same A/B on 0.2.0 (e88ee39d, 5 rounds) gave prompt 32k +2.9%, 120k +3.1%, 5/5 pairs each.
  • The single-pair A/B posted in [Engine] HC_Q8: skip the layers' bf16 hyper-connection copies on a layer split too #2427 (one 91,712-token prompt, a different reserve config) saw prompt and decode unchanged. The runs here have 5-10 pairs per size and see the gain consistently.
• --batch 2 on the split, two concurrent requests (45k + 49k and 4k + 4k tokens), patched, stock and stock again. Run in the 0.2.1 default (one serial group) and with --batch-groups 2 (pipelined, 2 groups of 1). All answers complete, correct and coherent, no errors in the engine logs.
• Output check, same prompts on both arms: 1 of 4 cases bit-identical, the others diverge after 85-504 characters with equivalent content. With the split pinned (layer_split "7", --prefill 4096) the divergence moves later. Changing only the placement on stock (auto vs pinned) diverges about as early as the patch does, so placement is a large part of the difference, beside the bf16 → Q8_0 read.
• KL on the split (on [Engine] HC_Q8: skip the layers' bf16 hyper-connection copies on a layer split too #2427's branch, 2026-10-15, the same one-line change; not repeated on 0.2.1): tools/quant_quality.py, --layer-split auto, CUDA_VISIBLE_DEVICES=0,2, reserve raised so the MTP head fits, the same on both arms. Prompt path (prompt/pcompare, 20 prompts of 256-1000 tokens, full vocabulary): top-1 100% (20/20), KL mean / median / max 0.00134 / 0.00042 / 0.00781. Decode (run/compare, teacher-forced, 4,995 positions): top-1 95.10%, KL mean / p99 0.02858 / 0.4818, perplexity 4.062 vs 4.020. The decode figure includes the cache-size effect (1539 → 1932 slots): changing only the cache size moves it about as much (STRATA_HC_Q8: hyper-connection projections as int8 + fp32 scale per 32 #704: top-1 94.1%, KL 3.4e-2). Draft acceptance unchanged (53.7% vs 54.0%, 3 runs each).

Per-run results

Run 2, decode tok/s (rounds 0-9):

• 4k stock 28.1 28.5 28.6 28.4 27.8 27.8 30.3 28.0 28.0 29.2
• 4k patched 31.0 29.7 29.7 30.6 30.7 30.2 30.6 30.3 29.6 30.8
• 32k stock 30.0 31.4 32.1 33.7 33.1 32.4 33.6 33.6 33.8 33.1
• 32k patched 32.6 35.5 36.3 36.5 36.6 35.2 36.2 35.6 34.7 34.5

Run 2, prompt 32k tok/s:

• stock 1119.7 1122.8 1124.5 1134.4 1127.3 1135.2 1127.1 1138.7 1125.3 1128.4
• patched 1164.4 1157.8 1159.6 1161.6 1167.0 1160.3 1165.3 1170.9 1171.1 1170.4

Run 1, prompt tok/s (rounds 0-4):

• 4k stock 496.8 490.9 497.2 493.1 498.3 / patched 495.1 507.9 506.2 502.0 501.1
• 32k stock 1126.7 1137.6 1135.3 1124.5 1128.3 / patched 1150.4 1161.6 1171.2 1157.6 1160.2
• 120k stock 1336.4 1317.0 1301.8 1334.3 1338.5 / patched 1379.8 1380.4 1382.3 1378.3 1378.6

Run 1, decode tok/s:

• 4k stock 28.3 29.2 28.9 28.5 28.8 / patched 19.1 30.3 30.4 31.6 29.4
• 32k stock 32.9 31.2 33.3 32.0 31.8 / patched 27.2 35.1 36.1 35.2 35.9
• 120k stock 33.2 34.5 32.3 32.9 33.1 / patched 32.9 38.2 36.7 35.6 37.0

Failed:
None observed. One observation in the decode numbers: 3 of 20 patched servers across the 0.2.0 and 0.2.1 five-round runs decoded 10-35% slower for their whole lifetime (0 of 20 stock). They showed the same cache layout, cache hit rate, draft acceptance and graph captures as the fast ones. The 10-round decode run (10 patched servers) had none, and the cause is not found.

Not run:

• The KL check on 0.2.1 itself (the numbers above are from the pre-reset branch, same change). The single-GPU KL of the Q8_0 read is in docs/QUANTS.md (UD-Q4_K_XL: mean 0.0562 → 0.0526, p99 0.925 → 0.716).
• HIP / SYCL / Metal builds (the gate is shared code. gfx12 defaults HC_Q8 on, so a HIP split would be affected the same way).
• A split with --keep-canonical (the gate sits inside !keep_canonical, unchanged).
• More than two GPUs, and --batch-mtp.
• No test in ctest covers this path. I did not add one: the change is a gate on an existing, tested path, and a unit test would only restate it.

Limits

• The gain comes from the freed VRAM, so it depends on how far the experts already fit: on cards that hold all experts it is smaller.
• Part of the measured gain is the auto split moving (K=6 → 7). That is a consequence of the freed VRAM, not separate from it, but a manual layer_split sees only the cache part.
• In batch mode stock itself is not reproducible run to run for one of the four cases (both group modes), so batch output comparisons are only content checks.

🤖 Generated with Claude Code

[qiuzan2001]

One thing the description does not mention: setup writes STRATA_HC_Q8=1 into the run config of every CUDA install of an Unsloth UD-* file (unsloth_default_env, setup.py:2283-2284, any GPU count). So this gate change reaches every setup-made UD layer split on CUDA by default, not only users who set the variable; there the prompt path and the token read switch from the bf16 copy to the per-device dequantized scratch (hc_bf16_view). A prompt and decode A/B on the split (same config, with and without the change) would show the speed side of that. Also, the comment at src/core/native_dense.cpp:411-415 ("A row the arena kept (--keep-canonical, a layer split) is quantized on the GPU from the arena's bf16", "the single-GPU default: add_hc_skip") no longer matches the code after this change.

[paltian]

Using this version as my daily driver with 2 and 3 GPUs.

[paltian]

A same-setup A/B for this patch, on a 2x RTX 5060 Ti layer split. Identical 91,712-token prompt, gpu: [0,2], arena (every expert in RAM), --vram-reserve-mib 1000 --vram-reserve-later-mib 1400, STRATA_HC_Q8=1, STRATA_ROUTE_TAIL_SKIP=7, --layer-split auto. Only the patch differs:

	without the patch	with the patch
canonical tensors skipped	303	495
dense weights loaded	1416 MiB	216 MiB
free VRAM, CUDA1 (before the cache)	7.70 GiB	8.84 GiB
CUDA1 expert cache	2144 slots	2538 slots (+18.4%)
profiled pairs (both cards)	5285 (93.4%)	5597 (94.0%)
decode cache hit rate	73.4-73.8%	77.1%
experts over PCIe	9.4-9.5%	8.0%
prompt 91,712 tokens	1276.3 / 1299.5 tok/s	1296.5 tok/s
decode	38.8 / 38.9 tok/s	39.4 tok/s

The freed 192 tensors are exactly the 4x48 hc_*_down/up rows. The prompt and decode are unchanged; the expert cache and its hit rate grow and the PCIe share falls. The decode and draft numbers are content-dependent (the summaries generated differ in length: 5050/6474 vs 3491 tokens), so the load metrics and the identical prompt are the reliable part - the draft acceptance difference (67% -> 63%) is not a patch effect.

[paltian]

A KL check for the one value change this patch makes, on the layer split. tools/quant_quality.py prompt / pcompare (the batched prompt path, full vocabulary, 20 prompts of 256/512/768/1000 tokens), and run / compare (teacher-forced through the decode windows, 4,995 positions) for the decode.

The yardstick is single-GPU by design, so the split is spelled out: --layer-split auto in the engine args plus CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,2, and the reserve raised (--vram-reserve-mib 2000 --vram-reserve-later-mib 4000, the same on both setups) so the MTP draft head fits beside the split. Both runs then load - the unpatched one skips 303 tensors, the patched one 495; nothing else differs.

Prompt path (where the patch changes the read: the bf16 copy -> the Q8_0 through hc_bf16_view):

top-1	100.00% (20/20)
KL mean / median / max	0.00134 / 0.00042 / 0.00781

Decode path (the same hc_bf16_view on the per-token reads, layer.cpp:1410; the verify windows already take the raw Q8_0 with and without the patch, verify.cpp:1096):

top-1	95.10%
KL mean / p99	0.02858 / 0.4818
perplexity	4.062 vs 4.020

The decode figure is larger than the prompt path's (0.0013) because the decode is teacher-forced and the difference accumulates over the sequence. It is not the read alone: the patch also frees ~1.1 GiB per card, so the expert cache grows (1539 -> 1932 slots), and the cache's GPU/CPU rounding moves the yardstick KL by about as much on its own (#704: "Changing only the cache size (no HC_Q8) moves it as much: top-1 94.1%, KL 3.4e-2"). The draft acceptance is unchanged (same prompt, T=0, 3 runs each: 53.7% vs 54.0%).

The direction is the Q8_0 read the single-GPU path has used since b60aab1 (docs/QUANTS.md: closer to Q8_0, mean KL 0.0562 -> 0.0526, p99 0.925 -> 0.716). (An earlier decode figure of 0.00000 in this thread came from a single-GPU run where the unpatched engine also skips the bf16 rows, so it did not exercise the patch. The prompt check without --mtp - before the reserve was raised - gave mean KL 0.00316, median 0.00019, max 0.03132.)

[paltian]

The A/B is in the table above: same config (2x RTX 5060 Ti, --layer-split auto, STRATA_HC_Q8=1, identical 91,712-token prompt), only the gate differs - the head card's expert cache grows 2144 -> 2538 slots, the hit rate 73.5 -> 77.1%, and the prompt (1296.5 vs 1276-1300 tok/s) and decode (39.4 vs 38.8-38.9) are unchanged. You are right that unsloth_default_env writes STRATA_HC_Q8=1 for every CUDA UD-* install: that is the intended reach - a setup-made split gets the freed VRAM and reads the same Q8_0 the single-GPU path has used since b60aab1.

And you are right about native_dense.cpp:411-415: "the single-GPU default: add_hc_skip" no longer describes the code. I have updated the comment in the branch (b7ea7b7).

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[paltian]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

[paltian]

Rebased on 0.2.1 and re-validated. The description now follows the template. On the two points @qiuzan2001 raised: the description now states that setup enables `STRATA_HC_Q8=1` for CUDA UD-* installs.
I ran the A/B you asked for, 5 and 10 rounds, prompt +2.8-3.4%, decode +7-8%, faster in every pair. Per-run numbers are in the description's Validation section.