[hireymage]

The rest of the Turing port, rebased onto main (0.1.22).

What main already covers after the rebase (dropped from the PR):

• the portable pre-sm_80 QSA scorer (main has its own fp32-FMA path) and the sm_80 guard on the tensor-core prompt attention (a751715),
• a per-device fused_gr shared-memory opt-in capped at the card's limit.

What the PR adds on top (+43/−20, single commit):

• the runtime floor: device_info refuses anything below compute capability 7 — main's runtime check still demands sm_120, so a binary built for sm_75 would never start,
• fused_gr chunking on top of main's per-device opt-in: the down kernel carries tokens in slices that fit the card's opt-in (Turing, 64 KB → 6-token slices; 4 tokens when no opt-in is reported), so full 8-token windows no longer fail with "invalid argument" on Turing,
• the setup/CMake floor is 7.5 (RTX 20 / 30 / 40 / 50) without an experimental flag: CMake's arch refusals, setup.py's gpu_problem and the GPU hint/error texts, and docs/MULTI_GPU.md updated.

Compiled clean for CMAKE_CUDA_ARCHITECTURES=75 (121/121, CUDA 12.6, Linux). Runtime verified end-to-end on an RTX 2070 (sm_75) at the 0.1.20 base: model loads, OpenAI/Anthropic serving works, greedy output identical to the sm_120 build.

[Adamyno]

Data point: RTX 2080 Ti 22GB (sm_75) + Skylake-X CPU — incl. comparison vs llama.cpp on the same box

Hardware: i9-7980XE (18C/36T, quad-channel DDR4-3600, 125 GiB RAM), modded RTX 2080 Ti 22GB,
Linux, CUDA 13.0, fork at turing-sm75 (PR #87 head).

Setup: Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, max-context 262144, KV int8 + --kv-resident 32768,
expert cache auto → 9246/24576 experts resident (~15 GiB), MTP draft.

Serve-log results:

• output 45.2 tok/s @2.4K prompt, 49.3 warm; 46.9 tok/s @100k context
  (tg effectively context-independent: 40.2 → 46.9 tok/s going 68K → 100K)
• prefill 380 tok/s @2.4K → 677 tok/s @100k (99,927 tokens in 147.5 s)
• expert cache hit 90-93%; KV streaming 99.8% VRAM hit @short ctx, 92.4% @100k
• MTP draft acceptance ~0.73

Same-box comparison vs llama.cpp (FN tensor 3 cards: Q4_K_XL, 512K ctx, MTP n2, all three 2080 Tis):
~27.6-29.7 tok/s decode, ~132 tok/s prefill @37.7K.
So Strata gives ~+70-80% decode on ONE GPU vs llama.cpp across THREE, at 3.0 bpw vs 4.25 bpw,
and ~5x prefill. Not a controlled A/B (quant and GPU count differ), but the direction is clear:
a single 22GB card now runs what the 3-GPU llama.cpp setup was serving, faster.

Build/install experience: smooth — zero source changes needed. Three notes that may help others:

1. On rate-limited networks the FetchContent llama.cpp clone crawls; the documented
   STRATA_GGML_DIR escape hatch (depth-1 fetch of the pinned commit) worked first try,
   cmake configure in 4 s.
2. CUDA 13.0 nvcc not on PATH — pass -DCMAKE_CUDA_COMPILER explicitly; arch 75 accepted
   by the fork guard.
3. Sharp edge in the pack step: strata_pack.py asserts on IQ quants. The right tool for
   IQ3_XXS is iq_pack.py --compat-bf16 (projections are already native BF16, zero
   conversions), then mtp_fetch / mtp_pack --experts q2_0 / mtp_rt for the draft.
   Worth a line in the docs since the error doesn't say which tool to use instead.

CPU note for older Intel + Turing combos: Skylake-X has AVX-512F/BW but not VNNI/VBMI;
the engine detects this and correctly runs the AVX-2 multi-token path ("this CPU has no
AVX-512") — still hits all numbers above with 17 workers. The 12GB reference box (Zen4,
full AVX-512+VNNI) reaches the same speed with 4x fewer resident experts, so the ceiling
on Skylake-X is CPU-side, not VRAM-side.

Bonus: it also runs behind llama-swap on-demand — 48 s cold start, GPU fully released
when idle (21.7 GB → 9 MiB). Happy to test anything on this box.

[Niko1221]

Thanks @hireymage. Part of this is in main now (the runtime floor from #81 in the 0.1.23 branch, and 0.1.22's sm_80 guard on the new prompt-attention kernel, which had broken sm_75 builds). Could you rebase onto main (0.1.22) so the rest of the Turing port is on top of that?

[hireymage]

Rebased onto main (0.1.22, a751715). Your 0.1.22 already covers the portable pre-sm_80 QSA scorer and the sm_80 guard on the prompt-attention kernel, so the rebased PR carries only the rest (+43/−20): the runtime floor (device_info now refuses below compute capability 7 instead of requiring sm_120 at run time), fused_gr token slicing on top of your per-device opt-in (6-token slices on Turing's 64 KB, 4 when no opt-in is reported), and the 7.5 floor in CMake/setup.py without the experimental flag. docs/MULTI_GPU.md updated to match.

Compiles clean for arch 75 (121/121, CUDA 12.6, Linux). Runtime verification from the 0.1.20 base (RTX 2070) stays valid for the parts the rebase kept: model loads, serving works, greedy output identical to the sm_120 build.

Heads-up: #88 (pascal-sm61) is stacked on the old turing-sm75 and will need the same treatment once this settles.

[Adamyno]

Follow-up on the rebase (f6a3328) from this box:

• Pulled the rebased head, builds clean on CUDA 13.0 too (arch 75, STRATA_GGML_DIR
  route, zero source changes).
• Quick serve re-check vs the pre-rebase numbers in my comment above:
  • server up in 42 s cold start (48 s before)
  • 52.8 tok/s warm @1.5K prompt (49.3 before)
  • 693 tok/s prefill @13k, 60.1 tok/s generation, draft acceptance 41/42 on that run
  • expert cache hit 85.2% warm; same config as before (262K ctx, KV int8,
    --kv-resident 32768)
• The 6-token fused_gr slicing moves the needle a little; nothing regressed.

Porting note for other Linux users: since the rebase, the HTTP layer belongs to
python -m serve.server --engine strata --config ... --port N — the binary itself
takes no --port anymore. (Our llama-swap entry had to switch to the server wrapper.)

Happy to re-verify once this lands in main / 0.1.23.

[hireymage]

Rebased on v0.1.23 (cd97daf), so the branch now carries the batched verify window from #109 as well.

Conflicts were small and both resolved on the rebased head:

• device.cu / setup.py / CMakeLists.txt: the runtime floor from Accept sm_80+ GPUs at runtime, matching the build guard and the README #81 (cc < 8) lowered to cc < 7 with the RTX 20 series texts (build guard: LESS 75).
• fused_gr.cu: 0.1.23 added the stage-profiler stamp after the down-kernel launch; the Turing chunked down-launch (opt-in smem below 80 KB -> token slices) keeps that single gpu_stamp call after its chunk loop, so the profiler sequence is unchanged.

Verified on Linux: configure + build for arch 75 (183/183, tests off), strata-device accepts sm_75 and refuses sm_61 with the new texts. The new per-device smem opt-ins from D-1 (#4755942) fall back cleanly on Turing (qsa_prompt_attn returns false above the card's opt-in, and the fused-GR chunking caps the dynamic smem at what the card reports).

[Niko1221]

Thanks @hireymage, and @Adamyno for the 2080 Ti numbers. One fix: cc_major < 7 lets Volta 7.0/7.2 through while the message and CMake say 7.5. We'll compare major*10+minor against 75 when merging (and lower it once a V100 run is reported, see #101), then gate it: output on RTX 30/40/50 byte-identical, the AMD build unchanged. Planned for the next release.

[Niko1221]

Merged into 0.1.27 (released): your commit, under your name, rebased onto 0.1.26, with two changes. The runtime floor compares major*10+minor against 75, so Volta isn't let through; it sits beside main's HIP check. And a card that reports no shared-memory opt-in gets the token slice that fits the 48 KB default for its own tile, so AMD's smaller tile keeps one launch. The ready-made engine now includes sm_75 code. RTX 30/40/50 output is byte-identical to 0.1.26 on all four quants we test, and the AMD build and tests pass. The ready-made image encoder has no sm_75 code yet, so with images on the GPU, setup compiles it. Thanks @hireymage, and @Adamyno for the numbers; reports from RTX 20 owners are very welcome.

[eightman999]

Verified on Tesla V100 (sm_70): current main a790805 (Engine 0.1.27) builds and runs with only the CMake architecture floor lowered from 75 to 70.

No CUDA source changes were needed, and native_qsa_score.cu was built normally. The existing __CUDA_ARCH__ >= 800 guards / pre-sm_80 fallback cover Volta correctly.

• GPU: Tesla V100-PCIE-32GB (sm_70)
• CUDA: 12.4 / nvcc 12.4.131
• OS: Ubuntu-family Linux
• Configure: PASS after changing only LESS 75 -> LESS 70
• Build: PASS, all targets including native_qsa_score.cu
• Runtime: PASS with Qwen3.8-Flash-Next Q2_0
• Resident experts: 20,193
• Expert cache: 26.0 GiB
• Decode: ~46.0-49.6 tok/s warm
• CUDA/runtime errors: none
• Output sanity check: correct (17 × 23 = 391)

This is consistent with my earlier V100 measurements (~47-52 tok/s), so I did not observe a performance regression.

One caveat: I saw intermittent gcc/nvcc segfaults during highly parallel (-j16) builds on this machine; retrying with -j1 completed cleanly. This looked like local host-toolchain flakiness rather than an sm_70-specific failure.

So for Volta, architecture-floor-only support passes: lowering the floor to sm_70 is sufficient on current main. The file exclusion from my #101 is no longer needed.

[Adamyno]

Closing the loop: v0.1.27 (a790805) verified on this box (RTX 2080 Ti 22GB, CUDA 13.0, arch 75).

Builds clean from main (STRATA_GGML_DIR route, zero source changes) and serves through our
llama-swap on-demand entry with the official thinking-mode sampling forwarded
(temperature=1.0, top_p=0.95, top_k=20):

• prefill 695 tok/s @13k prompt
• generation 42.6 tok/s @13k with the official sampling (drafts accepted 48 of 80)
• expert cache hit 89.5%, KV streaming 97.1% VRAM-hit @13k
• greedy spot-checks unchanged from the rebase smoke (57.9 tok/s warm @1.5K)

The runtime-floor fix (major*10+minor >= 75) and the 48 KB tile slice both behave as
described on sm_75. Thanks for the merge — and for crediting the numbers.