[Engine] sm_80 tuning measured on a CMP 170HX (GA100): prompts up to 2.0x, decode +20 %, conversation switches 5-10x faster; five PRs to follow #3712

Area

[Engine] (speed, kernels, models, the server and API)

What do you want to do?

Make Strata fast on sm_80 (GA100-class) cards with every expert resident. Today none of the engine's per-card tables has an sm_80 entry: the wide dense dequant default, the MMVQ-IL rows and the fused prompt-expert thread shapes. #3252 reports A100 slowdowns, and you said you have no sm_80 card. We have one and spent the last weeks tuning for it. We would like to agree on scope here before opening five PRs. The PRs are written and build on CUDA sm_80, HIP gfx1100 and HIP gfx906. Their per-PR GPU runs are in progress.

Card and setup. CMP 170HX: GA100, sm_80, 70 SMs, 64 GB HBM2e, PCIe Gen2 x4 (the engine's probe reads 1.6 GB/s host to device). The card runs at a 1650 MHz clock ceiling with a +250 MHz V/F offset (an undervolt), and its power cap was 235 W or 250 W across these runs.

• Model: Qwen3.8-Flash-Next IQ3_S native pack, every expert resident (24,576 slots, 46.84 GiB).
• Engine args: --kv int8 --spec 3 --mtp --prefill 8192 --max-context 262144 --conversation-cache-mib 8192.
• Software: Linux, driver 610, CUDA 13.3. Measured against 9de04cd (0.2.1.4). The PR branches are now rebased onto main 011c9e6 (0.2.2, engine 0.1.47). None of the 238 commits since 9de04cd overlaps these changes. The rebase had two textual conflicts, each resolved by keeping both sides. This box has two sockets, so 0.2.2's NUMA-split CPU expert pool turns itself on here. The rerun below sets STRATA_NUMA_POOL=0 in both arms, because the automatic split on the arena path gives remote reads and an unstable decode (NUMA pool split on a two-socket Xeon (2x 6226R, 2x R9700): on/off numbers, and the "arena path" limitation #3708).

Overall result. Fresh engine per run, same box.

How the two columns were set up:

• Stock is the release binary with STRATA_PF_FUSED=1 STRATA_OWNED_PRICE=exact.
• Our branch turns on the switches of all five PRs and several existing upstream switches, for example STRATA_SIDE_MIXER=2, STRATA_GR_UP75=1 and STRATA_QIL=1.
• It also turns on two switches that are in no PR: STRATA_GDN_OUT_NORM_LOOP and STRATA_MTP_PRIO_ALWAYS. Neither showed a gain.
• The park and restore "before" times are from our builds with the new switches off. We have no conversation-check run of the stock binary.
• Greedy answers (11 prompts, 256 tokens, temperature 0) are identical to stock for every bitwise change.

A same-session rerun is queued and will replace this table. It runs on the rebased branches and has three rows:

1. stock;
2. stock plus upstream's existing switches;
3. the five PRs merged, with their switches.

Switches. Every switch is opt-in except the two sm_80 defaults in PR 1, which are bitwise. The effects below were measured on our working builds. Per-switch off/on pairs on the PR branches themselves are in progress, and the table will be updated with them.

The five PRs. Each PR has one commit per feature, so any of them can be split further.

1. sm_80 defaults: the two bitwise default changes plus PF_FUSED_NT and PF_BATCH. Upstream's default fused kernels keep identical SASS. Cost: the 256-thread instances add 365,632 bytes of sm_80 device code to every CUDA build.
2. Prefill over a slow host link: four opt-in switches. Metal gets stubs for the new entry points.
3. Tensor-core prefill: f16 fused experts and the GDN scan. Both are opt-in and change rounding. Stacked on PR 1. Teacher-forced KL is not run yet.
4. Conversation cache: background copies, a ready checkpoint pool, VRAM slots. Opt-in and CUDA-only. Compatibility: an existing STRATA_CKPT_POOL=1 now also pre-allocates 2 ready checkpoints (+236 MB of host RSS).
5. Decode: graph-node priorities and the Q6_K hoisted-load GEMV. Stacked on PR 1, because its table builds on PR 1's sm_80 rows.

PR 1's MMVQ-IL table is the sm_80 counterpart of #3453 (sm_89). The two touch the same table list.

Validation so far.

• Rebased on 011c9e6, every commit of PRs 1-4 and their branch tips build on CUDA sm_80, HIP gfx1100 (ROCm 7.14) and HIP gfx906, with no new warning in any touched file compared with 011c9e6 built the same way. PR 5's builds are finishing.
• Before the rebase, PR 3's sources also built for the Dockerfile's architecture list (75, 80, 86, 89, 120). That check has not been redone on 011c9e6.
• Not run: SYCL and Metal builds, any AMD hardware, CUDA 12, and other sm_80 cards.

Questions.

1. Five thematic PRs, or a finer split?
2. Is the 365 KB of sm_80 device code acceptable in every CUDA build, or should it go behind a CMake option?
3. PRs 3 and 5 are stacked on PR 1. Should we open them as drafts naming PR 1, or wait for PR 1 to merge?

What do you do today instead?

We run our fork's branch (github.com/linxuhao/Strata, opt-sm80 and opt-sm80-hip) with the switches above.

🤖 Generated with Claude Code