[zsariboga]

What

STRATA_HC_HOST=1 (opt-in, CUDA): the 192 hyper-connection down/up projections (1.17 GiB BF16 on Swift 1.5 IQ3_XXS)
leave VRAM and live in pinned, mapped host memory. The freed VRAM goes to the expert cache.

• They are loaded through the normal loader into a temporary device arena (so they take the engine form), copied once
  into pinned mapped host memory, and the arena is freed.
• The verify windows read the Q8_0 copy, so this needs STRATA_HC_Q8=1. A native GGUF whose hc down/up rows are BF16
  (the GSQ-RCO quants) now also gets a load-time Q8_0 copy, so the Q8 read is taken there too.
• The prompt path stages a layer's pair to the device on its own stream with two slots: the next half's pair is copied
  while this half computes. Without staging it read the pair over PCIe and prompt reads fell from ~1,900 to ~220 tok/s.

Without the variable nothing changes: the default path is the same code as before.

Why

On a 12 GB card the expert cache is what decides decode speed, and the hyper-connection pair is the largest dense
block left in VRAM. With STRATA_HC_Q8=1 the decode windows already read the Q8_0 copy, so the BF16 pair is only
needed by the prompt path, which can stage it.

Measured

RTX 5070 Ti Laptop 12 GB (PCIe 5.0 x8), Core Ultra 7 255HX, 64 GB DDR5-5600, Windows 11, Swift 1.5 IQ3_XXS (GSQ-RCO),
this branch (main 7695859 + this commit), the same config in both arms except STRATA_HC_HOST:

	STRATA_HC_HOST unset	STRATA_HC_HOST=1
expert cache	2,494 slots (4.04 GiB)	3,221 slots (5.22 GiB), +29%
agent decode, ms/token	11.376 / 11.778 (mean 11.58)	11.602 / 11.346 (mean 11.47), -0.9%
CPU expert time per verify window	13.4 / 15.4 ms	11.5 / 12.8 ms, -15%

Agent decode: 32 real opencode turns from held-out sessions, 256 tokens each, temperature 1.0, engine ms/token,
order off, on, on, off. On this branch the bigger cache takes ~15% of the CPU experts' time away, but the decode
window barely moves (-0.9%, inside the run-to-run noise): with the stock routing the GPU side is the longer path here.

On my daily fork, which also runs cache-aware routing (PR #1017) and so turns cache slots into speed much more
directly, the same change (2,968 -> 3,221 slots there) measured -8.4% ms/token, and the two-slot prompt staging another -1.3%; warm prompt reads were unchanged (1,923 vs
1,868-1,909 tok/s).

Checked

• CUDA build (sm_120, MSVC + CUDA 13) and both arms above start, answer and call tools.
• Not built for HIP or SYCL: I have no such card. The host-staging code is in the CUDA prompt path and the loader;
  STRATA_HC_HOST is read only there.

About me

I run Strata every day on a gaming laptop (the hardware above) as the backend of opencode, and keep a fork with the
measurements, the config and the rejected experiments in
docs/LAPTOP_12GB_TUNING.md
(55 -> ~140 tok/s engine decode on this laptop since 0.1.38). My earlier PRs: #1536, #1537.

[Niko1221]

Thanks for the clear write-up. On this tree the change is -0.9% (noise) because the larger cache only pays through cache-aware routing, which is not in main, and the files around it have moved (the HC_Q8 views). We are not porting it now. If routing lands, or you can show a gain of a few percent on current main, please send a rebased version.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.