Skip to content

helper cache: configurable VRAM allowance with allocation-floor checks - #1458

Closed
saikiran-rs wants to merge 1 commit into
Niko1221:mainfrom
saikiran-rs:community/helper-vram-reserve-20261007
Closed

saikiran-rs wants to merge 1 commit into
Niko1221:mainfrom
saikiran-rs:community/helper-vram-reserve-20261007

Conversation

@saikiran-rs

Copy link
Copy Markdown
Contributor

Helper expert caches use a fixed 512 MiB allowance during admission. Add STRATA_REMOTE_RESERVE_MIB so a measured setup can explicitly choose a different allowance, while retaining 512 MiB when unset.

What changed

  • Apply the allowance to automatic sizing and explicit helper slot counts.
  • Require space for the existing device work buffers plus 16 MiB of headroom before filling the cache.
  • Reject malformed, out-of-range and undersized values. Admission arithmetic uses subtraction to avoid overflow.
  • Add a CPU-only policy test covering the unchanged default, parsing, the buffer floor and capacity boundaries; document the setting in SECOND_GPU.md.

This changes helper cache admission only. The historical 128 MiB setting was used on one Linux RX 7900 XTX + RX 6800 XT machine; the default remains 512 MiB. Its combined tuning results are documented in the companion benchmark report, with no isolated allowance speedup claimed.

Validation

Based on upstream d5ea7133741e67743c0e886bb426c0ce8d69cf6c (0.1.40.3):

  • CPU-only CMake build and ctest -R '^remote_reserve_test$': passed.
  • Changed remote_experts.cpp compiled against the HIP SDK in a fresh isolated build configuration: passed.
  • git diff --check: passed.

No new GPU/model inference run was performed on this isolated PR. Historical full-context/quality results refer to the earlier combined research build. CUDA and Windows runtime testing remain unmeasured.

Related #1184 handles primary-device selection and resident-RAM helper support; #848 handles a layer split's reserve. Their changes are not duplicated here.

@Niko1221

Copy link
Copy Markdown
Owner

Thanks. No speed claim, so it is outside this speed round; it can go in as a plain setting if you rebase it.

@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants