Repository navigation
serve: --moe-slots auto, --moe-prefetch (off by default) - #169
Merged
Merged
Conversation
…e-prefetch, off by default auto reads the model's expert sizes (moe/GgufIndex) and takes MemAvailable, less the other tensors the GPU holds and a reserve (8 GiB or 15%), divided by the mean expert size, clamped to the model's experts with a floor of 64 per layer. The gate-ahead prefetch is now off unless --moe-prefetch N asks for it: on Qwen3.8-Flash-Next it doubled decode time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… streaming Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
enabled auto-merge (squash)
September 26, 2026 23:50
|
Docs7 for 1bit-monster/engine
Commit |
Collaborator
Author
|
Reviewed. |
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 1, 2026
…ation, #170 -ot/input-layer) + held hrx-system (#786) * hw-gate: pass the gfx device library to its own configure, and recognise the zero-cases exit The gate's build failed at #772 with clang++: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library - not a kernel result and not the pin: hw-gate.sh configures third_party/llama.cpp directly, so cmake/hrx.cmake's fix (engine#713, #771) never reached it, and the kernel-corpus compile is its own amdclang++ invocation that does not inherit the llama.cpp build's CMAKE_C/CXX_FLAGS either. The box only builds because build-candidate{19,20}.sh export HIP_DEVICE_LIB_PATH into the whole run - which the workflow does not. - Read the gfx device library from the resolved toolchain (its pip tree, where the 2026-10-31 layout put it) and pass it as GGML_HRX_HIP_FLAGS="-O3 --rocm-device-lib-path=…" plus GGML_HRX_HIP_COMPILER=<toolchain>/bin/amdclang++ - the same pair cmake/hrx.cmake now passes - and export HIP_DEVICE_LIB_PATH as a fallback for anything else in the tree. - A changed flag set drops the cached CMakeCache.txt (a marker beside .gpu-kernels-pin), so an existing build tree cannot keep configuring without it. - ops: the pin carries the zero-cases guard (tests/test-backend-ops.cpp, fork engine#657) - a run that executed no case exits 1 and says "error: no test case ran (check -b/-o/-p and --test-file): refusing to report a pass". Recognise that as its own outcome: ops SKIP naming the selection that matched nothing, never a PASS (an empty selection must not look green) and never a bare FAIL that reads like a kernel fault. Verified on the box without a GPU, with HIP_DEVICE_LIB_PATH/ROCM_PATH/HIP_PATH unset: the gate's own configure arguments against /opt/rocm-therock produce all 50 gfx1151 code objects and the embedded .inc, where the same configure without the flag reproduces the CI error. bash -n clean. * Bump HRX: llama.cpp 1d32c22609d6 + held hrx-system 82804197bcbe The fork line moved two commits past 1991650479db - #169 (hrx: host-weight residency instrumentation, weak leases, and an opt-in bound, engine #669) and #170 (loader: an explicit -ot wins over the arch's lazy-read hint, LLAMA_INPUT_LAYER_ON_DEV moves the input layer) - so this cut carries the new tip 1d32c22609d6eb23d48a26ba14e680e40afd6e17. Replaces the 1991650479db cut (#772), which was gated on the previous run. - third_party/llama.cpp: 1991650479db -> 1d32c22609d6 - hrx-system: unchanged at the held 82804197bcbe (config/hrx-system-hold.config); the hold is not lifted by this PR - registry/architectures.json: regenerated against the new gitlink (tools/registry_build.py): 294 hrx / 7 npu, 0 architectures added or removed, so the only change is the recorded source pin; --check-pins says "records the committed pins" Checked on the new tip before cutting: the ggml-hrx HIP add-on hook (GGML_HRX_HIP_ADDON_DIR) is present, tools/mtmd/models/models.h exists (the file whose absence in the runner's checkout failed the first #772 gate), and test-backend-ops still carries the zero-cases guard the improved gate row now recognises. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--moe-slots autosizes the expert cache from free memory:MemAvailable, less the other tensors the GPU holds and a reserve (8 GiB or 15%), divided by the mean expert size frommoe/GgufIndex. It's clamped to the model's experts, with a floor of 64 per layer.--moe-prefetch Ncontrols the gate-ahead prefetch, now off by default. On Flash-Next it doubled decode time because the drive is already the limit.Verified on strixhalo with
1bit serve -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --moe-slots auto --ctx-size 4096:🤖 Generated with Claude Code