Repository navigation
verify: initialize remote expert helper state in both batch paths - #1676
Closed
saikiran-rs wants to merge 1 commit into
Closed
saikiran-rs wants to merge 1 commit into
saikiran-rs wants to merge 1 commit into
Conversation
saikiran-rs
marked this pull request as ready for review
October 9, 2026 07:47
Owner
|
Ported with your authorship. The batch graphs share the solo graph's weight/mask publishing, so begin/end around both pool calls matches. On our 2xA4000 box with --batch 3 --remote-expert-opt we get 6/6 correct replies before and after (we could not reproduce the garbage output there), so this stays a review-based fix; if you can, send the 1/3 -> 18/18 control on a current build. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When concurrent requests use
--remote-expert-opt, batch verification can returnrepeated characters or unrelated output. The batch graphs consume the helper's
mask and weighted sum, but
run_slot_rowsandbatch_polldo not initialize thehelper with the current layer's routing weights or finish its work around the
CPU/helper pool call.
What changed
Add guarded
remote_opt_->begin(h_w_, 0, S)andremote_opt_->end()calls aroundboth batch pool calls, matching the existing solo verifier lifecycle. This is six
added lines in
src/core/verify.cpp. The helper-disabled path executes no newcalls. Model weights, sampler, KV layout and GPU kernels are unchanged.
Validation
Historical Linux HIP validation on v0.1.41 (
fb58e0d) with RX 7900 XTX + RX 6800 XT,Xeon E5-2696 v4, 128 GB DDR4 and Qwen3.8 Flash-Next UD-Q4_K_XL:
Disabling QFUSE also passed 1/3; disabling the helper passed 12/12.
correct identity, arithmetic, meaningful-language and cross-session checks,
with three native slots busy in every round. Another 12 configured-sampling
replies passed.
WinoGrande, each out of 200, with zero invalid outputs. These scores compare
modes of the patched runtime, not a before/after quality gain.
cross-session leakage, including the generated output and API reservation.
These are correctness results from the recorded 2026-10-08 stack, which also has
separate Python API compatibility changes. They are not a new isolated speed
benchmark or a claim of bitwise/universal model-quality equivalence.
This PR is a draft: fresh PR-branch CUDA/HIP build checks and CUDA hardware testing
remain pending. No new model/GPU runs were performed for publication because the
machine is running a separate experiment. The original serving setup is unchanged.
Extra notes
Related #1209/#1524 change batch layout/scheduling and #1636/#1637 change batch
MTP/adaptation. This change only supplies the missing helper lifecycle at the
two existing pool calls. It does not include the helper allowance setting from
#1458 or the performance report from #1459.