Skip to content

[Engine] sycl: defer pipelined one-token commits to the host verdict (port of c81f9984) - #2436

Open
deriknel wants to merge 1 commit into
Niko1221:mainfrom
deriknel:pr/sycl-pipeline-one-token-commit
Open

deriknel wants to merge 1 commit into
Niko1221:mainfrom
deriknel:pr/sycl-pipeline-one-token-commit

Conversation

@deriknel

@deriknel deriknel commented Oct 14, 2026 •

Copy link
Copy Markdown

Problem and result

With --pipeline-windows 2 on SYCL, greedy text forks from the serial loop's within the first few hundred characters. A pipelined verifier (set_always_publish) captures its one-token window with the in-graph state advance, and pl_commit_async then runs the commit graph on top of it, so every one-token window advances the first stage's recurrent (GDN) state twice. c81f998 fixed this in the CUDA verifier on 2026-10-09; the SYCL verifier still has the old condition. With this change the pipelined text matches the serial loop's.

Related discussion

Needs #2433, without which --pipeline-windows does not run on SYCL at all. Ports c81f998. No scope agreement needed: a focused fix.

Approach

sycl/src/core/verify.cpp, the two conditions c81f998 changed in src/core/verify.cpp:

  • Verifier::record_window: self_commit is off when always_publish_ is set.
  • Verifier::commit: the branch that skips the commit graph for a one-token window is not taken when always_publish_ is set.

Only --pipeline-windows calls set_always_publish(true), so the serial loop does not change. The branch still merges cleanly into main.

Validation

Setup for every run: main e88ee39 built for SYCL (AOT bmg-g31, oneAPI 2026.1, Compute Runtime 26.35.39758.10, IGC 2.41.5) in a container; host CachyOS, kernel 7.2.11, xe. 2x Intel Arc Pro B70 32 GB, layer split 26, Swift 1.5 Flash-Next IQ3_S with every expert in VRAM, --max-context 262144, int8 KV, --spec 4 --spec-min-p 0.6, --prefill 4096. Each build has only this PR's commit on top of main. Greedy check: 5 fixed prompts (prose, code, maths, an edit, a story), temperature 0, 400 tokens, thinking off, compared character by character. The serial reference is main with pipeline windows off.

Passed:

  • main + [Engine] sycl: a pipelined window still on the GPU is polled again, not failed #2433 + this PR, --pipeline-windows 2: all 5 greedy texts identical to the serial loop's, in two rounds.
  • Same build: decode 88.8 / 126.4 tok/s (prose / code) pipelined, against 83.0 / 106.6 serial on main (+7% / +19%).
  • Same build: a ~148K-token prompt (answer found) and 10 prompts of ~13K tokens (all found), no verify: error, no lost device, no xe reset.
  • ctest on both cards: the same 3 failures as on main; iq_parity with fixtures, 0 failures.

Failed:

Not run:

  • CUDA, HIP and Metal builds: the change is in sycl/ only.

Limits

None known. On 0.1.43.1 we also confirmed the cause with STRATA_ONE_TOKEN_COMMIT=0 (identical text and state hashes), not repeated on 0.2.0.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.

Port of c81f998 (CUDA) to the SYCL verifier. A pipelined verifier
(set_always_publish) captured its one-token window with the in-graph state
advance, and pl_commit_async then ran the commit graph on top of it, so every
one-token window advanced the first stage's GDN state twice. Greedy text
with --pipeline-windows 2 forked from the serial loop's within the first few
dozen tokens on 2x Arc Pro B70; with this change both loops give identical
text and state fingerprints (STRATA_STATE_HASH), as STRATA_ONE_TOKEN_COMMIT=0
already did.

Co-Authored-By: Claude Opus 5.5
@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
@deriknel deriknel changed the title SYCL: defer pipelined one-token commits to the host verdict (port of c81f9984) [Engine] sycl: defer pipelined one-token commits to the host verdict (port of c81f9984) Oct 20, 2026
@deriknel

Copy link
Copy Markdown
Author

/reopen

@github-actions github-actions Bot reopened this Oct 20, 2026
@github-actions

Copy link
Copy Markdown

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

sheigl added a commit to sheigl/Strata that referenced this pull request Oct 23, 2026
…(cherry-pick of upstream PRs Niko1221#2433 + Niko1221#2436)

Verifier::done() still carried dpct's placeholder for CUDA's error check and treated a window that is still
running as an error: --pipeline-windows 2 on a SYCL layer split failed the first request with 'verify: error'
(HTTP 400). With the upstream fix a window still on the GPU is polled again, as the CUDA verifier does.

Measured on this branch (B65 + B70, IQ3_S split, --spec 4, streamed experts, 2.84 GiB mirror, 3 sequential
2,280-token requests per arm, 200 tokens out, temperature 0): serial 52.5 / 60.0 / 61.0 tok/s decode against
--pipeline-windows 2's 58.6 / 65.4 / 65.7 tok/s (+8-12%), zero 'verify: error' lines, identical prompts and
completions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants