Skip to content

perf(gdn): optimize post-conv and causal-conv kernels - #155

Merged
localai-bot merged 6 commits into
localai-org:mainfrom
richiejp:row/KERNEL-SSM-MAMBA-POSTCONV-TILE
Aug 11, 2026
Merged

localai-bot merged 6 commits into
localai-org:mainfrom
richiejp:row/KERNEL-SSM-MAMBA-POSTCONV-TILE

Conversation

@richiejp

@richiejp richiejp commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Row

KERNEL-SSM-MAMBA — Qwen GDN post-conv and causal-conv kernel sublevers; one row per PR.

Before starting

  • Issue/PR search and existing claim: this is a stacked follow-on to perf(gdn): dispatch exact causal-conv chunks #127, which supplies the exact causal-conv chunk descriptors measured here. No separate matching open PR was found for these two opt-in kernel specializations.
  • Roadmap or matrix row, plus scripts/ready-for-helper.py result when applicable: .agents/kernel-matrix.md:157 (KERNEL-SSM-MAMBA, INVENTORIED), feeding ROAD-V1-C2-LOCAL-BF16. Spikes: .agents/specs/sm120-qwen35-postconv-token-tile-2026-08-08.md and .agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md.
  • Exact current-code and test/evidence anchors inspected: src/vt/cuda/cuda_gdn.cu, src/vt/cuda/gdn_prefill_conv.h, tests/vt/test_gdn_prefill_conv.cpp, tests/vt/test_ops_gdn.cpp, and pinned vLLM/FLA causal-conv plus fused post-conv kernels recorded with file:line anchors in the two spikes.

What changed

Add two independently selectable, default-off CUDA specializations. VT_GDN_POSTCONV_TOKEN_TILE=1 ports the upstream-shaped 16-token/four-warp post-conv schedule while preserving the existing 128-lane reduction tree byte-for-byte. VT_CONV_CHANNEL_TILE=1 specializes the causal-conv path for width four at the existing one-channel-per-thread grid; arm =2 is retained as the measured 256-channel/two-channels-per-thread falsification arm. Production dispatch and the portable mutation gate share one callback dispatcher, so the tests prove that each selected arm is actually invoked.

Evidence

  • scripts/agent-preflight.sh passes after rebasing onto canonical upstream/main c05cee1d2; all record gates, mutation suites, live-row audit, and committed-range document checks are green.
  • Tests covering this change: contained CUDA rebuild including an explicit vllm-bench relink; portable dispatch 9/9 cases, 88/88 assertions; CUDA GDN 67/67 cases, 4631/4631 assertions; Qwen3.5 paged-forward 4/4 cases, 8/8 assertions. The accepted production arms produce identical token hashes.
  • Same-change doc obligations: docs/STATUS.md and docs/BENCHMARKS.md updated; the model/feature surface did not expand. Full profiler evidence and the void stale-link series are retained in the two specs and .agents/benchmark-record.md.

Speed claims

  • The operator ran the numbers under ${GPU_LOCK} and they are recorded in docs/BENCHMARKS.md with the repro recipe.

On the local RTX 5070 Ti / Qwen3.5-4B c32 workload, the post-conv tile reduces graph-node time 227.887 to 122.587 ms (1.859x) and improves every observed enclosing axis, including total/output throughput by 0.532%. The causal-conv K4 arm reduces graph-node time 234.605 to 219.506 ms (6.44%) and improves whole-run total/output throughput by about 0.12%. The 256-channel arm is 4.05% slower than the K4 arm and is recorded as falsified.

Honest gaps

  • This PR is stacked on perf(gdn): dispatch exact causal-conv chunks #127. Until perf(gdn): dispatch exact causal-conv chunks #127 merges, GitHub shows its exact-chunks commits in this PR too; after that merge, this PR reduces to the opt-in follow-on commits.
  • Both specializations remain opt-in pending repeated A/B and hardware-unavailable 27B/35B gates. The 4B result is not extrapolated to the release models.
  • Same-tool residuals remain: post-conv is 1.135x and causal conv K4 is 1.509x slower than pinned vLLM.
  • The first production causal-conv profile series was void because vllm-bench had not relinked the rebuilt CUDA library. The accepted series explicitly relinked the benchmark and trace-proved the specialized kernels.
  • Current main c05cee1d2 independently fails check-device-leakage.py: src/vllm/v1/worker/gpu/runner.cpp contains one new hard-coded DeviceType::kCUDA (DSR kcuda 1 > baseline 0). perf(gdn): dispatch exact causal-conv chunks #127 and this PR inherit that merged-tree failure; neither kernel diff introduced it.

mudler added a commit that referenced this pull request Aug 8, 2026
…re-push sandbox (#159)

Two guards on main were RED and between them blocked every open external
contributor PR (#127, #154, #155) and every push. Both premises were verified in
the tree before changing anything.

1) check-device-leakage: src/vllm/v1/worker/gpu/runner.cpp named
   vt::DeviceType::kCUDA in the device-agnostic shared layer (DSR bucket 'kcuda'
   1 > baseline 0). It came in with the QueueSupportsAsyncInputCombine rescope
   during the PR #140 fix round - ours, not the contributors'; richiejp reported
   it in #127's honest gaps. Fixed the way the guard's own message prescribes,
   mirroring the SupportsAuxStream precedent: ask the backend, not the device.
   New vt::Backend::SupportsAsyncSampledTokenReadback() (base false) answers
   whether the host may validly read the sampled token id back between steps;
   CPU overrides true (host and device memory are one allocation) and CUDA
   overrides true (the id is device-mirrored). The runner asks
   vt::TryGetBackend(queue.device.type), whose nullptr for a device absent from
   the build also subsumes the old #ifdef VLLM_CPP_CUDA guard.
   SEMANTICS UNCHANGED: CPU async-ON, CUDA async-ON, discrete non-CUDA (ROCm
   gfx1201) async-OFF - the "!"-token hazard stays closed.

2) .githooks/pre-push ran check-policy.py inside a PARTIAL export
   (README.md docs scripts .agents), but policy_contract.py:428 asserts AGENTS.md
   is a non-symlink regular file and resolves its Markdown links against that
   sandbox. AGENTS.md and its .env.example link were both missing, so the hook
   failed closed on content that is fine in the real tree - every push refused.
   EXPORT_PATHS is now a superset of what the CHECKERS read.

Gates: check-device-leakage RED->GREEN (kcuda=0, DSR 32 == baseline 32); all four
hook checkers OK in the reproduced sandbox; test_async_llm 8/8-347,
test_engine_core 6/6-44, test_llm_engine 11/11-204 (CPU still resolves async-ON);
clean -Werror CPU build; full 11-gate record battery green.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude (Opus 5) via Claude Code
@richiejp
richiejp force-pushed the row/KERNEL-SSM-MAMBA-POSTCONV-TILE branch 3 times, most recently from b3f019a to a651838 Compare August 10, 2026 13:56
Select the measured sm_120 post-conv residual, reject the existing split grid, and bind the upstream 16-token CUDA experiment and gates.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Port the upstream 16-token per-head schedule behind an opt-in flag, preserve the existing reduction order byte-exactly, and record the sm_120 A/B evidence.

Row: row/KERNEL-SSM-MAMBA-POSTCONV-TILE

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Route runtime-width and both K=4 CUDA launch arms through the same portable, mutation-tested callback dispatcher. Pin strict arm-2 parsing and resolved non-K4 fallback semantics.

row/KERNEL-SSM-MAMBA-POSTCONV-TILE

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
@richiejp
richiejp force-pushed the row/KERNEL-SSM-MAMBA-POSTCONV-TILE branch from a651838 to 7c8ee10 Compare August 10, 2026 14:12
@localai-bot
localai-bot merged commit 19e3554 into localai-org:main Aug 11, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant