Fix: keep the host cancel ordered after the handshake clear

[Leaf-Salix]

Based on feat/kernel-mode-integration-test at 0d55fac7 — the tip at the time of writing, so no rebase is needed. Final diff is 7 files, +97/-11, and contains no fault injector.

Four commits: the fix, its regression coverage, and an add/revert pair that makes the hazard reproducible. The pair is kept in the history on purpose — it is how a reviewer can watch the scenario fail and then stop failing. The tree at 84c44df5 is identical to the tree at a35ad53f (git diff --stat a35ad53f 84c44df5 is empty), so nothing from the reproducer ships.

Motivation

This closes N2-new-2 from the unified issue list (HIGH, introduced this round):

补偿阶梯把 cancel 写在 caller stream、把握手清零移到 AICPU stream，两个 memset 跨流无序，取消信号可能被清零覆盖导致 AICore 永久自旋

The compensating cancel and the handshake clear write the same four bytes:

• TmrLaunchControl::host_cancel sits at offset 0 — asserted by src/common/task_interface/tmr_kernel_control.h:72.
• build_tmr_kernel_clear_plan sets regions = {control, reports} and cancel = {control.address + offsetof(TmrLaunchControl, host_cancel), 4} at src/common/tensormap_and_ringbuffer/kernel_clear_plan.h:63-64. The cancel word is inside the region the clear zeroes.

Two writers, one word — and with the clear on the AICPU stream and the cancel on the caller stream, nothing ordered them against each other. When the clear lands last, host_cancel reads zero and AICore never leaves its pre-window poll (src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp:333; A5 at src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp:313). The caller's wait(AicpuDone) then never fires, and the round is released only when STARS reaps the op at the op-execute timeout — 45 s, PLATFORM_OP_EXECUTE_TIMEOUT_US at src/a2a3/platform/include/common/platform_config.h:77.

The two writes, and the stream each rides

Both writes are aclrtMemsetAsync calls, and the handle passed in is the stream they land on — kernel_launch_owner.cpp:103-116 forwards that argument straight to the driver. Three handles are in play:

Handle	What it is
h.caller	The stream the caller handed to the launch — outside the runtime, owned by the caller
h.aicpu	The context's own hidden AICPU stream (KernelStreamKind::Aicpu) — inside the runtime
h.aicore	The context's own hidden AICore stream; same kind of thing. Not involved in this change

	clear (writes 0)	cancel (writes 0xff)	Ordered?
Before 0d55fac7	h.aicpu (:51)	h.caller (:40)	no — two streams, and no edge between them
After 84c44df5	h.aicpu (:55)	h.aicpu (:44)	yes — one stream, ordered by FIFO

The clear did not move. It has always been on the AICPU stream and this PR leaves it there; the only thing that moved is the cancel, from the caller's stream onto the AICPU stream. That direction is deliberate: the clear must complete before the AICPU task reads the control block, and the AICPU task is enqueued on that same stream, so the clear is already where it has to be. Moving the cancel in costs nothing else; moving the clear out would have needed a new cross-stream edge to keep the same guarantee.

So the whole change in one line each way:

before:  clear on the AICPU stream, cancel on the caller stream — nothing orders them
after:   both on the AICPU stream — the driver's FIFO makes the clear strictly precede the cancel

Where the split came from

Commit	clear stream	cancel stream	Ordered by FIFO?
5b2a5a5c Add: integrate kernel-mode PRs with an end-to-end launch test (#2216)	h.caller	h.caller	yes
29a1cd40 Refresh the kernel-mode integration line ... (#2241)	h.aicpu	h.caller	no
52d6f468 (this PR)	h.aicpu	h.aicpu	yes

#2241 moved the clear (and AicoreStart) to the AICPU stream and left the cancel behind, splitting two writers of one word across two streams. Before it, both rode the caller stream and FIFO ordered them, so the hazard is a regression from that refactor rather than an original design choice. This PR restores the single-stream property, on the AICPU branch where the clear now lives.

The fix

One argument changes, at src/common/platform/onboard/host/kernel_launch_sequence.h:44. The clear line shown below is unchanged context — it is quoted so the reader can see that only the cancel moved:

     auto compensate = [&](bool retry_core_done) {
         auto cleanup = [&](int rc) { ... };
+        // Cancellation and the handshake clear write the same control word, so
+        // they must share a stream: FIFO is what orders the cancel after the
+        // clear. Across two streams the clear can land last, reset the cancel,
+        // and leave AICore spinning before its register window opens.
-        if (!cleanup(ops.cancel_waiting_aicore(ops.context, h.caller))) return;   // :40, caller stream
+        if (!cleanup(ops.cancel_waiting_aicore(ops.context, h.aicpu))) return;    // :44, AICPU stream
         ...
     };
 
     // unchanged, earlier in the same function:
     if (!step(KernelLaunchStep::Clear, ops.memset_handshake(ops.context, h.aicpu))) return result;   // :51 -> :55

The handle argument is the memset's target stream, with no layer in between:

// src/common/platform/onboard/host/kernel_launch_owner.cpp:112-116
ops.cancel_waiting_aicore = [](void *context, void *stream) {
    const auto &cancel = static_cast<Submission *>(context)->clear.cancel;
    return aclrtMemsetAsync(cancel.address, cancel.bytes, 0xff, cancel.bytes, stream);
};

Why this shape of fix rather than another:

• FIFO is the only ordering mechanism available inside a launch. #2249 established that a launch must not synchronize internally, so a host-side sync on the AICPU stream is not an option.
• An event-based fix would add a caller-stream edge. The sequence deliberately keeps the caller free of any edge to AICore so that capture propagates caller → AICPU → AICore; see the comment at kernel_launch_sequence.h:57-61. Recording an event after the clear and waiting it on the caller would break that.
• It turns "cancel only before AICPU submission" from positional into structural. A cancel queued on the AICPU stream cannot overtake an already-submitted AICPU task, so it can never land on a live handshake. Both compensation paths today run before any AICPU submission (kernel_launch_sequence.h:65-72), so this changes nothing today but survives a future refactor.
• It protects the second reader of that word. The AICPU's collect_reports treats a non-zero host_cancel as an abort signal (src/common/tensormap_and_ringbuffer/kernel_core_group.h:68,78). The issue text does not mention this reader, but the same ordering is what keeps it correct.
• Rejected: moving host_cancel out of the cleared region. It still has to be reset between rounds, and that reset is itself an ordering requirement, so the change would relocate the problem rather than remove it.

Two files are changed in the same commit because both pinned the old behaviour:

• tests/ut/cpp/common/test_kernel_launch_native.cpp:80-82 — the expectation and its comment named the caller stream for the cancel.
• docs/kernel-launch-binder.md:87 — the published sentence justified a caller-stream cancel with "the caller's stream … carries nothing but Start at that point", which is precisely the premise that produced the race.

Tests

Coverage added or extended

• tests/ut/cpp/common/kernel_binder_test_support.h + test_kernel_launch_binder.cpp:162-163 — the fake records the stream of every memset and the enqueue-failure test requires all of them to ride handles.aicpu. This is the deterministic guard: it fails on a caller-stream cancel at both compensation sites (fail == 7, the AicoreDone-retry path; fail == 8, the AICPU-launch path) and it carries the no_hardware label, so it runs with no device and in any CI. It closes a real gap: the only test that asserted the memset stream was the native UT, which is gated behind SIMPLER_ENABLE_HARDWARE_TESTS and therefore absent from a default build.
• tests/st/a2a3/tensormap_and_ringbuffer/kernel_mode_capture/test_kernel_mode_capture.py:354-362 — the launch_fail_compensation scenario now also asserts the ladder's own progress: three memsets (two clear regions plus the cancel), five event records, four event waits. Previously "the caller's sync returned" was the only assertion, and a ladder that bailed out at its first step would still sync clean, because the caller stream only has to carry Start in order to drain.

Executed results

Validation	Tree	Reproducer armed	Result and scope
Local UT test_kernel_launch_binder, macOS, no device	unfixed (h.caller restored)	n/a	Failed — the stream assertion fires under SCOPED_TRACE 8, i.e. the AICPU-launch compensation site. 0.3 s
Local UT test_kernel_launch_binder, macOS, no device	fixed	n/a	Passed. 0.2 s
a2a3 ST whole file, all 30 scenarios, 1 run	fixed	no	30 / 30 passed, 373 s wall clock
a2a3 ST launch_fail_compensation, 10 runs	fixed	no	10 / 10 passed, ~11 s per run
a2a3 ST launch_fail_compensation, 10 runs	unfixed	no	10 / 10 passed — see below: the scenario on its own does not catch this hazard
a2a3 ST launch_fail_compensation, 5 runs	fixed	yes	5 / 5 passed, ~12 s per run — the switch is inert on same-stream code
a2a3 ST launch_fail_compensation, 5 runs	unfixed	yes	5 / 5 failed, ~19.5 s per run — caller sync times out at 10 s
a2a3 ST launch_fail_compensation, 3 runs, committed tree 84c44df5	fixed	no	3 / 3 passed, ~11 s per run

Environment: Ascend a2a3 onboard runtime, CANN 9.0.0 (/usr/local/Ascend/cann-9.0.0), exclusive single-device task-submit jobs, checkout-local build environment. For the unfixed rows the same tree was rebuilt with h.caller restored (runtime .so sha256 12b35607…, versus b686bb14… fixed); the unfixed build reproduced the same digest across two independent builds.

In the whole-file run the environment's resource scheduler dispatched the 30 cases as 30 separate child jobs, each reporting its own [PASS …]; the outer pytest session therefore ends with no tests ran, which is expected for that dispatch and not a deselection. Per-scenario results were collected from the child job markers: 30 dispatched, 30 passed, 0 failed.

The unfixed failure is the intended mechanism, taken from the child process log:

File ".../test_kernel_mode_capture.py", line 347, in _run_launch_failure
    assert context.lib.aclrtSynchronizeStreamWithTimeout(context.caller, 10000) == 0
AssertionError

The caller's bounded sync expires because the ladder never completes: AICore is still spinning, so AicpuDone never reaches the caller.

The reproducer pair: 34202626 (armed, measured) → 84c44df5 (reverted)

The cross-stream interleaving is latent, not merely rare. Unarmed, the unfixed build passes the scenario 10/10, because by the time the host enqueues the cancel the clear's stream has usually drained. A test that cannot fail on the defect cannot demonstrate it — so commit 34202626 adds a switch that makes the interleaving deterministic, measures the defect with it, and commit 84c44df5 removes the switch again so that nothing of it ships.

34202626 — Add: make the cross-stream cancel/clear race reproducible in the capture ST

The switch lives in the capture observer and is armed by CAPTURE_OBSERVER_REORDER_CANCEL_CLEAR:

1. remember the stream of the handshake clear (the zero fill);
2. when the cancel (the all-ones fill) arrives on a different stream, put a zero fill back on top of it, on the cancel's own stream so FIFO places it after.

It therefore reproduces the hazard — "two writers of one word sit on different streams, and the clear lands last" — rather than the outcome of one particular build. Same-stream code is never touched, which is exactly why the fixed build passes with it armed while the unfixed build fails. The emulated fill goes through the resolved symbol rather than through the interception, so async_clears still counts only what the sequence itself issued, and the scenario itself is unchanged by the switch (the launch_fail_compensation assertions are identical in both trees).

What 34202626 established on a2a3 — two 5-run batches, scenario unchanged:

Against the unfixed tree (h.caller restored and the runtime rebuilt):

Runs	Result	Duration
1–5	5 failed / 0 passed	~19.5 s per run

and the failure is the mechanism the issue describes, taken verbatim from the child process log:

Traceback (most recent call last):
  File ".../test_kernel_mode_capture.py", line 347, in _run_launch_failure
    assert context.lib.aclrtSynchronizeStreamWithTimeout(context.caller, 10000) == 0
AssertionError

That is N2-new-2 reproduced on demand instead of hoped for. The emulated late clear resets host_cancel to zero, so AICore never leaves its pre-window poll; the compensating ladder's wait(AicoreDone) therefore never fires, AicpuDone is never recorded, and the caller — whose only path to a tail is AicpuDone — cannot drain. Its bounded 10 s sync expires, and without the switch the same round would have been released only by STARS at the 45 s op-execute timeout.

Against the fixed tree the switch is inert — cancel and clear share the AICPU stream, so its second condition never fires:

Runs	Result	Duration
1–5	5 passed / 0 failed	~12 s per run

Read together: the defect is real (34202626 + unfixed, 5/5 fail, at the failure point the issue predicts) and the fix removes it (34202626 + fixed, 5/5 pass). Without the fix, the same three numbers are 0/5 and 5/5 the other way.

84c44df5 — Revert: drop the cross-stream cancel/clear reproducer from the capture ST

Removes the switch so that the merged content carries no injector; its message records the same measurements. It is reverted rather than kept because it is a fault injector that nobody arms and that only reproduces an interleaving the fixed code cannot produce — shipping it would put a switch into CI that is red by construction. The property actually worth guarding, both memsets on one stream, is pinned deterministically by the unit test above, which needs no device. Keeping the two commits in the history is the point; shipping the mechanism is not.

The tree at 84c44df5 is byte-identical to the tree at a35ad53f (git diff --stat a35ad53f 84c44df5 is empty).

Reproduction

# Deterministic guard, no device required:
cmake --build build/ut --target test_kernel_launch_binder
ctest --test-dir build/ut -R test_kernel_launch_binder --output-on-failure

# Real hardware, single card, no reproducer:
python -m pytest tests/st/a2a3/tensormap_and_ringbuffer/kernel_mode_capture/test_kernel_mode_capture.py \
  -q --forked --platform=a2a3 --device <id> -k launch_fail_compensation

# Same run with the reproducer armed (on commit 34202626 or by re-applying it):
CAPTURE_OBSERVER_REORDER_CANCEL_CLEAR=1 python -m pytest \
  tests/st/a2a3/tensormap_and_ringbuffer/kernel_mode_capture/test_kernel_mode_capture.py \
  -q --forked --platform=a2a3 --device <id> -k launch_fail_compensation

Not validated:

• The whole ST file passes once (30/30 above), but only launch_fail_compensation was repeated (10 + 5 + 5 + 3 runs) and only it was used for the unfixed/reproducer comparisons. The other 29 scenarios are single-run evidence, not stability evidence.
• A5 hardware was not exercised. The host sequence is shared — src/a5/platform/onboard/host/CMakeLists.txt:93 compiles kernel_launch_native.cpp as well — so the change applies to A5, but only A2/A3 silicon was used here.
• A residual risk this PR does not close. In the compensate(false) path the cancel is now enqueued on the same stream as the launch_aicpu that just failed. If a real CANN enqueue failure leaves that stream in an error state, a later async operation on it might not execute, and the cancel would be lost. The scenario's fault injection returns before calling the real API (kernel_capture_observer.cpp, fail_prepare_step), so it cannot rule this out; verifying it needs an injection that makes the real call fail, which this PR deliberately does not add.
• The reproducer demonstrates that the hazard is real and that the fix removes it. It does not make the shipped scenario a regression guard — unarmed, the unfixed build passes it 10/10.

Commits

1. 52d6f468 — Fix: keep the host cancel ordered after the handshake clear
2. a35ad53f — Add: cover the compensation ladder and pin its memsets to the AICPU stream
3. 34202626 — Add: make the cross-stream cancel/clear race reproducible in the capture ST (diagnostic; reverted by the next commit)
4. 84c44df5 — Revert: drop the cross-stream cancel/clear reproducer from the capture ST

Based on 0d55fac7, the current tip of feat/kernel-mode-integration-test.

中文总结

• 动机： 关闭统一问题清单里的 N2-new-2（HIGH，本轮新增）。补偿阶梯的 cancel 与握手清零写同一 4 字节（host_cancel 位于 TmrLaunchControl offset 0；该字落在清零区间内），而 #2241 把握手清零挪到 AICPU 流却把 cancel 留在 caller 流，两个写方跨流无序 —— clear 后落地就把 cancel 抹成 0，AICore 卡在 pre-window 轮询直到 op-execute 超时（45 s）。
• 核心改动（两条流 → 一条流）： 两个写方是"清零"（写 0）与"取消"（写 0xff）。h.caller = 调用方传进来的那条流（域外）；h.aicpu = context 自有的隐藏 AICPU 流（域内）；h.aicore 同类、本次不涉及。
  • 修复前 0d55fac7：清零在 h.aicpu（:51）、取消在 h.caller（:40）—— 两条流之间没有任何边，先后无保证；
  • 修复后 84c44df5：清零仍在 h.aicpu（:55）、取消归到 h.aicpu（:44）—— 同一条流，驱动按 FIFO 执行，清零严格先于取消。
  • 清零没有挪动，动的只有取消。方向是"把 cancel 移进来"而不是"把 clear 移出去"：清零必须挡在 AICPU 任务读控制块之前，而该任务本身就排在这条流上，留在原地最自然；把清零挪出去反而要新增一条跨流边才能保住同样的保证。
• 修复： 只改一行（kernel_launch_sequence.h:44），把补偿期 cancel 也放到 AICPU 流，靠同流 FIFO 定序。同一提交内对齐了 UT 断言与那份已发布文档（文档原来用"caller 流此刻只有 Start"来论证旧行为，而这正是竞态的前提）。选它的理由：launch 内部不允许同步（#2249），事件方案会在 caller 流上加边、破坏 capture 的 caller→AICPU→AICore 传播；同流后 cancel 无法越过已提交的 AICPU 任务，把"仅在 AICPU 提交前取消"从位置约定升级为结构约束；顺带保护了 AICPU 侧 collect_reports 这个第二读者。
• 确定性守卫： UT 里 fake 记录每次 memset 的流，并要求全部落在 handles.aicpu；旧代码在两个补偿点都必红，且该 target 带 no_hardware 标签 —— 补上了默认构建里（ASCEND 门控的 native UT 之外）没有任何测试锁这条流的缺口。
• 真机证据（修复版；launch_fail_compensation 场景本身未改动）：
  • 该 ST 文件全量 30 条一次跑通：30/30 通过（373 秒），涵盖 capture/replay、多 callable、跨流、close 失败与两类错误路径等全部场景。
  • 第 3 个提交 34202626（加复现开关）验证了旧版问题确实存在：把那一行回退成 h.caller 并重建 runtime 后，5/5 失败（每次约 19.5 s），失败点经子进程日志核实为
    assert context.lib.aclrtSynchronizeStreamWithTimeout(context.caller, 10000) == 0 → AssertionError。
    失败语义：被人为后置的清零把 host_cancel 抹回 0 → AICore 停在 pre-window 轮询 → AicoreDone 不触发 → AicpuDone 不记录 → caller（唯一通往 tail 的路是 AicpuDone）拿不到 tail → 10 秒有界同步超时。这正是 issue 描述的"AICore 永久自旋"，从"偶发"变成"按需复现"。
  • 同一提交、同一开关，在修复版上 5/5 通过（每次约 12 s）：cancel 与 clear 已同流，开关的触发条件不成立 —— 说明修复恰好消除了这个交错。
  • 不武装开关时：修复版 10/10 通过、旧版也 10/10 通过 —— 这条 ST 单独抓不住该竞态，所以确定性守卫必须放在 UT。
• 为不污染代码，第 4 个提交 84c44df5 回退掉这个复现开关：git diff --stat a35ad53f 84c44df5 为空，即最终合并内容里不含任何注入器。删而不是留的理由：它靠"人为把字写回 0"制造交错，等于强行让修复失效 —— 演示的是危险交错存在，不是产品会进入它；留着等于往 CI 里放一个必红且无人武装的故障注入器。提交 3/4 成对保留在历史里，正是为了让 reviewer 能自己复现、也能看到它被移除。
• 未验证： 全量 30 条只跑了一遍，其中只有 launch_fail_compensation 做了重复与旧版对照（其余 29 条是单次证据、不是稳定性证据）；A5 真机未跑（host 序列两架构共用）；compensate(false) 的 cancel 现在与刚失败的 AICPU launch 同流，若真实 enqueue 失败污染该流，cancel 可能丢失 —— 现有注入在调用真实 API 之前就返回，无法排除，需要"让真实 API 失败"的注入才能验证，本 PR 有意不加。

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 2478cf0b-aa00-417c-a0b6-bb8007484d6e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}