[Bug] spmd_paged_attention_highperf b1_h32_kv8_s128_bs128_fp16 regressed: sim scheduler stall (-100) + onboard golden mismatch

[ChaoZheng109]

Platform

a2a3 (Ascend 910B/C hardware) and a2a3sim (Ascend 910B/C simulation) — fails on both.

Runtime Variant

tensormap_and_ringbuffer

Description

tests/st/a2a3/tensormap_and_ringbuffer/spmd_paged_attention_highperf/test_spmd_paged_attention_highperf.py::TestSpmdPagedAttentionHighPerf::test_run fails on the only non-manual case, b1_h32_kv8_s128_bs128_fp16 (batch=1, num_heads=32, kv_heads=8, head_dim=128, kv_seq=128, block_size=128, fp16).

This is a regression on main: the st-sim-a2a3 and st-onboard-a2a3 jobs both passed at commit 17fa04aa (2026-06-11, run 27324277512) and fail at the current HEAD 19b2c0be. The failure reproduces deterministically on simulation, so it is not hardware flakiness.

The failure presents differently on the two backends:

• a2a3sim: scheduler stall — the workload never completes and the runtime times out.
• a2a3 (onboard): numerical golden mismatch on out, and the resulting hang/fault poisons the device (cascading 507018 into the next test in the same worker).

Steps to Reproduce

# simulation (deterministic)
pytest tests/st/a2a3/tensormap_and_ringbuffer/spmd_paged_attention_highperf/test_spmd_paged_attention_highperf.py \
    --platform a2a3sim --device 0-15 -v

# onboard
python -m pytest tests/st/a2a3/tensormap_and_ringbuffer/spmd_paged_attention_highperf/test_spmd_paged_attention_highperf.py \
    --platform a2a3 --device <range> -v

Observed in CI run 27612538258:

• sim (ubuntu): https://github.com/hw-native-sys/simpler/actions/runs/27612538258/job/81640333256
• onboard: https://github.com/hw-native-sys/simpler/actions/runs/27612538258/job/81640333245

Expected Behavior

test_run passes for b1_h32_kv8_s128_bs128_fp16 on both a2a3sim and a2a3, as it did at 17fa04aa.

Actual Behavior

a2a3sim:

E       RuntimeError: run_prepared failed with code -100
[ERROR] handle_timeout_exit: [scheduler_cold_path.cpp:378] [STALL thread=1 idle_iterations=324] TIMEOUT_EXIT after_idle_iterations=324
[ERROR] aicpu_execute: PTO2 runtime failed with rc=-100
[ERROR] validate_runtime_impl: PTO2 runtime failed: orch_error_code=0 sched_error_code=100 runtime_status=-100

a2a3 (onboard):

E       AssertionError: Golden mismatch on 'out': max_diff=0.39404296875, rtol=0.005, atol=0.02
... (an earlier run showed max_diff=3.859375)
[ERROR] sync_run_streams: aclrtSynchronizeStreamWithTimeout (AICPU) failed: 507018
[ERROR] recover_device_or_mark_unusable: Device unrecoverable after AICore error 507018 ... force-reset the card

Git Commit ID

19b2c0b

Host Platform

Linux (aarch64)

Additional Context

• Regression window: passing at 17fa04aa (2026-06-11) → failing at 19b2c0be. Commits landed in between include Add: per-task ring sizing via CallConfig.runtime_env #1042 (per-task ring sizing), fix(l2_swimlane): unify func_id resolve & gate kernel names behind --enable-dep-gen #1048, feat(dfx): dependency- & MIX-aware scheduler-overhead model #1039, Fix: direction-aware host<->device tensor staging (#1047) #1051 (direction-aware host<->device tensor staging), Update: raise CORE_MAX_TENSOR_ARGS to 32, lower scalars to 16 #1056 (CORE_MAX_TENSOR_ARGS 32 / scalars 16). Not yet bisected — listed only to bound the search.
• Temporarily disabled from CI by marking the b1_h32_kv8_s128_bs128_fp16 case "manual": True (CI runs --manual exclude, so it is skipped on both sim and onboard while still runnable via --manual only). The fix PR must remove that "manual": True and confirm the case passes on a2a3sim and a2a3.
• Related: [Bug] SPMD Paged Attention Highperf Fails For >= 8192 sequences #1022 ([Bug] SPMD Paged Attention Highperf Fails For >= 8192 sequences) — different scenario (long sequences); this report is a small-case regression.

[ChaoZheng109]

PR #1063 resolves the a2a3sim half of this issue. The onboard half stays open.

Root cause (sim half)

The sim failure is a wall-clock scheduler-timeout false-latch, not a real
deadlock. #1035 (f0454712, 2026-06-12 — inside the 17fa04aa→19b2c0be
regression window) replaced the iteration-based idle cap with a wall-clock
SCHEDULER_TIMEOUT_MS, hardcoded to 2 s and shared by the sim and onboard
builds. Onboard wants the short 2 s so the AICPU dumps before STARS reaps the
op; but the sim build has no STARS to race and relied on the 5 s #897
distributed-init / HCCL-skew headroom — so the shared 2 s silently cut sim's
budget.

The idle_iterations=324 in the reported failure is the tell: a genuine
deadlock under the old iteration cap would report millions of idle iterations.
324 means the wall-clock budget fired after a starved scheduler thread
(oversubscribed CI runner, 13+ threads on 2 cores) accumulated only 324 spins
in 2 s of real time — exactly the starvation the 5 s headroom was sized for.

Fix

#1063 sources the budget per-variant from the already-split aicpu spin_hint.h
(PLATFORM_SCHEDULER_TIMEOUT_MS): onboard 2 s, sim 5 s. The
b1_h32_kv8_s128_bs128_fp16 case is re-enabled on a2a3sim only
("manual": True removed, platforms → ["a2a3sim"]).

Verification

• Local: test_run passes on a2a3sim at the restored 5 s budget (was
  TIMEOUT_EXIT rc=-100 at 2 s).
• CI st-sim-a2a3 green on head ddea5c7f, and re-running the sim job twice
  reproduced the pass — deterministic, not flaky:
  • ubuntu-latest: https://github.com/hw-native-sys/simpler/actions/runs/27614935320/job/81650317706
  • macos-latest: https://github.com/hw-native-sys/simpler/actions/runs/27614935320/job/81650318290

Still open: onboard half

The a2a3 onboard failure — golden mismatch on out (max_diff 0.39–3.86)
plus the 507018 device-poison cascade — is a numerical-correctness regression
independent of the scheduler timeout, and this PR does not address it. The
case therefore stays out of the onboard run (platforms = ["a2a3sim"]) pending
that fix. Leaving this issue open for the onboard half.

[ChaoZheng109]

Onboard golden-mismatch half of #1070 — root-cause investigation (handoff to kernel owner)

The a2a3sim TIMEOUT half was fixed in #1063 (per-platform scheduler budget). This is the a2a3 onboard out golden-mismatch half. It is not fixed — it is a latent cross-core race in the highperf decode kernel that I could characterize precisely but not land a fix for. Summary below for whoever owns UnpadAttentionDecoder (spmd_paged_attention_highperf/kernels/kernel/pa_kernel.cce).

Confirmed characterization

• Genuine flaky compute race (~33%) in the decode kernel — not host/runtime, not a flush/visibility-of-out issue.
• When it fails: all 32 output heads are over-tolerance at once, and the magnitude varies run-to-run (max_diff=0.394 @ (0,19,113) and 3.86 @ (0,28,123) both observed). All-heads + varying ⇒ a true cross-core race on a shared resource, not a deterministic stale read.
• For b1 (kv_split_core_num=1) the cores compute disjoint heads, so an all-heads failure implicates a shared FFTS-flag / cross-core-visibility point hit by every cluster's AIC↔AIV pipeline simultaneously.

Trigger vs. defect

• Trigger = Update: raise CORE_MAX_TENSOR_ARGS to 32, lower scalars to 16 #1056 (CORE_MAX_TENSOR_ARGS 16→32; PTO2TaskPayload 2880→4800 B) tightened AICPU→AICore dispatch timing. Bisect: Fix: direction-aware host<->device tensor staging (#1047) #1051 passes, Update: raise CORE_MAX_TENSOR_ARGS to 32, lower scalars to 16 #1056 fails; Update: raise CORE_MAX_TENSOR_ARGS to 32, lower scalars to 16 #1056's own CI reproduced it with Fix: direction-aware host<->device tensor staging (#1047) #1051 NOT in the tree (compare ea4d2045...04281fd7 = diverged), and the PTO-ISA pin (ddafa8da, 2026-05-19) and the kernel are unchanged across the window. ⇒ the defect is latent, only exposed by the timing shift.
• Defect = a cross-core sync gap in the decode AIC↔AIV handoffs (QK_READY_DECODER/SOFTMAX_READY_DECODER/UPDATE_READY_DECODER).

Ruled out — each by a CI experiment on investigate-1070-highperf-acc

1. Host aclrtSynchronizeDevice before the D2H copy-back → identical mismatch. out is computed wrong, not copied early.
2. AICore worker-loop exit pipe_barrier(PIPE_ALL) → identical mismatch. Not a final-flush issue.
3. FFTS flag-id collision at the split-KV reduce → reduce_flag_id was hardcoded 3, aliasing QK_READY_DECODER=3 (two distinct barriers on one 4-bit semaphore; wait_flag_dev(3) was also hardcoded). Moving it to a free flag changed nothing for b1 (the CombineScale reduce is trivial at kv_split_core_num=1). Still a real latent bug for kv_split_core_num>1 — fix kept on the branch (fix(#1070): dedicated FFTS flag…), worth landing independently.
4. DDR fence at the QK→s_gm handoff (dcci(ENTIRE_DATA_CACHE,CACHELINE_OUT)+dsb(DSB_DDR) before FftsCrossCoreSync(QK_READY_DECODER)) → still failed. Caveat: AIC writes s_gm via FIXPIPE, which dcci (data cache) likely does not cover — so this does not cleanly refute a missing-fence at the other handoff: the AIV→AIC SOFTMAX_READY_DECODER path writes p_gm via MTE3, where dcci would apply.

Where I'd point a fix

The kernel has no dcci/dsb anywhere — it relies entirely on FFTS mode2 + pipe ordering for cross-core GM visibility. The open question (unprovable from PTO-ISA source; the ffts_cross_core_sync/wait_flag_dev builtins are toolchain-defined): does FFTS mode2 imply a DDR fence? If not, each producer→consumer handoff must flush the producer's GM tile to DDR with the correct per-pipe primitive before signaling (FIXPIPE-written s_gm vs MTE3-written p_gm need different fences) — matching PTO-ISA's own TSyncCVID/SyncAll pattern (dcci + dsb(DSB_DDR) before the cross-core signal). Start at SOFTMAX_READY_DECODER (MTE3, where dcci is known to apply).

Repro

Branch investigate-1070-highperf-acc: b1 re-enabled on a2a3 + a per-head mismatch diagnostic in _compare_outputs ([acc-diag] … at (batch,head,dim) … rows_over_tol=N/32). Rerun only st-onboard-a2a3 ~3-4× to hit the ~33% flake (whole-suite reruns are noisy — multi_chip_dispatch has its own unrelated onboard flake).

[ChaoZheng109]

cc @MirkoDeVita98 — the onboard golden-mismatch half lives in the UnpadAttentionDecoder kernel you introduced in #899 (pa_kernel.cce), and you're the best person to judge the FFTS cross-core handoff design.

Source trace: the defect is latent since #899 (2026-06-02) and was only exposed by #1056 tightening AICPU→AICore dispatch timing — the kernel and the PTO-ISA pin are unchanged across the regression window. Full characterization + the 4 ruled-out fix attempts are in the comment above.

The two questions only you can answer quickly:

1. Is the reduce_flag_id = 3 (aliasing QK_READY_DECODER = 3) intentional, or a latent flag collision? (Real bug for kv_split_core_num > 1; a fix is on branch investigate-1070-highperf-acc.)
2. Do the decode *_READY_DECODER FFTS handoffs assume mode2 implies a DDR fence for the producer's GM tile (s_gm via FIXPIPE, p_gm via MTE3)? If not, that's the missing cross-core visibility fence behind the all-heads-wrong / varying-magnitude race.