[Bug] hbg sync_start early-dispatch scene segfaults in the native wait under parallel sim sweeps

[ChaoWao]

Platform

a2a3sim (Ascend 910B/C simulation)

Runtime Variant

host_build_graph

Description

TestSpmdSyncStartEarlyDispatchHbg::test_run dies with Fatal Python error: Segmentation fault at a low rate when its directory is swept in parallel. The deepest Python frame is the native wait:

python/simpler/worker.py:11404 in _wait_run_handle
python/simpler/worker.py:4227  in wait
python/simpler/worker.py:11046 in run
simpler_setup/scene_test.py:1802 in _run_and_validate_l2

worker.py:11404 is return chip_run.wait(-1.0 if timeout is None else max(0.0, timeout)), so the fault is inside the native wait beneath the binding rather than in Python.

This is newly landed code, not an old flake. #2095 (0a1a24121, merged 2026-09-03T08:00Z) added this case — 116 lines of new test plus a new orchestration source — and wired hbg's dormant early-dispatch machinery (staging queues, doorbell table, gated cores, sync_start rendezvous, the ED publish list). The first crash I saw was hours later, on the first commit after it that I happened to sweep.

Why it is worth an issue rather than a re-run: the rate reads as a flake, the case is in the ordinary sim sweep so it will redden unrelated PRs, and a segfault leaves no diagnosis behind. It reproduces on unmodified main.

Found while fixing an unrelated scene-test problem (#2118); that change is not involved — the fixture it adds is inert unless a test requests it, this case does not, and the crash reproduces without it.

Steps to Reproduce

The case alone does not reproduce it. Parallelism and/or host load is part of the trigger, so the directory sweep is the shape that matters:

# reproduces: 2 in 24 runs
python -m pytest tests/st/a2a3/host_build_graph -n 4 \
    --platform a2a3sim --device 0-15 -q -p no:randomly

# does NOT reproduce: 0 in 20 runs
python -m pytest tests/st/a2a3/host_build_graph/spmd_sync_start_early_dispatch \
    --platform a2a3sim --device 0-15 -q -p no:randomly

shape	commit	runs	segfaults
host_build_graph dir, -n 4	0edd65ceb	6	1
host_build_graph dir, -n 4	ecf18a0a2	18	1
the case alone	0edd65ceb	20	0
tensormap_and_ringbuffer dir, -n 4	ecf18a0a2	15	0

Both crashes were the same case and both were segfaults, not assertion failures.

The last row is the useful one but it is not a clean control, so read it carefully: the tmr directory contains spmd_sync_start_early_dispatch's tensormap twin, which drives the same scene through the pre-#2095 machinery, and 15 parallel sweeps of it produced no segfault. Every one of those 15 runs also hit an unrelated TestL3Group::test_run device-allocation error under my --device 0-15 -n 4 invocation, so the two sweeps are not identical in shape or in how far they got. It is suggestive of the fault being on hbg's new path, not proof.

Host: 320-core aarch64, load average 32–60 (shared box, other work running).

Expected Behavior

The case passes, or fails with a diagnosable error. A parallel sweep should not lose an xdist worker to a native fault.

Actual Behavior

Fatal Python error: Segmentation fault

Thread 0x0000ffff9419b940 (most recent call first):
  File "python/simpler/worker.py", line 11404 in _wait_run_handle
  File "python/simpler/worker.py", line 4227 in wait
  File "python/simpler/worker.py", line 11046 in run
  File "simpler_setup/scene_test.py", line 1802 in _run_and_validate_l2
  File "simpler_setup/scene_test.py", line 1703 in _run_and_validate
  File "simpler_setup/scene_test.py", line 1284 in run_class_cases
  File "simpler_setup/scene_test.py", line 1983 in test_run
  ...
Extension modules: numpy._core._multiarray_umath, ..., torch_npu._C, ... (total: 25)
[gw0] node down: Not properly terminated

pytest reports 1 failed, so on a red CI run this looks like an ordinary test failure until the log is opened.

Git Commit ID

ecf18a0

CANN Version

9.0.0 (V100R001C10SPC001B250)

Driver Version

Not applicable — a2a3sim runs on the host CPU and opens no device.

Host Platform

Linux (aarch64)

Additional Context

Related: #2106 — a follow-up to #2095 in exactly this machinery (the progress_flags PUBLISHED/COMPLETED byte and the ED publish-list seal). It is filed as low-priority technical debt and states that "the fetch_or encoding is correct and validated". Worth re-reading if this fault turns out to be in that race window, because the PUBLISHED bookkeeping write racing the COMPLETED store is described there as the reason the encoding has to be monotone — and the wake-list sentinel protocol it mentions is what a stranded or double-freed waiter would corrupt.

What the case exercises, from its own docstring: "A wide flagged producer feeds a MIX sync_start early-dispatch consumer (hbg)" — the producer stays deliberately wider than the device and carries no sync_start, while the consumer's require_sync_start needs every block co-resident. #2095's description notes that "partially staged SPMD consumers fall through so their remaining blocks dispatch off the ready queue", which is the interleaving this scene is built to hit.

The sync_start area has form for timing-sensitive defects that only surface under a specific interleaving — #1455 (drain-retry ABA across reusable barrier state) and #1304 (early-dispatch launch-latch double-claim). Both were on the tmr path, so they are prior art on the shape of the bug rather than candidates for this one.

Two things I did not do, so nobody assumes they are covered:

• No core dump or native backtrace. faulthandler prints the Python stack, not the C++ one, so the faulting instruction is unknown. Getting it wants ulimit -c unlimited plus gdb, or an ASAN build — sanitizers.yml already exists for that and would be the cheapest next step.
• No narrowing of which concurrent work matters. The trigger might be any of the ~15 directories in the sweep, or plain CPU pressure. I established only that the case alone is not enough.

cc @ChaoZheng109 as #2095's author.