[Code Health] Require upgraded a5 environment supporting sdma

[jvjhfhg]

Category

Technical Debt (cleanup, refactor)

Component

Host Runtime

Description

PR #1179 landed the a5 SDMA workspace overlay (ensure_sdma_workspace → aclnnShmemSdmaStarsQuery) but gated it behind an opt-in macro because the available a5 CANN drops do not expose working SDMA primitives:

• CANN 9.1.T500: aclnnShmemSdmaStarsQuery creates STARS streams but aclrtSynchronizeStream fails with AICPU exception 0x715002a, which poisons the AICPU context and turns every subsequent kernel launch into 507018. This breaks all a5 communication cases, not just the SDMA demo.
• CANN 9.1.0 (timestamp=20260625): HcclCommInitRootInfo itself returns HCCL_E_INTERNAL (4) — base HCCL never comes up.

The overlay was verified working on a separate a5 box whose CANN does expose the primitive (sdma_async_completion_demo passes). So this is purely an environment gating issue, not a code defect.

Currently the overlay defaults OFF via option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ... OFF) and the SIMPLER_ENABLE_PTO_SDMA_WORKSPACE env var. With it OFF: ensure_sdma_workspace is a no-op, CommContext.workSpace stays 0, the SDMA demo self-skips, and all other comm cases run normally.

(revised 2026-07-22) PR #1392 (ca000263) added the URMA deferred completion backend behind the same gating pattern (SIMPLER_ENABLE_PTO_URMA_WORKSPACE, default OFF), and PR #1421 (a7a29ee3) added its two-device demo. The two overlays are mutually exclusive — see the 2026-07-22 update below.

Location

(revised 2026-07-22 — extended to both overlays and the post-#1403 mechanism)

- `src/a5/platform/onboard/host/CMakeLists.txt` — `option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ... OFF)` + `option(SIMPLER_ENABLE_PTO_URMA_WORKSPACE ... OFF)`, the mutual-exclusion `FATAL_ERROR` guard, and the shared `if(SDMA OR URMA)` `PTO_ISA_ROOT` check (`-DPTO_ISA_ROOT=` required; `$ENV{PTO_ISA_ROOT}` is explicitly rejected since #1403)
- `simpler_setup/runtime_compiler.py` — `_init_a5` resolves the pinned pto-isa checkout via `ensure_pto_isa_root()` when `_sdma_workspace_enabled() or _urma_workspace_enabled()` (overlay-keyed, per #1351); path stored for RuntimeBuilder to pass by value as `-DPTO_ISA_ROOT=` (#1403)
- `simpler_setup/runtime_builder.py` — `_compile_target` env-var→CMake-define forwarding for both overlay vars
- `examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py` — `pytest.skip` when the SDMA env var is unset
- `examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/test_urma_deferred_completion_demo.py` — `pytest.skip` when the URMA env var is unset (#1421)

Proposed Fix

Once the st-onboard-a5 CI CANN exposes a working aclnnShmemSdmaStarsQuery (verify with nm -D $ASCEND_HOME_PATH/lib64/libascendcl.so | grep -ic SdmaStars), re-enable the overlay. Two options:

Option A — CI env var only (no code change): set SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON in the st-onboard-a5 job. (revised 2026-07-22) No manual PTO_ISA_ROOT export is needed — and ambient $ENV{PTO_ISA_ROOT} is explicitly rejected since #1403; _init_a5 resolves the pinned checkout automatically once the overlay var is set, and RuntimeBuilder passes it by value. The existing env-var→CMake define forwarding and test skip gate pick it up automatically.

Option B — make SDMA the a5 default (revert the gating):

(revised 2026-07-22 — step 2 rewritten; the originally described mechanism no longer exists post-#1351/#1403)

1. CMakeLists.txt: option(... OFF) → set(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ON); move the PTO_ISA_ROOT check + pto-isa include back out of the if-guard.
2. runtime_compiler.py _init_a5: make env_manager.ensure("PTO_ISA_ROOT") unconditional again (mirror _init_a2a3). Obsolete. Since [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 the a5 pto-isa resolution is keyed on the overlay being enabled, and since [Code Health] Stop transporting PTO_ISA_ROOT via the environment; pass the pin-resolved path by value / -D #1403 it flows by value (ensure_pto_isa_root() + -DPTO_ISA_ROOT=), same as _init_a2a3. Flipping the option ON pulls the guard in automatically — no _init_a5 edit needed.
3. runtime_builder.py _compile_target: the env-var→define forwarding can stay (harmless) or be removed.
4. test_sdma_async_completion_demo.py: remove the env-var pytest.skip gate.

Optionally also have build_runtimes.py auto-resolve PTO_ISA_ROOT for a5 the same way it does for a2a3 — (2026-07-22) obsoleted by the overlay-keyed resolution from #1351.

Verification after re-enabling:

python -m pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py -v --platform a5 --device <ids> -s
python -m pytest examples/workers/l3/allreduce_distributed/test_allreduce.py -v --platform a5 --device <ids> -k onephase

The SDMA demo must PASS (not skip) and the allreduce regression must stay green.

Priority

Low (no impact today, good to fix eventually)

---

Update (2026-07-13): prerequisite — resolve #1351 before re-enabling — ✅ RESOLVED 2026-07-20

(status annotated 2026-07-22)

Per the discussion below, flipping SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON (under either Option A or Option B) is blocked on #1351.

Enabling the overlay makes a5 host_runtime.so compile pto-isa headers into itself, but the PTO-ISA build/run version guard (#1096 + #1194) is still scoped to arch == "a2a3" — it was never extended to a5 when #1179 introduced this dependency. Re-enabling now would embed pto-isa on a5 with:

• no pin verification
• no build-metadata recording
• no load-time staleness rejection
• no cmake-cache / ccache invalidation on a pin bump

— silently re-arming the 507018/507899-class stale-binary failure #1194 was written to prevent.

#1351 was closed as COMPLETED on 2026-07-20, implemented exactly as recommended here: the a5 guard is keyed on the overlay being enabled (SIMPLER_ENABLE_PTO_SDMA_WORKSPACE / SIMPLER_ENABLE_PTO_URMA_WORKSPACE truthy), not on arch == "a5" — so a default-OFF a5 build is untouched, and flipping the overlay ON automatically pulls in pin verification / build metadata / staleness rejection / cache invalidation. Re-enabling can no longer bypass the guard.

Re-enable checklist (updated 2026-07-22):

• [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 resolved (PTO-ISA version guard extended to a5, overlay-keyed)
• CANN exposes working aclnnShmemSdmaStarsQuery (verify via nm -D ... | grep -ic SdmaStars)
• Proceed with Option A or Option B

---

Update (2026-07-20): URMA overlay added in PR #1392, same gating pattern — ✅ LANDED 2026-07-22

(status annotated 2026-07-22)

PR #1392 (landed as ca000263) introduces the a5 URMA deferred completion backend. Its host-side URMA workspace is gated behind SIMPLER_ENABLE_PTO_URMA_WORKSPACE (default OFF), mirroring the SDMA overlay. When OFF:

• a5 host_runtime does not require PTO_ISA_ROOT.
• comm_hccl.cpp does not compile/link the PTO-ISA UrmaWorkspaceManager.
• comm_alloc_windows / comm_alloc_domain_windows cannot fail due to URMA workspace initialization.
• Existing non-URMA a5 communication paths keep current behavior.

(revised 2026-07-22) Manual validation on real a5: set SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON and rebuild — the pinned pto-isa checkout is resolved automatically (no manual PTO_ISA_ROOT export; ambient exports are rejected since #1403).

PR #1421 (landed as a7a29ee3) added examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo, the concrete vehicle for the two-device verification step below.

URMA re-enable checklist (updated 2026-07-22):

• [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 resolved — PTO-ISA version guard / build metadata / cache invalidation covers a5 host runtime (overlay-keyed)
• st-onboard-a5 CI CANN exposes working URMA/HCCL async workspace primitives
• (added 2026-07-22) Resolve the SDMA/URMA mutual exclusion first if both overlays are ever needed simultaneously — see the 2026-07-22 update below
• Enable SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON in the a5 validation environment (pin-resolved pto-isa flows automatically)
• Verify the real a5 URMA deferred completion path on two devices:
  python -m pytest examples/a5/tensormap_and_ringbuffer/urma_deferred_completion_demo/ -v --platform a5 --device <ids> (must PASS, not skip)
• Verify existing non-URMA a5 communication cases remain green
• Only after the above, consider enabling URMA in CI or changing the default

---

Update (2026-07-22): #1392 / #1421 landed; SDMA+URMA mutually exclusive

• PR feat(a5): add URMA deferred completion backend for tensormap_and_ringbuffer #1392 landed as ca000263: URMA deferred completion backend + SIMPLER_ENABLE_PTO_URMA_WORKSPACE gating (default OFF), mirroring the SDMA overlay.
• PR feat(a5): add URMA deferred completion demo for tensormap_and_ringbuffer #1421 landed as a7a29ee3: urma_deferred_completion_demo on main.
• [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 closed COMPLETED 2026-07-20 — the re-enable precondition from the 2026-07-13 update is satisfied.

New constraint from #1392 — SDMA and URMA overlays are mutually exclusive. src/a5/platform/onboard/host/CMakeLists.txt hard-fails at configure time (FATAL_ERROR) if both SIMPLER_ENABLE_PTO_SDMA_WORKSPACE and SIMPLER_ENABLE_PTO_URMA_WORKSPACE are ON, because CommContext carries a single workSpace/workSpaceSize pair that both backends would claim. The re-enable plan must therefore either:

• enable only one overlay in CI (the other stays opt-in local), or
• first split the CommContext workspace fields per backend, then enable both.

Remaining blocker for both overlays: st-onboard-a5 CI CANN exposing working SDMA (aclnnShmemSdmaStarsQuery) / URMA primitives.

Optional cleanup while re-enabling (non-blocking leftovers from the #1392 review): src/a5/platform/onboard/host/CMakeLists.txt links c_sec/nnopbase unconditionally (should fold into the overlay if block); src/a5/runtime/tensormap_and_ringbuffer/runtime/backend/urma/urma_completion_scheduler.h kCqeBytes is unused.

[jvjhfhg]

Added docs/a5-sdma-overlay.md as the single source of truth for the isolation status, with cross-referenced comments at every gating point. Updates to this issue:
Location — add:

• docs/a5-sdma-overlay.md — central doc (status, gating points, dev-under-isolation, re-enable steps)
• examples/a5/.../kernels/aiv/kernel_sdma_tget_async.cpp — workSpace == 0 self-skip guard now references the doc
  Proposed Fix — both Option A and Option B are unchanged, but after re-enabling also update docs/a5-sdma-overlay.md (the "Current state (default OFF)" section + re-enable verification) so the doc reflects the new always-on reality. The code comments at the 5 gating points already point to this doc, so no scattered comment edits are needed beyond flipping the gate itself.
  The doc covers: what the overlay is, why it's gated (CANN 9.1.T500 → AICPU exception 0x715002a → 507018; 9.1.0 → HCCL_E_INTERNAL), current default-OFF behavior, the 5 gating-point file table, how to develop non-SDMA / new-SDMA features under isolation, and the Option A/B re-enable steps with verification commands.

[ChaoZheng109]

⚠️ Required precondition before flipping SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON (both Option A and Option B): resolve #1351 first.

Turning the overlay ON makes a5 host_runtime.so compile pto-isa headers into itself, but the PTO-ISA build/run version guard (#1096 + #1194) is still scoped to arch == "a2a3" — it was never extended to a5 when #1179 added this dependency (see #1351). So re-enabling via either option here would embed pto-isa on a5 with no pin verification, no build-metadata recording, no load-time staleness rejection, and no cmake-cache / ccache invalidation on a pin bump — silently re-arming the 507018/507899-class stale-binary failure #1194 was written to prevent.

Please add '#1351 resolved' as an explicit checklist item before the re-enable steps (Option A and Option B).

[jvjhfhg]

⚠️ Required precondition before flipping SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON (both Option A and Option B): resolve #1351 first.

Turning the overlay ON makes a5 host_runtime.so compile pto-isa headers into itself, but the PTO-ISA build/run version guard (#1096 + #1194) is still scoped to arch == "a2a3" — it was never extended to a5 when #1179 added this dependency (see #1351). So re-enabling via either option here would embed pto-isa on a5 with no pin verification, no build-metadata recording, no load-time staleness rejection, and no cmake-cache / ccache invalidation on a pin bump — silently re-arming the 507018/507899-class stale-binary failure #1194 was written to prevent.

Please add '#1351 resolved' as an explicit checklist item before the re-enable steps (Option A and Option B).

@ChaoZheng109 Added to the issue body.

[wxwnnzdyd]

Adding the URMA follow-up here so #1315 remains the single tracking issue for re-enabling A5 PTO async workspace overlays after the CI environment is
ready.

For PR #1392, the host-side URMA workspace is now gated behind SIMPLER_ENABLE_PTO_URMA_WORKSPACE and defaults to OFF, mirroring the existing SDMA
overlay gating. When OFF:

• A5 host_runtime does not require PTO_ISA_ROOT.
• comm_hccl.cpp does not compile/link the PTO-ISA UrmaWorkspaceManager.
• comm_alloc_windows / comm_alloc_domain_windows do not fail because of URMA workspace initialization.
• Existing non-URMA A5 communication paths should keep using the current upstream behavior.

URMA can still be validated manually on real A5 by explicitly enabling the overlay:

export SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON
export PTO_ISA_ROOT=/path/to/pto-isa

URMA re-enable checklist, alongside the existing SDMA checklist:

- [ ] #1351 is resolved, so PTO-ISA version guard / metadata / cache invalidation also covers A5 host runtime.
- [ ] The st-onboard-a5 CI CANN environment exposes working URMA/HCCL async workspace primitives.
- [ ] Enable SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON in the A5 validation environment with a pinned PTO_ISA_ROOT.
- [ ] Verify the real A5 URMA deferred completion path on two devices.
- [ ] Verify existing non-URMA A5 communication cases remain green.
- [ ] Only after the above, consider enabling URMA workspace in CI or changing the default.

[jvjhfhg]

Adding the URMA follow-up here so #1315 remains the single tracking issue for re-enabling A5 PTO async workspace overlays after the CI environment is ready.

For PR #1392, the host-side URMA workspace is now gated behind SIMPLER_ENABLE_PTO_URMA_WORKSPACE and defaults to OFF, mirroring the existing SDMA overlay gating. When OFF:

• A5 host_runtime does not require PTO_ISA_ROOT.
• comm_hccl.cpp does not compile/link the PTO-ISA UrmaWorkspaceManager.
• comm_alloc_windows / comm_alloc_domain_windows do not fail because of URMA workspace initialization.
• Existing non-URMA A5 communication paths should keep using the current upstream behavior.

URMA can still be validated manually on real A5 by explicitly enabling the overlay:

export SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON
export PTO_ISA_ROOT=/path/to/pto-isa

URMA re-enable checklist, alongside the existing SDMA checklist:

• [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 is resolved, so PTO-ISA version guard / metadata / cache invalidation also covers A5 host runtime.
• The st-onboard-a5 CI CANN environment exposes working URMA/HCCL async workspace primitives.
• Enable SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON in the A5 validation environment with a pinned PTO_ISA_ROOT.
• Verify the real A5 URMA deferred completion path on two devices.
• Verify existing non-URMA A5 communication cases remain green.
• Only after the above, consider enabling URMA workspace in CI or changing the default.

@wxwnnzdyd Added to the issue body.

[yanghaoran29]

Status update (2026-07-21)

Checklist progress

• [Code Health] Extend PTO-ISA build/run version guard to a5 onboard SDMA overlay (missed after #1179) #1351 — PTO-ISA pin/metadata/staleness/ccache guard now covers a5 async overlays (closed via Fix: extend PTO-ISA version guard to a5 async workspace overlays #1404).
• CANN behaviorally completes aclnnShmemSdmaStarsQuery — still blocked (see retest below).
• Option A / Option B — not started; will not flip defaults or CI env until the behavior probe PASSes.

URMA checklist remains in the same state: #1351 done; working-CANN verification still pending.

Empiric retest on current pin (this machine)

Environment:

• CANN 9.1.T500 (ASCEND_HOME_PATH=/usr/local/Ascend/cann-9.1.T500)
• SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON
• current pto_isa.pin (83d01313…)
• rebuilt a5 tensormap_and_ringbuffer host_runtime with overlay ON
• devices via task-submit (--device-num 2)

Result on sdma_async_completion_demo:

1. Created 48 STARS streams OK
2. aclrtSynchronizeStream fails → AICPU exception 0x715002a
3. subsequent launches report 507018

So the gate is still environment behavior, not “missing symbols”. On this drop, SdmaStars* lives in libopapi.so (~6 symbols); probing libascendcl.so alone is misleading (often 0).

What we are / are not doing

• Not Option B (default ON) while 9.1.T500-class CANN is the local/CI baseline.
• Not Option A (force-on in CI) until sdma_async_completion_demo PASSes on the target CANN.
• Transport cleanup + doc refresh (correct probe + this retest note) is in Fix: pass pin-resolved PTO_ISA_ROOT by value and -D #1418 (Fixes #1403; related to this issue). Central doc: docs/a5-sdma-overlay.md.

This issue should stay open until a CANN drop passes the behavior probe (not just nm).