Update: collapse the AICPU scheduler timeout to one 20 s constant

docs/capability-survey.md

@@ -169,8 +169,8 @@ The a2a3 mirror of this ordering is self-documented as unverified on a2a3
 silicon.
 
 Three timeouts exist, and the constants are **defaults, not the operative
-values**: op-execute 45 s, stream-sync 50 s, scheduler 10 s
-(`src/a2a3/platform/include/common/platform_config.h:69,85,75`). All three are
+values**: op-execute 45 s, stream-sync 50 s, scheduler 20 s
+(`src/a2a3/platform/include/common/platform_config.h:77,100,90`). All three are
 env-overridable with ordering validation
 (`resolve_onboard_timeout_config`, `device_runner_base.cpp:66-110`;
 [troubleshooting/local-timeout-defaults.md](troubleshooting/local-timeout-defaults.md)),

docs/dfx/args-dump.md

@@ -914,7 +914,7 @@ on the timeout ordering — the three budgets are tuned so the **AICPU detects t
 hang first**, dumps, and only then the hardware/host timeouts fire:
 
 ```text
-SCHEDULER_TIMEOUT_MS (10 s, onboard)  <  PLATFORM_OP_EXECUTE_TIMEOUT_US (45 s)  <  PLATFORM_STREAM_SYNC_TIMEOUT_MS (50 s)
+SCHEDULER_TIMEOUT_MS (20 s)           <  PLATFORM_OP_EXECUTE_TIMEOUT_US (45 s)  <  PLATFORM_STREAM_SYNC_TIMEOUT_MS (50 s)
    AICPU declares hang,                   STARS reaps the AICore op              host stream sync gives up
    flushes + dumps in-flight              and poisons the context                and surfaces the error
 ```
@@ -936,7 +936,7 @@ the no-progress budget without onboard-only ordering limits. CI restores the
 old fast-fail values through these env vars: 2 s scheduler, 3 s op-execute,
 and 4 s stream-sync for onboard jobs; 5 s scheduler for sim jobs.
 
-- **Device-side graceful flush (primary).** At 10 s of no progress
+- **Device-side graceful flush (primary).** At 20 s of no progress
   the AICPU declares the hang, runs the end-of-loop flush, *and*
   dumps the **partial output** of every task still RUNNING on a core
   — written at the `after_completion` stage, reflecting current GM,
@@ -963,9 +963,9 @@ This ordering is load-bearing: if the timeouts were inverted (STARS
 reaping before the AICPU's budget, as in earlier versions), the
 device-side dump would never run on a real AICore hang and you would
 only recover what was already in the buffer. The chain lives in
-`spin_hint.h` (`PLATFORM_SCHEDULER_TIMEOUT_MS`, surfaced as
-`SCHEDULER_TIMEOUT_MS` — 10 s for onboard and sim defaults) and
-`platform_config.h` (`PLATFORM_OP_EXECUTE_TIMEOUT_US` /
+`platform_config.h`, which holds all three (`PLATFORM_SCHEDULER_TIMEOUT_MS`,
+surfaced as `SCHEDULER_TIMEOUT_MS` — 20 s, one value for onboard and sim,
+`PLATFORM_OP_EXECUTE_TIMEOUT_US` /
 `PLATFORM_STREAM_SYNC_TIMEOUT_MS`). The env overrides use those constants as
 their unset fallback and keep the `#897` distributed-skew trade-off.
 

docs/troubleshooting/local-timeout-defaults.md

@@ -1,9 +1,15 @@
 # Local Runtime Timeouts
 
 Local runs use production-friendly timeout defaults. Onboard platforms wait up
-to 10 s for AICPU scheduler no-progress, 45 s for STARS op-execute timeout,
-and 50 s for host stream synchronization. Sim platforms use a 10 s scheduler
-timeout and do not have STARS or ACL stream-sync timeouts.
+to 20 s for AICPU scheduler no-progress, 45 s for STARS op-execute timeout,
+and 50 s for host stream synchronization. Sim platforms use the same 20 s
+scheduler timeout and do not have STARS or ACL stream-sync timeouts.
+
+The scheduler budget is a single constant (`PLATFORM_SCHEDULER_TIMEOUT_MS` in
+each arch's `platform_config.h`) shared by onboard and sim, because both run
+the same no-progress watchdog. It is sized to outlast a slow CPU-sim kernel on
+an oversubscribed host while still firing well before the 45 s STARS op-execute
+timeout onboard.
 
 This means a real local hang can take much longer to surface than it does in
 CI. CI restores the old fast-fail values with environment overrides:

docs/user/how-to/debug-a-failed-run.md

@@ -54,7 +54,7 @@ export ASCEND_PROCESS_LOG_PATH="$LOGDIR"
 ### If it is merely slow, not stuck
 
 The timeouts are compile-time defaults — op-execute 45 s, stream-sync 50 s,
-scheduler 10 s — and all three are environment-overridable, with the ordering
+scheduler 20 s — and all three are environment-overridable, with the ordering
 between them validated at startup. Raising them is how you tell "genuinely
 hung" from "legitimately long":
 

docs/user/reference/cli.md

@@ -78,7 +78,7 @@ python -m simpler_setup.tools.swimlane_converter <chip_swimlane_records_*.json>
 | -------- | ------ |
 | `SIMPLER_OP_EXECUTE_TIMEOUT_US` | Overrides the op-execute timeout (default 45 s) |
 | `SIMPLER_STREAM_SYNC_TIMEOUT_MS` | Overrides the stream-sync timeout (default 50 s) |
-| `SIMPLER_SCHEDULER_TIMEOUT_MS` | Overrides the scheduler timeout (default 10 s) |
+| `SIMPLER_SCHEDULER_TIMEOUT_MS` | Overrides the scheduler timeout (default 20 s) |
 | `ASCEND_PROCESS_LOG_PATH` | Redirects the device log into a directory you own; the directory must already exist |
 | `ASCEND_HOME_PATH` | CANN toolkit location; required for hardware platforms |
 

src/a2a3/platform/include/common/platform_config.h

@@ -77,10 +77,17 @@ constexpr int PLATFORM_MAX_AICPU_THREADS_JUST_FOR_LAUNCH = 6;
 constexpr uint64_t PLATFORM_OP_EXECUTE_TIMEOUT_US = 45000000;  // 45s
 
 /**
- * Default onboard AICPU scheduler no-progress timeout (milliseconds).
- * Shared with host-side timeout ordering validation.
- */
-constexpr int32_t PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS = 10000;
+ * Default AICPU scheduler no-progress timeout (milliseconds).
+ * One value for every platform variant: onboard and sim run the same
+ * no-progress watchdog, and a single constant keeps them from drifting.
+ * Must stay below PLATFORM_OP_EXECUTE_TIMEOUT_US so that onboard the AICPU
+ * declares the hang and flushes its diagnostics before STARS reaps the op.
+ * Sized to outlast a slow CPU-sim kernel on an oversubscribed host, where
+ * the AICPU scheduler threads share cores with the AICore threads.
+ * Shared with host-side timeout ordering validation. Overridden at runtime
+ * by SIMPLER_SCHEDULER_TIMEOUT_MS when that env var is valid.
+ */
+constexpr int32_t PLATFORM_SCHEDULER_TIMEOUT_MS = 20000;
 
 /**
  * Default host-side stream synchronization timeout (milliseconds).

src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_types.h

@@ -19,17 +19,6 @@
 #include "host_build_graph/runtime_types.h"
 #include "spin_hint.h"
 
-// host_build_graph host-orch build: RuntimeContext embeds SchedulerState by
-// value, so this header is compiled into the host libhost_runtime.so. The AICPU
-// spin_hint.h that defines PLATFORM_SCHEDULER_TIMEOUT_MS is not on the host
-// include path; supply it here. The value only sizes an on-device scheduler
-// timeout and is never consumed host-side (the scheduler does not run on the
-// host). host_runtime_EXPORTS is CMake's auto-define for the host shared-lib
-// target, so the AICPU/AICore builds keep the real platform constant.
-#ifdef host_runtime_EXPORTS
-constexpr int32_t PLATFORM_SCHEDULER_TIMEOUT_MS = 2000;
-#endif
-
 // =============================================================================
 // Profiling macros (compile-time gated)
 // =============================================================================
@@ -74,11 +63,11 @@ constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024;  // Check for a latched sch
 // kills the slower-but-correct poller mid-poll — see the distributed
 // startup-skew scenario in issue #897.
 //
-// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in spin_hint.h).
-// Onboard keeps it below the STARS op-execute and host stream-sync budgets so
-// the AICPU can flush diagnostics before the host-visible timeout chain fires.
-// Sim has no STARS or ACL stream-sync timeout, but uses the same no-progress
-// watchdog shape. See spin_hint.h for the per-variant rationale.
+// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in
+// platform_config.h), one value across every platform variant. Onboard keeps it
+// below the STARS op-execute and host stream-sync budgets so the AICPU can flush
+// diagnostics before the host-visible timeout chain fires. Sim has no STARS or
+// ACL stream-sync timeout, but runs the same no-progress watchdog.
 constexpr int32_t SCHEDULER_TIMEOUT_MS = PLATFORM_SCHEDULER_TIMEOUT_MS;
 constexpr uint64_t SCHEDULER_TIMEOUT_CYCLES =
     static_cast<uint64_t>(SCHEDULER_TIMEOUT_MS) * (PLATFORM_PROF_SYS_CNT_FREQ / 1000);

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_types.h

@@ -63,11 +63,11 @@ constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024;  // Check orchestrator erro
 // kills the slower-but-correct poller mid-poll — see the distributed
 // startup-skew scenario in issue #897.
 //
-// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in spin_hint.h).
-// Onboard keeps it below the STARS op-execute and host stream-sync budgets so
-// the AICPU can flush diagnostics before the host-visible timeout chain fires.
-// Sim has no STARS or ACL stream-sync timeout, but uses the same no-progress
-// watchdog shape. See spin_hint.h for the per-variant rationale.
+// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in
+// platform_config.h), one value across every platform variant. Onboard keeps it
+// below the STARS op-execute and host stream-sync budgets so the AICPU can flush
+// diagnostics before the host-visible timeout chain fires. Sim has no STARS or
+// ACL stream-sync timeout, but runs the same no-progress watchdog.
 constexpr int32_t SCHEDULER_TIMEOUT_MS = PLATFORM_SCHEDULER_TIMEOUT_MS;
 constexpr uint64_t SCHEDULER_TIMEOUT_CYCLES =
     static_cast<uint64_t>(SCHEDULER_TIMEOUT_MS) * (PLATFORM_PROF_SYS_CNT_FREQ / 1000);

src/a5/platform/include/common/platform_config.h

@@ -103,10 +103,17 @@ constexpr int PLATFORM_MAX_AICPU_THREADS_JUST_FOR_LAUNCH = 14;
 constexpr uint64_t PLATFORM_OP_EXECUTE_TIMEOUT_US = 45000000;  // 45s
 
 /**
- * Default onboard AICPU scheduler no-progress timeout (milliseconds).
- * Shared with host-side timeout ordering validation.
- */
-constexpr int32_t PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS = 10000;
+ * Default AICPU scheduler no-progress timeout (milliseconds).
+ * One value for every platform variant: onboard and sim run the same
+ * no-progress watchdog, and a single constant keeps them from drifting.
+ * Must stay below PLATFORM_OP_EXECUTE_TIMEOUT_US so that onboard the AICPU
+ * declares the hang and flushes its diagnostics before STARS reaps the op.
+ * Sized to outlast a slow CPU-sim kernel on an oversubscribed host, where
+ * the AICPU scheduler threads share cores with the AICore threads.
+ * Shared with host-side timeout ordering validation. Overridden at runtime
+ * by SIMPLER_SCHEDULER_TIMEOUT_MS when that env var is valid.
+ */
+constexpr int32_t PLATFORM_SCHEDULER_TIMEOUT_MS = 20000;
 
 /**
  * Default host-side stream synchronization timeout (milliseconds).

src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_types.h

@@ -32,19 +32,6 @@
 #include "spin_hint.h"
 #endif
 
-// host_build_graph host-orch build: RuntimeContext embeds SchedulerState by
-// value, so this header is compiled into the host libhost_runtime.so. The AICPU
-// spin_hint.h that defines PLATFORM_SCHEDULER_TIMEOUT_MS is not on the host
-// include path; supply it here. The value only sizes an on-device scheduler
-// timeout and is never consumed host-side (the scheduler does not run on the
-// host). host_runtime_EXPORTS is CMake's auto-define for the host shared-lib
-// target, so the AICPU/AICore builds keep the real platform constant.
-#ifdef host_runtime_EXPORTS
-constexpr int32_t HBG_LEGACY_SCHEDULER_TIMEOUT_MS = 2000;
-#else
-constexpr int32_t HBG_LEGACY_SCHEDULER_TIMEOUT_MS = PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS;
-#endif
-
 // =============================================================================
 // Profiling macros (compile-time gated)
 // =============================================================================
@@ -89,12 +76,12 @@ constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024;  // Check for a latched sch
 // kills the slower-but-correct poller mid-poll — see the distributed
 // startup-skew scenario in issue #897.
 //
-// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in spin_hint.h).
-// Onboard keeps it below the STARS op-execute and host stream-sync budgets so
-// the AICPU can flush diagnostics before the host-visible timeout chain fires.
-// Sim has no STARS or ACL stream-sync timeout, but uses the same no-progress
-// watchdog shape. See spin_hint.h for the per-variant rationale.
-constexpr int32_t SCHEDULER_TIMEOUT_MS = HBG_LEGACY_SCHEDULER_TIMEOUT_MS;
+// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in
+// platform_config.h), one value across every platform variant. Onboard keeps it
+// below the STARS op-execute and host stream-sync budgets so the AICPU can flush
+// diagnostics before the host-visible timeout chain fires. Sim has no STARS or
+// ACL stream-sync timeout, but runs the same no-progress watchdog.
+constexpr int32_t SCHEDULER_TIMEOUT_MS = PLATFORM_SCHEDULER_TIMEOUT_MS;
 constexpr uint64_t SCHEDULER_TIMEOUT_CYCLES =
     static_cast<uint64_t>(SCHEDULER_TIMEOUT_MS) * (PLATFORM_PROF_SYS_CNT_FREQ / 1000);
 constexpr int32_t STALL_DUMP_READY_MAX = 8;

src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_types.h

@@ -63,11 +63,11 @@ constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024;  // Check orchestrator erro
 // kills the slower-but-correct poller mid-poll — see the distributed
 // startup-skew scenario in issue #897.
 //
-// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in spin_hint.h).
-// Onboard keeps it below the STARS op-execute and host stream-sync budgets so
-// the AICPU can flush diagnostics before the host-visible timeout chain fires.
-// Sim has no STARS or ACL stream-sync timeout, but uses the same no-progress
-// watchdog shape. See spin_hint.h for the per-variant rationale.
+// The budget is platform-defined (PLATFORM_SCHEDULER_TIMEOUT_MS in
+// platform_config.h), one value across every platform variant. Onboard keeps it
+// below the STARS op-execute and host stream-sync budgets so the AICPU can flush
+// diagnostics before the host-visible timeout chain fires. Sim has no STARS or
+// ACL stream-sync timeout, but runs the same no-progress watchdog.
 constexpr int32_t SCHEDULER_TIMEOUT_MS = PLATFORM_SCHEDULER_TIMEOUT_MS;
 constexpr uint64_t SCHEDULER_TIMEOUT_CYCLES =
     static_cast<uint64_t>(SCHEDULER_TIMEOUT_MS) * (PLATFORM_PROF_SYS_CNT_FREQ / 1000);

src/common/platform/onboard/aicpu/spin_hint.h

@@ -25,16 +25,13 @@
 
 #define SPIN_WAIT_HINT() ((void)0)
 
-// Wall-clock budget (ms) of no task progress before the dispatch loop aborts
-// with SIMPLER_ERROR_SCHEDULER_TIMEOUT. On real hardware this must sit *below* the
-// STARS AICore op-execution timeout (PLATFORM_OP_EXECUTE_TIMEOUT_US, 45 s)
-// so the AICPU detects the hang and flushes its diagnostics (args dump,
-// in-flight partial output) before STARS reaps the op and poisons the
-// context. Chain: this < op-exec < host stream-sync (platform_config.h). The
-// default 10 s scheduler budget covers the distributed-init / HCCL skew #897
-// sized at 5 s while still firing well before STARS. The runtime consumes it
-// as SCHEDULER_TIMEOUT_MS (see scheduler_types.h). Host may override this per
-// run via SIMPLER_SCHEDULER_TIMEOUT_MS after validating the timeout ordering.
-constexpr int32_t PLATFORM_SCHEDULER_TIMEOUT_MS = PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS;
+// The no-progress budget PLATFORM_SCHEDULER_TIMEOUT_MS is not defined here: it
+// is one value across every platform variant and lives in platform_config.h,
+// which the host also reads for timeout-ordering validation. On real hardware
+// it must sit below the STARS AICore op-execution timeout
+// (PLATFORM_OP_EXECUTE_TIMEOUT_US, 45 s) so the AICPU detects the hang and
+// flushes its diagnostics (args dump, in-flight partial output) before STARS
+// reaps the op and poisons the context. Chain: scheduler < op-exec < host
+// stream-sync, all three in platform_config.h.
 
 #endif  // PLATFORM_A2A3_AICPU_SPIN_HINT_H_

src/common/platform/onboard/host/device_runner_base.cpp

@@ -67,7 +67,7 @@ namespace {
 
 HostRuntimeTimeoutConfig resolve_onboard_timeout_config() {
     RuntimeTimeoutConfig order_defaults{
-        PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS
+        PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS
     };
     RuntimeTimeoutParseStatus parse_status;
     RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(order_defaults, &parse_status);

src/common/platform/sim/aicpu/spin_hint.h

@@ -19,11 +19,12 @@
  * especially on resource-constrained CI runners (e.g., 2 cores running 13+
  * threads).
  *
- * Two mitigations live here. The CPU hint (pause/yield) plus sched_yield() let
- * the OS scheduler give time slices to threads doing real work, and
- * PLATFORM_SCHEDULER_TIMEOUT_MS below keeps the no-progress budget generous so a
- * slow CPU-sim task (e.g. matmul-heavy kernels) making real progress is not
- * mistaken for a deadlock.
+ * The CPU hint (pause/yield) plus sched_yield() let the OS scheduler give time
+ * slices to threads doing real work. The companion mitigation is the
+ * no-progress budget PLATFORM_SCHEDULER_TIMEOUT_MS, which is sized to keep a
+ * slow CPU-sim task (e.g. matmul-heavy kernels) making real progress from being
+ * mistaken for a deadlock; it is one value across every platform variant and
+ * lives in platform_config.h.
  */
 
 #ifndef PLATFORM_A2A3SIM_AICPU_SPIN_HINT_H_
@@ -32,6 +33,8 @@
 #include <cstdint>
 #include <sched.h>
 
+#include "common/platform_config.h"
+
 #if defined(__aarch64__)
 #define SPIN_WAIT_HINT()                        \
     do {                                        \
@@ -48,14 +51,4 @@
 #define SPIN_WAIT_HINT() sched_yield()
 #endif
 
-// Wall-clock budget (ms) of no task progress before the dispatch loop aborts
-// with SIMPLER_ERROR_SCHEDULER_TIMEOUT. Unlike onboard there is no STARS
-// op-execution timeout to race here, so this keeps the full #897 distributed-init
-// / HCCL-skew headroom. A generous budget also avoids false timeouts when an
-// oversubscribed CPU-sim kernel (e.g. matmul-heavy) makes real but slow
-// progress; raise further if a slow kernel still false-times-out. The runtime
-// consumes it as SCHEDULER_TIMEOUT_MS (see scheduler_types.h). Host may
-// override this per run via SIMPLER_SCHEDULER_TIMEOUT_MS.
-constexpr int32_t PLATFORM_SCHEDULER_TIMEOUT_MS = 10000;
-
 #endif  // PLATFORM_A2A3SIM_AICPU_SPIN_HINT_H_

tests/ut/cpp/common/test_runtime_timeout_config.cpp

@@ -21,7 +21,7 @@
 namespace {
 
 constexpr RuntimeTimeoutConfig kDefaults{
-    PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_ONBOARD_SCHEDULER_TIMEOUT_MS
+    PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS
 };
 constexpr RuntimeTimeoutConfig kCiTightTimeouts{3000000, 4000, 2000};
 
@@ -93,7 +93,7 @@ TEST(RuntimeTimeoutConfig, UnsetEnvKeepsDefaults) {
 
     EXPECT_EQ(cfg.op_execute_timeout_us, 45000000u);
     EXPECT_EQ(cfg.stream_sync_timeout_ms, 50000);
-    EXPECT_EQ(cfg.scheduler_timeout_ms, 10000);
+    EXPECT_EQ(cfg.scheduler_timeout_ms, 20000);
     EXPECT_EQ(validate_runtime_timeout_order(cfg), RuntimeTimeoutOrderStatus::OK);
 }