[Code Health] hbg: ready queues are sized by a fixed constant, and one is reachable by more tasks than it has slots

[ChaoWao]

Category

Robustness (potential edge-case failure)

Component

AICPU Scheduler

Description

All nine of the scheduler's ready queues are reserved at the same fixed
PTO2_READY_QUEUE_SIZE (8192) regardless of how many tasks can route to each, and
on examples/a2a3/host_build_graph/qwen3_14b_decode one of them is reachable by
more tasks than it has slots.

enqueue_ready routes every task to exactly one queue — DUMMY shape or a failed
predicate to the dummy queue, a sync-start requirement to the per-shape sync queue,
everything else to the per-shape ready queue — and a task occupies one slot at a
time, so the count of tasks that can reach a queue is an upper bound on its peak
occupancy. A false push then latches PTO2_ERROR_READY_QUEUE_OVERFLOW and fails
the run. That is deliberate and correct as a response (silently dropping the task
would stall the run instead), but it means queue capacity is a correctness bound,
not a tuning knob — there is no retry path.

The bound is exceeded on qwen3-14b decode, and the counts that show it are already
printed by main:

$ SIMPLER_HBG_BIND_BREAKDOWN_ENABLE=1 SIMPLER_LOG_LEVEL=TIMING \
    python examples/a2a3/host_build_graph/qwen3_14b_decode/test_qwen3_14b_decode.py \
      -p a2a3 -d $TASK_DEVICE --manual include --rounds 3
...
host-orch phase=record_node       total_ns=148731 count=277
host-orch phase=graph_submit      total_ns=200812 count=40

The run replays one 277-node Definition 40 times, so 40 x 277 = 11,080 nodes enter
the outer task set on top of the host's own tasks. Even that loose upper bound
already exceeds 8192; instrumenting the routing rule per queue puts
ready_queue[AIC]'s reachable population at 9,400 against its 8,192 slots.

What keeps this closed today is drain rate, not the bound. The peak occupancy
the run actually reaches is 101 of 8192, so the queue never fills and nothing
reports the gap — the capacity is simply never compared against the population that
can arrive. Any workload that widens the DAG's antichain (a larger batch, a branchier
graph, more layers at the same nodes per layer, or a slower consumer) closes the
distance, and the failure mode when it does is a latched fatal error rather than
back-pressure.

Both architecture trees carry the same constant, so this is not arch-specific.

Location

On 93adc386:

• src/a2a3/runtime/host_build_graph/runtime/pto_runtime2_types.h:104 —
  #define PTO2_READY_QUEUE_SIZE 8192
• src/a5/runtime/host_build_graph/runtime/pto_runtime2_types.h:103 — same constant
• src/a2a3/runtime/host_build_graph/runtime/shared/pto_runtime2_init.cpp:99-116 —
  all nine queues reserved at that constant, before orchestration knows the task set
• src/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.h:548-570 —
  enqueue_ready's routing and the if (!pushed) latch_ready_queue_overflow();
• src/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.h:527 —
  latch_ready_queue_overflow, a single CAS into sched_error_code with no retry

Proposed Fix

Derive each queue's capacity from the population that can reach it, and raise the
reservation ceiling to admit the largest.

The population is computable at bind time: walk the shared-memory task window for
the host's own tasks, then per Graph submission add the node histogram of the
Definition it references — counted once per distinct Definition and applied per
submission, so a run replaying one Definition 40 times pays one 277-node walk. A
predicated task counts toward both the dummy queue and its shape's queue, since the
predicate is evaluated on device.

Costs, measured:

• the arena reservation grows 11,515,584 -> 16,824,064 bytes for a ceiling of 32768
  across nine queues, allocated once per worker and reused across runs
• arena_h2d grows 632 -> 720 bytes; the +88 is the per-queue capacities travelling
  in the layout, which since host_build_graph: partition the runtime arena into copy and no-copy zones #1894 / host_build_graph: move the completion mailbox into the device-only zone #1895 is their only channel to the device
• the population pass itself costs 0.0021 ms on qwen's 47 tasks and 0.0076 ms on
  DeepSeek-V4 FLASH decode's 1,131, i.e. ~6.6 ns a task

Note that this bounds the exposure rather than removing it: a population past
whatever ceiling is chosen still has to clamp, and the clamp should report at bind
time instead of letting the run fail later. Removing it for any population needs a
full-queue push the caller can recover from, and both shapes of that are currently
blocked — spinning until the push succeeds deadlocks at aicpu_thread_num=1, where
the pushing thread is the only consumer, and parking the task for retry from the
thread's own dispatch loop needs a thread identity that none of the four calling
chains carries.

A tighter bound would shrink capacity without touching the push path at all: the
DAG's antichain width rather than the reachable count, which the Definition's fanin
CSR already carries.

I have this implemented and measured on a branch and can open the PR.

Related: #1706

Priority

Medium (minor risk, should fix in next few releases)